Apache Allura™

• milestone: limbo --> mar-24

• Description has changed:

Diff:

--- old 
+++ new 
@@ -1,1 +1,1 @@
-Offshot of #1618. Imported tickets didn't show in ticket list, not even when searched for *:*. This is similar to issues seen on production with tickets created by normal means, so Solr is suspected. I also saw ANQP on sandboxes, so it is also suspected. Both are known to looked at during last week, so need to start with just retesting.
+Offshot of [#1618]. Imported tickets didn't show in ticket list, not even when searched for *:*. This is similar to issues seen on production with tickets created by normal means, so Solr is suspected. I also saw ANQP on sandboxes, so it is also suspected. Both are known to looked at during last week, so need to start with just retesting.

Is indexing turned off for imported tickets? That would prevent them from being put in solr. Might be best to manually index them rather than relying on the ArtifactSessionExtension anyway.

Well, nothing special is done wrt to indexing with imported tickets - they are just created with normal model API calls, etc. And indexing worked w/o any issues when import was initial implemented.

Anyway, with so many refactors happening, touching this area, I decided to skip doing adhoc manual tests and write unittests instead. And what I found that even with existing ForgeTracker testsuite, we don't have a unittest for tracker ticket list page (aka tracker front page). Or rather, we have it, but it's commented out (test_home). When uncommented, it fails. When same session flush/monqtask run logic applied to it as test_search, it still fails. I finally traced this to MockSOLR being very limited in its capabilities (and silently return empty list in case it doesn't understand query). So, I'm working first on elaborating MockSOLR to be able to execute ticket list query. Then we'll have reproducibly testable answer to that question - "Is indexing turned off for imported tickets?"

Ok, there's now functional test for import, and also integration test (requires manual setup). The bad news is that importing 1000 tickets takes an hour (56min), and then yet ~20mins for all of imported tickets to be indexed by SOLR and available in ticket list. It's not immediately clear how to handle this - moving actual import processing into tasklet leads to problem of reporting final import status for example. Apparently, there's just to much processing for such bulk operation as importing (like sending out email which is a problem on its own), and some (or as much as possible) of this processing should be turned off during import, and scheduled to happen en masse later.

Ok, there're now functional and integration tests for import, and the core works well. But they uncovered/confirmed infrastructural problem with handling big imports - nginx/apache/some other request proxy times out for long processing of big ticket set. Issues is ticketed and solutions proposed as [#1861].

• status: open --> code-review
• assigned_to: Paul Sokolovsky --> Dave Brondsema

• status: code-review --> closed

• milestone: apr-14 --> apr-7