Apache Allura™

• labels: owasp --> owasp, ux
• milestone: forge-backlog --> forge-mar-22

• labels: owasp, ux --> owasp, ux, for-support

• Description has changed:

Diff:

--- old
+++ new
@@ -7,3 +7,4 @@
 * change `<hr>` for something better
 * add some metadata on the title line, like last modified date
 * if you choose to search history, the results links need "?version=N"
+* change sort order (relevance, date, etc)

• Description has changed:

Diff:

--- old
+++ new
@@ -7,4 +7,5 @@
 * change `<hr>` for something better
 * add some metadata on the title line, like last modified date
 * if you choose to search history, the results links need "?version=N"
-* change sort order (relevance, date, etc)
+* change sort order (relevance, date, etc) and direction
+* add search syntax help and examples (like we do for tickets)

Originally by: wellread1

It seems that in the last 1.5 years there has been no action to address the known lack of ability sort answer sets chronologically.

Please take a look at the KeePass forum https://sourceforge.net/p/keepass/discussion/. The forum software was updated in Nov 2012 and searching has been nearly useless since. Old, irrelevant hits are mixed in and listed high on the search results.

Try a quick search, you will find many hits on the first results page that are over 5 years old and pertain to obsolete versions of KeePass.

Originally by: wellread1

Thank you for describing the workaround (via allura-users-help) that allows a user to add a date filter to a search, e.g. the following search will find posts mentioning "install" and posted since Feb 1 2013:

install AND mod_date_dt:[2013-02-01T00:00:00Z TO *]

• Description has changed:

Diff:

--- old
+++ new
@@ -1,8 +1,9 @@
 Specifically the wiki search results should look better, but let's see how much improvement we can get through the generic search result template.  E.g. https://sourceforge.net/p/allura/wiki/search/?q=faq  A few suggestions

-* only include primary artifacts (no comments)
-* artifact type (WikiPage) doesn't need to be shown in that case
+* an option to only include primary artifacts (no comments)
+* artifact type (WikiPage) doesn't need to be shown in that case - best to stop including that in the title field when indexing?
 * exclude deleted pages
+* better relevance - e.g. a title match should be better than a body text match.  I also think we are matching across too many fields in the first place.  E.g. a search for "forgewiki" or "WikiPage" or the mount point shouldn't match the "internal" fields in the solr document.
 * show a preview of the content
 * change `<hr>` for something better
 * add some metadata on the title line, like last modified date

• Description has changed:

Diff:

--- old
+++ new
@@ -10,3 +10,5 @@
 * if you choose to search history, the results links need "?version=N"
 * change sort order (relevance, date, etc) and direction
 * add search syntax help and examples (like we do for tickets)
+
+Here's a mockup of how https://sourceforge.net/p/allura/wiki/search/?q=faq could look: http://screencast.com/t/06dC4NjT60

• labels: owasp, ux, for-support --> owasp, ux, for-support, 42cc

• status: open --> in-progress

Created:

• [#2835] Search: re-style generic search template (3cp)
• [#2835] Search: improve wiki search (3cp)
• [#2835] Search: sort results (2cp)
• [#2835] Search: content preview and highlight matches (3cp)
• [#2835] Search: wiki syntax help and examples (1cp)
• [#2835] Search: check that other apps working with new generic search template (2cp)

If something would require solr upgrade, I'll create separate ticket for that.

Closed #287 (Search: re-style generic search template). allura:je/42cc_2835

I'll be trying to review these quickly since there are a lot of parts for this ticket, hopefully will make it all go smoother.

The template changes look good. There are placeholders so I can't merge it yet. If we get to a point where some parts of this ticket are completely done, I'll merge them before the rest.

One more thing: can we make this template apply to the "search whole project" page too? E.g. /p/testproject/search?q=test&history=False

If we get to a point where some parts of this ticket are completely done, I'll merge them before the rest.

Ok, I'll remove placeholders when closing a next ticket.

One more thing: can we make this template apply to the "search whole project" page too? E.g. /p/testproject/search?q=test&history=False

Yes, we can. I'll create another ticket for that if you don't mind. I think it'll require some cosmetic work (handling paging, new search options, etc)

• milestone: forge-mar-22 --> forge-backlog

Closed #288 (Search: improve wiki search). allura:je/42cc_2835

For wiki search I've changed solr parser to 'dismax'. It better for user-generated queries, it matches on multiple fields, with some smart analyzers and tokenizers of user input. Standard parser matches only on text (that's why there was code to append all other fields to it). I've made it match on title and text fields, seems like this is the only fields we should match on things like wiki pages and comments.

Also, I've made some changes in the indexation of wiki pages. I've added the title field as a copy of title_s field that already presented in the index (my first try was to match on title_s field, but it turned out that solr doesn't apply all the fancy analyzers and tokenizers to it, so matches were not always relevant with this approach).

I've left indexation of tickets as it was (appending all the field into text), 'cause new indexation breaks some searches with standard solr parser.

I think we should change other apps' searches to this approach (use 'dismax' parser).
However, ticket's search seems to be a special case. It uses ability to match on specific fields a lot ('status:open and labels:42cc' and queries like that). With 'dismax' parser we're losing this ability, so ticket's search should be separate (actually, it is now).

I've removed placeholders, so you can merge it to master. Wiki now uses a better search. All other apps use a new template, but with old solr parser.

Also, if you don't like how artifact type is placed next to title in search results, you can revert commit 332a696 :)

• dismax seems to be working pretty well. Can you index the labels too? Those might not be too common, but will be generally useful for all artifacts. For wikis in specific, when labels are used, that wiki page makes a link to e.g. wiki/search/?q=labels_t:foo which no longer works. Maybe we want to have a url param to make it use the standard solr parser? Then that sort of field-specific query could still work. (After we do a solr upgrade, Extended DisMax sounds great, but we're not there yet)
• the help button is still a placeholder
• in solarize(), hard-coding a type check for 'ticket' is a little ugly. Could the Ticket.index() method build the 'text' key itself? Or, another option would be to have an artifact attribute called something like index_all_fields and it Artifact sets it to False and Ticket sets it to True. Then solarize() would check for that instead.
• Copying title_s seems a little odd, but if you say it's necessary I'll take your word for it :) Can you add a comment in the main Artifact.index method explaining why it has to be duplicated?

After those are addressed, I think I'll be ready to merge the current progress.

Additional things going forward:

• besides the issue with searching the label field specifically, using dismax for all tools' search (not tracker) sounds fine to me. Seems like refactoring the wiki search controller into some sort of shared helper method would be nice.
• comment links should go to the nice version of the post (e.g. https://sourceforge.net/p/allura/wiki/Allura%20Wiki/#323a instead of https://sourceforge.net/p/allura/wiki/_discuss/thread/f0ec390d/323a/). There should be a method to get this URL, it works for forum posts in search results already.

Can you index the labels too? Those might not be too common, but will be generally useful for all artifacts. For wikis in specific, when labels are used, that wiki page makes a link to e.g. wiki/search/?q=labels_t:foo which no longer works. Maybe we want to have a url param to make it use the standard solr parser?

I think we can just add a url param to use standard parser. But how it would be accessible from UI? Maybe another group of radio buttons (like for project/app search) or something?

Copying title_s seems a little odd, but if you say it's necessary I'll take your word for it :)

I've been thinking of it too, and now it's seems to me that it's not necessary. I think I'll try to get rid of title_s for all artifacts (maybe except for ticket, since we're not changing them)

I'll create a new ticket on our side to deal with issues that prevents you from merging the current progress.

besides the issue with searching the label field specifically, using dismax for all tools' search (not tracker) sounds fine to me. Seems like refactoring the wiki search controller into some sort of shared helper method would be nice.

Yep, I've been thinking about this approach too. I think we'll deal with this in #292 (Search: check that other apps working with new generic search template) after all tweaks for wiki work fine.

comment links should go to the nice version of the post

For wiki pages it's a little trickier than for forums. I think it would be a new ticket too.

Created:

• #303: [#2835] Search: fix issues that prevents from merging current progress to master (2cp)
• #304: [#2835] Search: 'nice' comments links in search results (1cp)

I'd only use the standard parser for the label-specific search link for now. We may want an option for it on the main search form at some point, but lets keep it simple for now and not put it there yet.

Ok. We'll add a url parameter to use standard parser, without any UI for it.

Thanks for such immediate response :)

Closed #303. je/42cc_2835

I guess, you should be able to merge current progress to master now :)

Closed #290 (Search: content preview and highlight matches)

This requires changes in solr config (text field should be stored). See diff for solr_config/

branch: je/42cc_2835_content_preview

Closed #289 (Search: sort results)

77b869d..112fb4a t289_sort_search -> je/42cc_2835_content_preview

• qa: Dave Brondsema