Apache Allura™

[#1510] (wont-fix) is related. It proposes a UI for filtering tickets. It would also be realtime rather than be delayed like solr is, after tickets are updated.

• milestone: forge-backlog --> someday

ML thread discussing it http://mail-archives.apache.org/mod_mbox/incubator-allura-dev/201311.mbox/%3C52790763.8070109%40brondsema.net%3E

Probably should post here as implementation proceeds, with how you think it'll work out, since there are some open questions like: can we generate the solr query in the client side, will we have one solr field that has to get re-parsed to set the filter state, or separate q & fq params.

• Labels: --> 42cc
• Status: open --> in-progress
• Milestone: someday --> forge-backlog

If it's really only issue of open/closed for now, than it cna be as simple as a single dropbox with values like "All", "Open", "Closed" (the latter two would add to filter all statuses from open_status_names/closed_status_names) and create query on server side (still need to figure out how to parse query or user fq field).

On the other hand, if there's need for more granular filtering (e.g. include tickets with specific set of statuses) that it'd require more thought.

We want more then just open/closed, and more than just the status column. We should have a filter dialog for all the standard columns that are limited choice: milestone, status, owner, creator. Other fields (labels and custom fields) can be added in a phase 2 I think.

Solr has facet support, and perhaps that should be used to determine all the valid choices for a column. Then it'll dynamically reflect the current choices.

One difficulty will be that some views (e.g. milestone) use a mongo query, so adding a filter will have to switch over to a solr query. This is done in some places already; there's some helper functions to change the field names.

Attached is an example of what Google Spreadsheets has. Something along those lines is what I'm thinking, but doesn't have to be as nice as that necessarily.

• status: in-progress --> code-review

Closed #480, #481. je/42cc_4019

Look at what we have so far. I think it may require some polishing, but basically it works.

I took a fairly brief look at it. Seems like its on a good track. I haven't looked enough at the architecture yet to have any comments on that area. I do have a question: if this is using solr facets, will that make it challenging later to extend this to custom fields? And are we compounding complexity by adding filtering to both solr & mongo views? Mongo views were to avoid solr indexing delays, but at some point it might make sense to simplify and drive it all from solr.

Few things I noticed so far: after you filter down on a facet, you don't have any way to change the values for that facet (or clear them). Also when on a milestone view, the facet choices for status aren't correct (at least on my test project).

The UI could use some polish, but we can get to that a bit later.

if this is using solr facets, will that make it challenging later to extend this to custom fields?

I think it should be pretty straightforward, since Allura adds custom fields to solr index already, and we can easily make facets queries for them.

And are we compounding complexity by adding filtering to both solr & mongo views?

A little bit. And it's somewhat obscure for users, I think. If you enable a filter in such a view, some tickets, which match that filter, but not indexed yet, will disappear.

Few things I noticed so far: after you filter down on a facet, you don't have any way to change the values for that facet (or clear them).

If you uncheck a checkbox for corresponding facet and re-submit, then you will get back all the other values in options.

This is because the way we query solr (one query to both filter and get facets counts).

I think it would be better if we update filtering results via ajax whenever user changes filter, but it more work and maybe we can get to that later.

Also when on a milestone view, the facet choices for status aren't correct (at least on my test project).

Hmm, could you specify how exactly? Are you sure that all tickets have been indexed?

Example of issue with status filtering on a milestone view: http://sf-dbrondsema-1015.sb.sf.net/p/testit/tickets-gfarm/milestone/gfarm-2.5.8-rc1/ has wrong status choices. Equivalent search view (all indexed) http://sf-dbrondsema-1015.sb.sf.net/p/testit/tickets-gfarm/search?q=_milestone:gfarm-2.5.8-rc1 has correct status choices.

I see. Looking forward to hear further feedback from you, then I'll create tickets to improve all concerns.

For UI polish, I think we should make it fairly similar to the earlier image attachment, like:

• expand the list of choices
• center it near the column
• probably make it a floating dialog, so it can be narrower (to help with centering) and taller (to see all the expanded choices)
• icon on column header, so users know clicking will offer some options. This unicode char looks good to me and widely supported on desktop browsers: http://unicode.johnholtripley.co.uk/25BE/ We may want a different icon or modified icon to put on columns that have current filters.

In the code:

• In search_artifact the fields & values in fq.append should be quoted and/or escaped. [c70e94f] was a similar fix, but a broader solution might be needed
• The get_facets method isn't clear to me what's going on right away. I came across the test case for it later and it is good and does help a lot. Perhaps I might add a comment like # drop "_s" to the name[:-2] line, but not a big deal.
• in ticket_search_results.html avoid <div style="clear:both;"></div> if you can help it. I know its around in many places, but its old and not good practice.
• having {{filter|safe}} in an html file makes me nervous. It seems that is set as filter=json.dumps(filter) in paged_search which is called via controllers and widgets, so it is actually safe currently, but those two lines are far apart and if in the future anything changed in between them that could insert dangerous JS, that'd be a security hole. I think {{h.json.dumps(filter)|safe in the html file would work, and that'd be much better IMO.
  • However, that's not the end of the story. Even JSON isn't actually safe, see http://jsfiddle.net/kEkQ3/ for example. The browser finds tags and parses that into the DOM before parsing the JS. We need a helper function to change < to \u003C or \x3C e.g. http://jsfiddle.net/kEkQ3/1/ if we want to put user-provided data into a JS var.

Excellent work on adding the right entries in LICENSE and rat-excludes.txt

• status: code-review --> in-progress
• QA: Dave Brondsema

• status: in-progress --> code-review

Force-pushed je/42cc_4019

Please add multiselect to ForgeTracker/LICENSE in addition to top-level LICENSE. (We're trying to keep LICENSE files within in package accurate so that we can have separate releases for them some day if we want).

The filter UI is looking very good. I like the use of green to indicate when a filter is in use. It might be a little too subtle for some people, but I like it (maybe make the whole field name green?). There's just a bit of unnecessary text in drop-down layer that I'd like to remove. Can you remove/hide the "Filter by" button thing (button.ui-multiselect) and the close button (.ui-multiselect-close)?

Is it possible to have a filter choice for "blank" or "not set"? For example, on the Owner column, this would be for tickets that aren't assigned to anyone. It can also happen to the milestone field which can be None after an import.

I'm still seeing some odd behavior on a milestone view. The filter choices for status/owner/milestone seem to include all tickets and not those shown for the selected milestone. Example: https://sf-dbrondsema-1015.sb.sf.net/p/project1/test-tracker/milestone/Group%205/

escape_solr_arg should use something like http://docs.python.org/2/library/stdtypes.html#str.translate instead of doing character-by-character looping, checking, and joining in python. And test_escape_solr_arg should include a test case that has a backslash in it, since that has special handling in escape_solr_arg.

This is going to be really nice :D

• status: code-review --> in-progress

• status: in-progress --> code-review

Closed #558, #559, #560. Updated je/42cc_4019

Couple of notes:

1. str.translate wouldn't work because it maps one char to one char
2. matches count for "Not set" option seems like imposible to calculate
3. "Not set" option is always present in options, because of (2)

Looking pretty good, I think we're very close. One bug I found is that the facet choices seem to be only from the values in the current 'page' of tickets (although the facet counts do come from the whole result set). I noticed this on a large milestone that had several pages of tickets, but you can force the issue by setting ?limit=2 for example in the URL.

• status: code-review --> in-progress

• status: in-progress --> code-review

Closed #579. Force-pushed je/42cc_4019 (rebase on fresh master)

• status: code-review --> closed
• Milestone: forge-backlog --> forge-may-2