Ticket search results

Figure out how to deal with big tracker imports

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:54 -0000

When more or less big (<100 tickets) tracker import is run against sandbox (with nginx frontend) it ends up with:

$ time python allura_import_test.py data/sf100.json -u https://sf-psokolovsky-3014.sb.sf.net -a tckd18490839ab2c834b16b -s 7c43a517b310ab7477f251922282456fa184c6440f2769c76fecfece04b1a58125063030b166d4e8
Importing 100 tickets
Traceback (most recent call last):
  File "allura_import_test.py", line 93, in 
    res = cli.call(url, doc=doc_txt, options=json.dumps(import_options))
  File "allura_import_test.py", line 64, in call
    raise e
urllib2.HTTPError: HTTP Error 504: Gateway Time-out

real	0m31.727s
user	0m0.300s
sys	0m0.048s

I.e. nginx or some other intermediate component times out in 30s. So, the current approach of providing import data in ForgePlucker format as one big JSON and then executing import synchronously (to return status to the client) is not viable and needs replacement/augmentation. Following choices can be proposed: 1. Instead of submitting JSON with multiple tickets, provide API call to import ticket one by one (this was one of solutions originally proposed). We still can use ForgePlucker as interchange format, it will be just parsed by import script, instead of within Allura. 2. Go into other direction, and make import feature be UI functionality, where user uploads JSON to a web page, and then uses it to track progress/get final status. 3. API-esque compromise between 1 & 2: there's one call to post import job (import id is returned), it is then processed as async task, and there's another API call to check status of import id. 4. Regardless of approach chosen, there should be also optimization on how imported tickets are created - they should just go to Mongo, and as little as possible post- and side-processing performed. The only thing which should be done on them is SOLR indexing, and even for that, it probably makes sense to just queue up complete tracker re-index after import is complete (unless choice 1 is used). Choice 1 is by far the easiest to implement - it would just require what #1767 already queued up, just need to split chunks to be of size 1.

Project shard mongo indexes are not created by standard setup-app

Paul Sokolovsky — Thu, 23 Jun 2011 18:48:53 -0000

Looking at both localdev and sandboxes, project shards' mongo indexes are not created with the standard setup. There's command "paster ensure_index" to create them, but that is not described in Allura docs, not apparently being run on sandbox setup. To reproduce:

[root@psokolovsky-3024 ~]# mongo
> use project-data-t                      
switched to db project-data-t
> db.system.indexes.find()                       
{ "name" : "_id_", "ns" : "project-data-t.config", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.discussion", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.artifact_link", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.page_history", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.artifact_feed", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.wiki-globals", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.page", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.thread", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.globals", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.bin", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.forum", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.portal_config", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "project-data-t.git-repository", "key" : { "_id" : 1 } }

The real fix here apparently should be to run ensure_index as part of standard "paster setup-app" sequence.

Tweak sample git clone command for multi-repo cases

Paul Sokolovsky — Sun, 22 Feb 2015 15:23:21 -0000

Recently we got complaint/suggestion from TG project about sample git clone commands we provide. We suggest form "git clone ", but that doesn't work well for project with multiple repos, for example TG gets: ~~~~ git clone git://git.code.sf.net/p/turbogears2/tg2 turbogears2 git clone git://git.code.sf.net/p/turbogears2/tg2devtools turbogears2 git clone git://git.code.sf.net/p/turbogears2/tg2docs turbogears2 ~~~~ As a stop-gap measure, Mark suggested using just "git clone " if there're >1 repos in a project. That should work well until we settle how to make sure that name for even one repo is selected by users well (i.e. we guide users to select what they want, instead of current default "code").

Revamp tracker import for TG project migration

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:57 -0000

From Mark's mail: I know this is interrupt driven work, but if you could take a look at this issue and let me know what you find, when trying to import this JSON dump to allura, that would be great. I'd really like to have a major open source project come onto newforge, and I think TG is a good first candidate, as it's a reasonably active open source project where we have good ties already, so it should give us good feedback about the platoform. Also, just so everybody knows if anybody else expresses interest in migrating, I think it's worth our while right now to provide them with some help. --Mark Ramm ---------- Forwarded message ---------- From: Mark Ramm Date: Tue, Mar 1, 2011 at 6:21 PM Subject: Fwd: [tg-trunk] Trac Migration Status Update ---------- Forwarded message ---------- From: Christoph Zwerschke Date: Sat, Feb 26, 2011 at 9:10 PM Subject: [tg-trunk] Trac Migration Status Update To: TurboGears Trunk After the SF hosted trac option turned out to be unusable for our purposes (it's only trac 11.2, no plugins installable, no API or database access etc.), I tried the second (and preferred) option for hosting the TG tracker on SF today, the Allura tracker. The following scripts already exist to export from Trac as JSON and then import JSON to Allura: http://sourceforge.net/p/allura/git/ci/e1c0a20357afc3af25f470cc3488ad3d5127c352/tree/migrate-3rdparty/ It turned out that the trac export script does not work with trac 0.10 which we are running on our old server, so as a first step I have migrated it to trac 0.12 on my own server. There were several more small problems with the export script which I could fix easily. The script also stumbled over a oversized ticket containing only Chinese spam (#852) which I have deleted already. So I have now a ~10MB JSON file with our trac database. Btw, there are ~1905 tickets for TG1 and ~540 tickets for TG2. When I tried to import that to Allura, I got an "Request Entity Too Large" error - obviously because the import script tries to import everything with one API call only. So I modified the import script to read the data in chunks (slicing the artifacts list), but now I only get internal server errors with no further clue what's wrong. The error message says more info may be available in the server error log, but only SF staff has access to that log. Another problem that needs to be solved if we go this way, is the user mapping from our old Trac accounts to SF accounts. The import script has a usermap option, and maybe we will be able to map some users through their email addresses, but this is also something that can be only done by SF staff. @Mark: Any chance you will be able to solve these problems in the short run? If not, I suggest we migrate to Florent's server for the time being. However, since Trac cannot work with remote repositories, this also means we will have to move the repositories there. We can consider moving to SF at a later time again. -- Christoph

Discussion Thread moderation is broken

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:55 -0000

This came up with [#1546]. There, project somehow managed to have Announcement flag set on a thread, which triggered error on display. However, further investigation showed that it doesn't really seem to be possible to set such a flag on a thread in current UI, because of: 1. Bad UI layout of thread actions (see screenshot) https://sourceforge.net/p/allura/discussion/general/thread/8dcaf833/ 1. Wrongly tested condition (always false) for checking if thread moderation action should be shown 1. Broken thread moderation widget which causes exception in EW on render

Fix UndefinedError: 'MagicalC' object has no attribute 'announcements_table'

Dave Brondsema — Thu, 20 Aug 2015 22:07:53 -0000

https://sourceforge.net/p/lanex/discussion/ ~~~~ Module /var/local/allura/ForgeDiscussion/forgediscussion/templates/discussionforums/index.html:22 in block "content" << {% if announcements %}

Announcements

{{c.announcements_table.display(value=announcements)}} {% endif %} {% set visible=0 %} >> {{c.announcements_table.display(value=announcements)}} Module jinja2.environment:352 in getattr << """ try: return getattr(obj, attribute) except AttributeError: pass >> return getattr(obj, attribute) UndefinedError: 'MagicalC' object has no attribute 'announcements_table' ~~~~

fix UndefinedError: 'strftime' is undefined

Dave Brondsema — Thu, 20 Aug 2015 22:07:54 -0000

e.g. http://sourceforge.net/p/c-3/svn/18/tree/ ~~~~ Module /var/local/allura/Allura/allura/templates/widgets/repo/tree_widget.html:33 in top-level template code << {{lib.abbr_date(dirent.last_commit.date)}} {% if dirent.last_commit.author_url %} >> {{lib.abbr_date(dirent.last_commit.date)}} Module /var/local/allura/Allura/allura/templates/jinja_master/lib.html:59 in template << {% macro abbr_date(start_time) %} {{h.ago(start_time)}} {% endmacro %} >> {{h.ago(start_time)}} UndefinedError: 'strftime' is undefined ~~~~

Provide admin action to refresh SCM repo

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:55 -0000

I really hope that this won't be required too often in production, but when it will be needed, it will save us unsatisfied user and/or support request. And it's for sure useful for development and debugging. "It" is ability to refresh a repo from web UI. Typical usecase during debugging: create a repo, then go to FS and replace empty created repo with one containing content useful for testing, then hit "refresh" action, and be able to see it in web UI. So, I at least add SCM Admin action, /p/test/admin/git/refresh, and ideally that action should be available in Admin actions list for SCM repo.

Add branch name(s) to commit titlebar (git)

Jenny Steele — Thu, 20 Aug 2015 22:07:55 -0000

This was broken out from [#1385] the bar showing a branch should contain the branch name, e.g., 3477a1c (master, dev)

Upgrade httplib2 to 0.6.0

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:56 -0000

httplib2 0.4.0 is old (2007) and throws warnings in py2.6. Latest is 0.6.0, I reviewed changelog (http://code.google.com/p/httplib2/source/browse/CHANGELOG), no incompatible changes appear to be made, submitted https://control.sog.geek.net/sog/trac/ticket/17979 , once that's done, upgrade and run tests.

Improve html5 validation performance

Paul Sokolovsky — Wed, 23 Mar 2011 14:41:38 -0000

Rick reported that adding online html5 validation increased testsuite runtime from 15 to 40 mins. There need to be more fine-grained control over which validations are to be run. Also, it would be nice to find alternative to online html5 validation.

Regenerate all thumnails for attachments

Rick Copeland — Thu, 20 Aug 2015 22:07:55 -0000

In our shared sharded database refactoring we lost the attachment thumnails. Regenerate them.

Fix validation errors across all apps

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:57 -0000

Now that #1233 is done and html5 theme fixes are backported, fix remaining validation errors across all apps, to be prepared to enable validation failures for tests.

Achieve 90% test coverage on Allura project

Rick Copeland — Thu, 20 Aug 2015 22:07:55 -0000

Cleanup Allura's BaseController

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:56 -0000

From https://engr.geek.net/hudson/job/allura/414/testReport/junit/alluratest/test_syntax/test_pyflakes/ : pyflakes failure -------------------- >> begin captured stdout << --------------------- Allura/allura/lib/base.py:47: undefined name 'OEmbedController' Allura/allura/lib/base.py:50: undefined name 'HostNeighborhoodController' That's BaseController referring to its subclasses.

Factor out test support code to separate package

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:53 -0000

From mail: 2. So, I'd like to propose to separate test support functionality into own top-level package, and consolidate all test related functionality under it (e.g., move twill-tests in there). 3. Do this in such way as to look forward to separate this into package reusable across projects (for example, current validation tests in allura is concurrent fork of sfpy's tests).

HTML vs XHTML issues with output pages

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:55 -0000

It turns out that we brand our pages as XHTML (see jinja_master/master.html), while send them as html mime type still. This is known to cause confusion and issues: * http://hixie.ch/advocacy/xhtml * http://webkit.org/blog/68/understanding-html-xml-and-xhtml/ We kinda already had confusion with <div/>, this issue was preceeded by http://bitbucket.org/rick446/easywidgets/issue/6/element-rendered-unclosed-for-jinja , so it's time to clear it up. As Rick puts it:

EasyWidgets widgets are supposed to specify their template engine + the template itself, so it's not really an EasyWidgets issue. Allura isn't supposed to be serving XHTML, not sure how that got missed in the jinja conversion. We should serve either html4 or html5, with my preference for html 5 mainly due to the short doctype (). Also note that allura pages are served with content-type of text/html and are being rendered as html, it seems (based on behavior when seeing
in the content). Also see http://hixie.ch/advocacy/xhtml . Please go ahead and ticket up a change of the master template to html4 or html5, as you feel most comfortable.

My guess is that we should do the same as sfpy/dist do, and they use HTML5 IIRC. I'm going to check that up.

Should not force admins to monitor new artifacts

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:55 -0000

Recently, there was introduced feature to make tool admins to monitor entire tool application, i.e. all its artifacts, including newly created. However, there's still remnant code at tracker_main.py(save_ticket):479 which explicitly subscribes admins to artifact on its each update. Such calls are ignored if admins already monitor entire app. However, if they choose not to, such a call would obviously unconditionally make them monitor each artifact. Such behavior probably should be removed, to all admins to decide themselves what they want to monitor: either entire app and thus all all tickets, or none at all (or just specific artifacts which they can subscribe to manually).

Need 3rd UI status of artifact monitoring

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:57 -0000

We currently have support for monitoring indvidual artifacts and an entire app (i.e. all its artifacts implicitly). Let's now think how "Subscription" indicator (envelope) should work: 1. If user is not subscribed to entire app, and not subscribed to ticket, the envelop should show "unsubscribed" and clicking should execute subscribe action. 2. If user is not subscribed to entire app, but subscribed to ticket, the envelop should show "subscribed" and clicking should execute unsubscribe action. 3. But if user subscribed to entire app, one cannot unsubscribe from just one artifact. The indicator should show some "implicitly subscribed" state and be unclickable. Failing to capture these desticntions is partly cause of test failure in https://engr.geek.net/hudson/job/allura/358

For sandbox setup, make admin1 a test project admin, and also user neighborhood admin (well, that means site superuser)

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:54 -0000

Currently, it's not possible to do any project admin action, nor neighborhood admin action, on a pristine sandbox, so it takes extra setup to show off any related functionality. Proposal: 1. make admin1 a test project admin 1. make admin1 an admin for User neighborhood (that's required for #1128)

Revamp testsuite running on sandboxes, again

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:55 -0000

After all changes to mongo storage model, testsuite doesn't run on sandboxes again.

Add UI to generate special API keys

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:56 -0000

Follow up to #1125. A special key is a key with attached adhoc capability token to access API which otherwise is not available for general use. User story is something like: Site admin (and only site admin) should be able to go to a special page, where one can enter username for whom key is generated, expiration date for a key (having sensible default like today + 1 day or 7 days), capability name as a string, and optional capability discriminator as a string. After submitting this information, site admin is presented with a key value, which is passed to intended user via separate means. There also should be page to present all currently active (non-expired) special keys.

Add to API keys support for expiration and special capabilities

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:55 -0000

Subtask of #1029. To support bulk import and other high-resource/sensitive operations, it makes sense to add to API keys expiration support as well as ability to attach extra "capabilities" data. Usecase: give user X one day to perform an import to project Y. As API keys are user-bond, capabilities apparently should be a dictionary, to allow capabilities discrimination ("import" capability, but only for project Y).

Improve inline markdown help

Paul Sokolovsky — Thu, 20 Aug 2015 22:07:57 -0000

https://sourceforge.net/p/allura/tickets/markdown_syntax could use more examples.

Create user preferences plugin API

Rick Copeland — Thu, 20 Aug 2015 22:07:54 -0000

We should use a plugin api to allow the various user preferences (display name, page size, api key, etc) to be stored either in mongo (default plugin) or alexandria (classic plugin), with the live site using alexandria. The implementation of the classic plugin will be another ticket. See also [#1398]