https://forge-allura.apache.org/p/allura/tickets/You searched for labels:performance AND status:openenThu, 17 Mar 2016 15:18:04 -0000https://forge-allura.apache.org/p/allura/tickets/8068/The `ensure_index` paster command misses some indexes: `CommitRunDoc` which uses the `repo_commitrun` collection doesn't get its indexes created, because there is no `mapper(...)` line for it at the bottom of `allura/Allura/allura/model/repository.py`. (There is no class to map it to currently) Also the activitystream indexes on `activities` and `nodes` don'te get created, despite code in `EnsureIndexCommand` that uses `activity_odm_session` to try to do so. Must be a bug in it.Dave BrondsemaThu, 17 Mar 2016 15:18:04 -0000https://forge-allura.apache.org/p/allura/tickets/8068/https://forge-allura.apache.org/p/allura/tickets/8019/Some repos can get a large number of cached branches or tags. The large lists cause a ming performance penalty: e.g. one request timing had 296ms in mongo, 2304ms in ming. This is particular true for the sidebar when it hits `repo.forks` property (twice), which loads many repo objects into memory. We could look into ways to not load those fields when not needed, or skip ming validation on them (see `validate_ranges` option of ming `Array`). Or just querying for a count and not getting the objects at all. Or cap the size of the lists. From forge-allura's own mongo, look at the size of these cached branches lists: ~~~~ > db['git-repository'].find({}).forEach(function(u) { print(u.cached_branches && u.cached_branches.length, u.url_path) }); undefined /p/test/ undefined /u/masterbunnyfu/ undefined /u/shujisado/ undefined /p/allura/ 0 /u/swelltt/ undefined /u/mtem/ undefined /u/barki2015/ 0 /p/allura/ 0 /u/dnayak/ 24 /u/brondsem/ 776 /p/allura/ 547 /u/saintaardvark/ 725 /u/lxj/ 550 /u/testmr/ 550 /u/vansteenburgh/ 583 /u/alexluberg/ 543 /u/masterbunnyfu/ 643 /u/jetmind/ 649 /u/heiths/ 577 /u/shujisado/ 641 /u/mishravikas/ 695 /u/yesidid180/ 758 /u/jredd/ 776 /u/pranav/ ~~~~Dave BrondsemaWed, 24 Feb 2016 19:32:39 -0000https://forge-allura.apache.org/p/allura/tickets/8019/https://forge-allura.apache.org/p/allura/tickets/7961/Like [#7873] did for git, cache branch & tag info from Mercurial. Existing code from [#7873] probably could be re-used be refactoring it up into common repo code.Dave BrondsemaTue, 11 Aug 2015 20:34:49 -0000https://forge-allura.apache.org/p/allura/tickets/7961/https://forge-allura.apache.org/p/allura/tickets/7914/From [docs](https://forge-allura.apache.org/docs/getting_started/administration.html#create-allura-sitemap-py): > This takes a while to run on a prod-sized data set. There are a couple of things that would make it faster, if we need/want to. > > 1. Monkeypatch `forgetracker.model.ticket.Globals.bin_count` to skip the refresh (Solr search) and just return zero for everything, since we don’t need bin counts for the sitemap. > 2. Use multiprocessing to distribute the offsets to n subprocesses. Remove speedup suggestions from docs when this is ready.Igor BondarenkoThu, 09 Jul 2015 08:42:14 -0000https://forge-allura.apache.org/p/allura/tickets/7914/https://forge-allura.apache.org/p/allura/tickets/7188/Right now it takes a query to find out if a project even has direct subprojects, so this change would save a query in those cases.Tim Van SteenburghFri, 06 Mar 2015 20:25:30 -0000https://forge-allura.apache.org/p/allura/tickets/7188/https://forge-allura.apache.org/p/allura/tickets/7187/Subforums aren't fully functional. Seems like you can only add them via Admin's 'add' form and not via the other 'add' form from main sidebar. And then there's no way to view subforums (except admin link) nor post a new topic to the subforum. Only can edit an existing post to move it to a subforum. We should remove them if we're not going to support/maintain them properly. Forum.subforums also runs an unindexed query, scanning all forum documents. Its called via a chain of forum widgets.Dave BrondsemaFri, 21 Feb 2014 17:01:25 -0000https://forge-allura.apache.org/p/allura/tickets/7187/https://forge-allura.apache.org/p/allura/tickets/6802/If a Last Commit Doc is not present for a tree view, it may take a while to generate that doc, so a better user experience would be to serve up the essential information immediately and use AJAX to fetch the last commit info. Do we want this to be configurable per SCM type? I.e. svn's last_commit_id might be fast enough all the time to not need to use ajax?Dave BrondsemaTue, 31 Mar 2015 15:26:32 -0000https://forge-allura.apache.org/p/allura/tickets/6802/https://forge-allura.apache.org/p/allura/tickets/6707/Sometimes `RepoRootController.commit_browser_data` is pretty slow. Based on log statements, it looks like the slowness is in the first 2 lines of the method. This is supported by timermiddleware instrumentation which has high timings for `svn_lib.log` and `git_tool._iter_commits_with_refs.next` and `git_tool.log.next` We could consider a max time limit similar to [#6695]Dave BrondsemaThu, 21 Aug 2014 05:08:39 -0000https://forge-allura.apache.org/p/allura/tickets/6707/https://forge-allura.apache.org/p/allura/tickets/6390/Files like project icons and screenshots are simple files, yet go through the whole Allura stack to be served up. When there are a lot of requests for these, it would be much better to serve them directly as static files via nginx or apache. Other allura files such as attachments are affected by this too, but they don't get as much traffic normally. The tricky part is that permission checks are necessary, e.g. if a project or neighborhood is not publicly accessible. We should make Allura `File`s have an alternate backend for a regular filesystem instead of mongo/gridfs. This should be controlled by deployers (e.g. in the .ini file) on a per-collection basis, to give lots of flexibility to allure site administrators. To handle non-public permissions, we could store the files in two directory trees, one for public files and one for private files. Any file that is not public (`*anonymous` read permission) should be stored in the private directory tree. Public files can be served by the webserver directly. The webserver could be configured so that any requests that 404 (file not in public tree) get proxied through to the Allura app so that permission checks are used. An alternate option would be to configure the webserver to use custom authentication via Allura, and then serve all files directly (public or not). An nginx module for that is at http://mdounin.ru/hg/ngx_http_auth_request_module/file/a29d74804ff1/README Allura code for referencing file URLs (icons, screenshots, for starters) will have to be modified to support using the different URL paths that go to the webserver directly. When project permissions change, there needs to be a mechanism to handle to moving icons from one location to another if their 'public' status changes. Use event handling could work well. (Not needed if custom webserver auth is used). We'll need a script to convert existing files from GridFS to the filesystem. Potentially a two-way script, so somebody could go the other direction too (but doesn't seem to be an immediate need). And would be most useful as a two-pass script: 1) move files, 2) delete files (so #2 can be run after everything is complete). SF has backup/delete/restore scripts, which will need to handle these files.Dave BrondsemaWed, 24 Dec 2014 22:50:01 -0000https://forge-allura.apache.org/p/allura/tickets/6390/https://forge-allura.apache.org/p/allura/tickets/5986/Forking allura on sf.net took about 18 minutes and a lot of that time was in the "cloning" phase.Dave BrondsemaMon, 06 Apr 2015 15:01:15 -0000https://forge-allura.apache.org/p/allura/tickets/5986/https://forge-allura.apache.org/p/allura/tickets/5866/Per [#5112], to improve milestone count performance, we should limit the sidebar to only show the X most recent milestones w/ count, and add some sort of UI to list more / the rest.Cory JohnsWed, 25 Feb 2015 13:41:11 -0000https://forge-allura.apache.org/p/allura/tickets/5866/https://forge-allura.apache.org/p/allura/tickets/5865/Per [#5112], the permission checks on milestone counts is too expensive. Change it so that, given a cutoff (e.g., 500), if there are more than that threshold, skip the permissions check and round to the nearest hundred (or multiple of the threshold, or some other reasonable rounding) and display as "> x00" where "x00" is whatever the rounded value is.Cory JohnsTue, 24 Mar 2015 19:43:25 -0000https://forge-allura.apache.org/p/allura/tickets/5865/https://forge-allura.apache.org/p/allura/tickets/5623/[forge:site-support:#2252] >**Branches and Tags** Visually they are confined by the layout rectangles on the left hand side of the layout. They occupy 3x more vertical space than in hgweb. Horizontally the space is so limited it can't even display properly the name of a branch. There is some number highlighted on each branch, its importance escapes me. **I am missing the most important information: last modified date.** And then also who modified it. Could definitely use some UI improvement, especially when there are a lot of branches/tags, with long names.Chris TsaiMon, 19 Jan 2015 15:43:34 -0000https://forge-allura.apache.org/p/allura/tickets/5623/https://forge-allura.apache.org/p/allura/tickets/5098/Commit hooks should send changed revisions to repo_refresh, and repo_refresh should minimize its work by just indexing the new revisions. Just an idea so far, we don't have measurements to know how significant of an improvement this would be.Dave BrondsemaTue, 06 Jan 2015 04:53:04 -0000https://forge-allura.apache.org/p/allura/tickets/5098/https://forge-allura.apache.org/p/allura/tickets/5095/It'd be helpful if project icons and user icons could be served through a CDN url, as specified in production.ini. However, we need to make sure that changes are reflected live in short order. The project icon has a `{{_id.generation_time}}` url param to ensure that happens locally; that should continue to work.Dave BrondsemaTue, 09 Oct 2012 20:52:15 -0000https://forge-allura.apache.org/p/allura/tickets/5095/https://forge-allura.apache.org/p/allura/tickets/4039/Right now all reads come from the mongo master, and we don't take advantage of the slaves. This is good because it means we also have current data when displaying a page (slaves can lag behind) but it also means we're not very efficient. Distributing reads to a slave would be helpful. Ideas: * use 'w' option to ensure writes propagate to slaves * that will slow down the writes, but we should make sure we don't have unnecessary writes anyway * http://www.mongodb.org/display/DOCS/getLastError+Command * http://api.mongodb.org/python/current/api/pymongo/collection.html#pymongo.collection.Collection.set_lasterror_options * add app logic to figure out when a session has made changes (logged in, some anonymous sessions) and use the master for reads, so data is current. All other reads can go to slaves.Dave BrondsemaSun, 22 Mar 2015 14:47:57 -0000https://forge-allura.apache.org/p/allura/tickets/4039/https://forge-allura.apache.org/p/allura/tickets/1472/Often times when a ticket is changed, it still shows up in searches and bins where it is no longer supposed to be -- and can stay in that state for a few minutes. If this is showing up now with the scale we have today, it's likely to grow worse as we migrate projects over to the beta. We should investigate the cause of these delays, and determine what we can do to scale up SOLR write performance. Probably our approach will be something like the following: For better search performance, we should have two solr cores we use, a 'hot' core and a 'cold' core. The hot core will be the one all online updates go to, and it should be committed frequently. Periodically (maybe once a day), all the indexing operations on the hot core will be replayed to the cold core and the hot core will be cleared out, keeping its index small. Queries will need to query both cores, and will need to verify each document ID is not in a 'deleted' list if the document comes from the cold core.Mark RammFri, 09 Oct 2015 21:39:52 -0000https://forge-allura.apache.org/p/allura/tickets/1472/