Ticket search results

Remove CommitRun usage

Dave Brondsema — Mon, 17 Jun 2019 15:19:14 -0000

Looks like CommitRun models & data haven't really been used since [#6272]. Usage is only in tests and repo refresh code, not actual site code. There are also some queries that can be pretty large, when thousands of commits are involved, like: ```python if CommitRunDoc.m.find(dict(commit_ids={'$all': run1.commit_ids}, _id={'$ne': run1._id})).count(): ```

Remove TreesDoc usage

Dave Brondsema — Mon, 05 Feb 2018 17:00:51 -0000

I think we can remove the TreesDoc model and its mongo collection which can be quite large. [#7828] did some prior analysis. My current analysis is that some tests & scripts will have to be updated, but generally everything should keep working. Proof of concept is in branch `db/remove_treestuff`. The TreesDoc model is used in some LastCommitDoc "refresh" logic currently. Should be able to remove that and/or replace it with calls to the lazy computation of LastCommitDocs. The only sticking point, I think, is that TreesDoc provides a mapping of commit_id to tree_id so that some cleanup knows what TreeDoc records can be deleted. So those would likely never be able to be cleaned up properly. Maybe ok, removing them is very infrequent - only for doing full repo refreshes, not even done when projects are removed (esp. since tree info can be shared across repos/forks)

Use repo directly instead of DiffInfoDoc

Dave Brondsema — Thu, 20 Aug 2015 22:06:09 -0000

See [#7828] for analysis of where DiffInfoDoc is used. Goal is to remove it altogether, using SCM data directly. Then also remove the building of DiffInfoDoc records during repo refresh. (If there are really slow computations we could keep using DiffInfoDoc for a cache of those results)

Analyze & document usage of repo collections

Dave Brondsema — Mon, 02 Mar 2015 17:03:19 -0000

We've done some work in the past for our SCM repos to be "indexless", that is use the git/hg/svn repo directly instead of indexing in mongo and using that. (Storing in mongo can take up a lot of space and also adds delay for the indexing process to run) Analyze where each of the following collections (models) is used. Perhaps cross-reference by page or function (e.g. browse repo, view commit, etc; also git/svn/hg). Then we can plan which pages' functionality needs to be updated to be able to remove them. Collections (with relative size factors based on sf.net data): * repo_trees (4x) * repo_tree (2x) * repo_last_commit (2x) * repo_diffinfo (1x) * repo_ci (very tiny) * repo_commitrun (very tiny)

Change SVN's browse commits graph to direct SCM access

Dave Brondsema — Thu, 20 Aug 2015 22:07:00 -0000

The "Browse Commits" interface can be incredibly slow, especially for larger repositories. Override the `commit_browser_data` method in ForgeSVN to get data directly from svn instead of doing lots of mongo lookups and data manipulation. That implementation in Allura core seems much more complex than is necessary for SVN anyway, since SVN is always a single linear list of commits (no "real" branches, merges, or multiple heads to consider)

Change hg browser to get "last commit" info from hg instead of mongo

Cory Johns — Thu, 20 Aug 2015 22:07:10 -0000

[#6686] changed git to pull last commit info directly from the SCM instead of mongo, and SVN already did the same. Now we just need to change the Mercurial browser to do the same.

Log view should use direct SCM access

Dave Brondsema — Thu, 20 Aug 2015 22:07:56 -0000

One particular slow point is multiple calls to symbolics_for_commit (one for each commit).

Switch to on-demand (indexless) tree browsing for git

Cory Johns — Thu, 20 Aug 2015 22:07:55 -0000

Tests using https://sourceforge.net/p/allura/pastebin/518bb63d7929e51a2548196a/ indicate that we should be able to switch the tree browsing for git to get its info directly from GitPython without relying on the fragile index data. This should give us comparable performance without having to wait for the index to complete. Create a GitTreeBrowser subclass to use this implementation (with tests), preferably using AJAX (tree / blob names can be pulled quickly, AJAX to fill in the rest of the last-commit data).

Pull branches and tags directly from SCM

Cory Johns — Thu, 20 Aug 2015 22:07:55 -0000

The list of heads (branches and tags) are currently indexed at the start of a refresh. However, if a head is added or removed while a previous refresh is running (refreshes can sometimes overlap, especially on large repos, and there is code in place to handle that), then the head cache can get out of sync even while the commits themselves are indexed. Pulling the list of heads directly from the SCM should not be expensive even on large repos, so we should do that instead of caching. (Looking up heads already hits the SCM directly.) While we're at it, we should make loading them for the sidebar AJAX to speed up page render time. --> Separate ticket

taskd leaves defunct git processes around

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

Not known to be causing a problem, but would be good to get it cleaned up ~~~~ allura 5996 1 0 Oct26 ? 00:00:00 sh -c paster taskd /var/local/config/production.ini 2>&1 | /usr/sbin/cronolog --symlink=/var/local/log/reactor_logs/reactor_log /var/local/log/reacto allura 5997 5996 23 Oct26 ? 5-18:07:05 \_ /var/local/env-allura/bin/python /var/local/env-allura/bin/paster taskd /var/local/config/production.ini allura 13252 5997 0 Oct26 ? 00:00:00 \_ [git] allura 13253 5997 0 Oct26 ? 00:00:00 \_ [git] allura 14060 5997 0 Oct29 ? 00:00:00 \_ [git] allura 14061 5997 0 Oct29 ? 00:00:00 \_ [git] allura 16158 5997 0 Oct31 ? 00:00:00 \_ [git] allura 16168 5997 0 Oct31 ? 00:00:00 \_ [git] allura 10658 5997 0 Oct31 ? 00:00:00 \_ [git] allura 10659 5997 0 Oct31 ? 00:00:00 \_ [git] allura 32389 5997 0 Nov01 ? 00:00:00 \_ [git] allura 32390 5997 0 Nov01 ? 00:00:00 \_ [git] allura 11163 5997 0 Nov05 ? 00:00:00 \_ [git] allura 11164 5997 0 Nov05 ? 00:00:00 \_ [git] allura 28112 5997 0 Nov06 ? 00:00:00 \_ [git] allura 28207 5997 0 Nov06 ? 00:00:00 \_ [git] allura 8364 5997 0 Nov08 ? 00:00:00 \_ [git] allura 8365 5997 0 Nov08 ? 00:00:00 \_ [git] allura 5434 5997 0 Nov16 ? 00:00:00 \_ [git] allura 5435 5997 0 Nov16 ? 00:00:00 \_ [git] allura 13571 5997 0 Nov19 ? 00:00:00 \_ [git] ~~~~