Ticket search results

Better handing of tmp dirs during merge

Dave Brondsema — Wed, 27 Jan 2016 20:43:32 -0000

Doing a one-click merge of a very large repo can potentially cause problems if the working dir is bigger than the tmp dir mount supports. * `def merge` can use a try/finally so that `shutil.rmtree` always runs even if the clone fails (e.g. out of disk space) * see if its possible to a shared clone (see also `_shared_clone`) so the object files don't have to get copied, reducing disk usage and execution time. It'd be nice if we could set a max size limit and check that in `can_merge` or `merge_allowed` but it doesn't seem to be a way to check the size of a working copy ahead of time. There is `git count-objects -vH` but that's just the repo size, not the working copy.

smtp_server hides errors

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

The `smtp_server` paster command sometimes errors (and may die, even) but nothing is logged.

add faulthandler to smtp_server

Dave Brondsema — Thu, 20 Aug 2015 22:07:20 -0000

Sometimes smtp_server dies with no cause in the logs. Lets add `faulthandler` to it like we've done with `taskd` already

Prevent spiders from requesting tarballs

Dave Brondsema — Thu, 20 Aug 2015 22:07:20 -0000

The following are examples of spiders requesting tarball creation. This is unnecessary and a waste of resources. We should make it not possible. We already have `rel=nofollow` but that apparently isn't working. I think the best solution is to require the URL to be a POST. ~~~~ "GET /p/z-i/code-0/208/tarball HTTP/1.0" 200 16400 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "GET /p/jhotdraw/svn/729/tarball HTTP/1.0" 200 17834 "-" "msnbot/0.01 (+http://search.msn.com/msnbot.htm)" "GET /p/fourpane/git4pane/ci/ec65df3a5ff2ec7be011c0722286e766c2b76d94/tarball HTTP/1.0" 200 18137 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)" "GET /u/lluct/me722-cm/ci/0aa649648a00979ad6ca9e9d61df4e44eb694259/tarball?path=/external/clang HTTP/1.0" 200 17918 "-" "YisouSpider" ~~~~

Use --ignore-externals in svn export

Dave Brondsema — Thu, 20 Aug 2015 22:07:20 -0000

The svn export command for creating code snapshots should use `--ignore-externals`

Rate-limit import tasks

Dave Brondsema — Thu, 20 Aug 2015 22:07:20 -0000

Import tasks can be resource-intensive and potentially abusive. Allow one import per tool type (or importer, perhaps) per project to run at simultaneously. This can be tracked via the `get/set_tool_data`. We'll have to make sure that error handling [#6530] decrements the count when done. Probably best to track an integer rather than a bool, so that its easier to customize if some deployments of Allura want to allow N simultaneous imports.

widget template shouldn't extend g.theme.master

Dave Brondsema — Thu, 20 Aug 2015 22:07:54 -0000

allura/templates/discussion/index.html isn't a full template that extends g.theme.master ON forum browse pages, it eventually calls forgediscussion/templates/discussion_widgets/discussion.html which is a widget template but does extend g.theme.master That is non-standard and has the effect of template globals (like `config`) not being present within the `custom_js` theme macro. We should see if these templates (and any other related ones) can be made more standard, to avoid such issues.

Set max limit on limit param

Dave Brondsema — Mon, 16 Nov 2015 16:09:52 -0000

There should be an upper bound on the `limit` param used by paging in various tools so it can't be abused. Perhaps exemptions for nbhd admins since it could be useful when used properly.

taskd can get into an infinite loop

Dave Brondsema — Thu, 20 Aug 2015 22:07:55 -0000

`check_running` isn't implemented correctly, so it ends up in an tight loop without sleeping

taskd leaves defunct git processes around

Dave Brondsema — Thu, 20 Aug 2015 22:07:10 -0000

Not known to be causing a problem, but would be good to get it cleaned up ~~~~ allura 5996 1 0 Oct26 ? 00:00:00 sh -c paster taskd /var/local/config/production.ini 2>&1 | /usr/sbin/cronolog --symlink=/var/local/log/reactor_logs/reactor_log /var/local/log/reacto allura 5997 5996 23 Oct26 ? 5-18:07:05 \_ /var/local/env-allura/bin/python /var/local/env-allura/bin/paster taskd /var/local/config/production.ini allura 13252 5997 0 Oct26 ? 00:00:00 \_ [git] allura 13253 5997 0 Oct26 ? 00:00:00 \_ [git] allura 14060 5997 0 Oct29 ? 00:00:00 \_ [git] allura 14061 5997 0 Oct29 ? 00:00:00 \_ [git] allura 16158 5997 0 Oct31 ? 00:00:00 \_ [git] allura 16168 5997 0 Oct31 ? 00:00:00 \_ [git] allura 10658 5997 0 Oct31 ? 00:00:00 \_ [git] allura 10659 5997 0 Oct31 ? 00:00:00 \_ [git] allura 32389 5997 0 Nov01 ? 00:00:00 \_ [git] allura 32390 5997 0 Nov01 ? 00:00:00 \_ [git] allura 11163 5997 0 Nov05 ? 00:00:00 \_ [git] allura 11164 5997 0 Nov05 ? 00:00:00 \_ [git] allura 28112 5997 0 Nov06 ? 00:00:00 \_ [git] allura 28207 5997 0 Nov06 ? 00:00:00 \_ [git] allura 8364 5997 0 Nov08 ? 00:00:00 \_ [git] allura 8365 5997 0 Nov08 ? 00:00:00 \_ [git] allura 5434 5997 0 Nov16 ? 00:00:00 \_ [git] allura 5435 5997 0 Nov16 ? 00:00:00 \_ [git] allura 13571 5997 0 Nov19 ? 00:00:00 \_ [git] ~~~~

Option to not always commit to solr

Dave Brondsema — Thu, 20 Aug 2015 22:07:54 -0000

Pysolr defaults to commit=True for add() and delete(). On a high-traffic allura instance, this can be too many commits and cause maxWarmingSearchers errors. There should be a config option (default True, for development) to control this flag to pysolr. Then, when not always committing, we need to determine how to make updates take effect quickly enough. SourceForge runs a 5 minute cron which executes the "commit" paster command, which creates a task to do a commit. That may not be enough. We may need to do [#1472] also.

Check **kwargs on ming find() NEEDS RELEASE

Dave Brondsema — Thu, 20 Aug 2015 22:07:52 -0000

Ming's get() supports **kwargs but find() does not. Ensure that find() requires at least one positional arg for the criteria, rather than returning everything in the collection. Ming does a few checks like that already in other places. Prevent https://engr.geek.net/git/?p=forge-classic.git;a=commitdiff;h=373110db792b11e0e5f82ded08c0c36e12b82a39 from recurring

post-receive-user incorrectly created when cloning a repo

Dave Brondsema — Thu, 20 Aug 2015 22:07:54 -0000

When forking a repo within Allura, the post-receive-user file should not be created. Currently it is created like the following, which is self-recursive and bad. ~~~~ #!/bin/bash # The following is required for site integration, do not remove/modify. # Place user hook code in post-receive-user and it will be called from here. curl -s http://sourceforge.net/auth/refresh_repo/p/simplemvcv/code/ DIR="$(dirname "${BASH_SOURCE[0]}")" if [ -x $DIR/post-receive-user ]; then exec $DIR/post-receive-user fi ~~~~

Git file handles not cleaned up in taskd procs

Dave Brondsema — Thu, 20 Aug 2015 22:07:53 -0000

See [#3312] for previous work on this issue; perhaps it just need to be applied to taskd. Our long-running taskd processes are accumulating git filehandles. Proof: ~~~~ $ sudo -u allura /usr/sbin/lsof -p 6406 | grep git | tail paster 6406 allura 336r unknown /p/odcleanstore/code.git/objects/pack/pack-a4022ff36c0239c4040456144c6ddfdf6e04e6fc.idx (stat: No such file or directory) paster 6406 allura 337r unknown /p/odcleanstore/code.git/objects/pack/pack-4afcdc5c1fa600738e95b3a33cf9cb646bae47b0.idx (stat: No such file or directory) paster 6406 allura 338r unknown /p/ismrmrd/code.git/objects/pack/pack-ec1295a6a871c039c33d1538d691b5de5325c7bf.pack (stat: No such file or directory) paster 6406 allura 339r unknown /p/ismrmrd/code.git/objects/pack/pack-ec1295a6a871c039c33d1538d691b5de5325c7bf.idx (stat: No such file or directory) paster 6406 allura 340r unknown /p/ufoai/code.git/objects/pack/pack-8e9dd5fb5ca71e1658e3f666d74f4127d55ed5fb.pack (stat: No such file or directory) ... ~~~~

Handle pymongo AutoReconnect during taskd restart

Dave Brondsema — Thu, 20 Aug 2015 22:07:53 -0000

A signal to taskd (e.g. SIGHUP for graceful restart) will frequently cause an error if pymongo is currently doing something: http://pastie.org/private/mdiglqtzkcydtyfusja **AutoReconnect** Raised when a connection to the database is lost and an attempt to auto-reconnect will be made. In order to auto-reconnect you must handle this exception, recognizing that the operation which caused it has not necessarily succeeded. Future operations will attempt to open a new connection to the database (and will continue to raise this exception until the first successful connection is made). ---- Could we implement an auto-retry option in ming? Probably with a setting that would only be enabled for taskd, initially.

Avoid dog-piling repo refreshes NEEDS PRE-CLEANUP

Dave Brondsema — Thu, 20 Aug 2015 22:07:54 -0000

If multiple refreshes for the same repo run, it can unnecessarily block the task queue, especially if the repo is very large. The task can check the repo status and abort if it says it's refreshing, then put a while loop around the refresh to check for any new unrecognized commits. One thing to note is that it has to execute at least once, even if no new commits, for the case of the user-started full refresh

Add graceful restart command to taskd

Dave Brondsema — Thu, 20 Aug 2015 22:07:56 -0000

Add a graceful restart to command taskd, so that new code can take effect for new tasks, but existing long-running tasks can keep running. 1. send the command - a separate paster command that sends a OS signal? need to make sure it goes to the right proc when multiple taskd processes are being used 2. taskd receives the orders and should continue its current task but then stop when done with it 3. when done, it should restart itself, e.g. http://www.daniweb.com/software-development/python/code/260268/restart-your-python-program 4. add appropriate logging at every step, so it is clear what happens

Better handling of inability to configure logging

Dave Brondsema — Thu, 20 Aug 2015 22:07:55 -0000

This silently hides major issues: ~~~~ except Exception: # pragma no cover print >> sys.stderr, ( 'Could not configure logging with config file %s' % self.args[0]) ~~~~

ensure_index should create indexes before dropping indexes

Dave Brondsema — Thu, 20 Aug 2015 22:07:54 -0000

If changing an index from (a,b) to (a,b,c) for example, it is important that the new index is created before the old index is dropped. Otherwise, while the index is being created, queries for (a) or (a,b) will not have an index to use. This is even more critical if an index creation fails for some reason, and the database is left without the index until a fix might be developed.