Apache Allura™

• size: --> 2

Another case:

Due to bug [#5022], thousands of curl requests to /auth/refresh_repo/p/projectname/code/ were issued. That resulted in thousands of tasks created like:

{ "_id" : ObjectId("50686e8971b75b10ebb20cb8"), "time_start" : null, "context" : { "project_id" : ObjectId("50682204b9363c76ee1d3aba"), "app_config_id" : ObjectId("50682204b9363c76ee1d3ae0"), "user_id" : null }, "time_stop" : null, "process" : null, "args" : [ [ "forgegit/model/git_repo/Repository#50682204b9363c76ee1d3ae6" ] ], "result_type" : "forget", "priority" : 10, "state" : "ready", "result" : null, "task_name" : "allura.tasks.index_tasks.add_artifacts", "kwargs" : { }, "time_queue" : ISODate("2012-09-30T16:08:41.579Z") }

With the only variation being _id and time_queue

• labels: stability --> stability, 42cc

Created #185: [#4927] Avoid multiple repo refreshes (3cp)

• status: open --> in-progress

then put a while loop around the refresh to check for any new unrecognized commits.

What do you mean by this? Should we check for new commits when repo refresh is finished and run refresh again or what? Could you explain this in more detail, please?

Yes, that's more or less correct. The problem that we're having is that, if many changes are done to a repo in a short period or a refresh takes a long time to complete (which is relatively common), we'll end up with all the task workers trying to refresh a single repo on the same set of commits, thus blocking all other tasks.

We want to add a check in the allura.tasks.repo_tasks.refresh() such that if the repo is currently in anything other than 'ready' state, the task will abort and not double-queue the refresh.

However, if the refresh takes a while, new commits may have come in that have not been processed and for which the corresponding refresh task has already aborted. So the task needs to check for new, unknown commits (see allura.model.repo_refresh.unknown_commits()) and re-queue itself to pick up those missed changes.

A while loop was just an initial thought; re-posting the task to the queue is a better implementation as it will allow other tasks to be processed during the break.

Thanks for clarifying.

Closed #185. Branch 42cc_4927.

• status: in-progress --> code-review

• qa: Cory Johns

• summary: Avoid dog-piling repo refreshes --> Avoid dog-piling repo refreshes NEEDS PRE-CLEANUP
• status: code-review --> validation

We have ~ 17 refresh tasks currently in "busy" state, most of which were orphaned by hard restarts of taskd workers and need to be cleaned up or they will block the corresponding repos from being refreshed.

The command to find the "busy" refresh tasks is:

db.monq_task.find({state: 'busy', task_name: 'allura.tasks.repo_tasks.refresh'}, {process: 1})

These will need to be compared to what the given workers are actually working on (obtained by sending a USR1 signal to the worker and watching allura.log for the response) to clear out any that are not actually busy.

• status: validation --> closed
• size: 2 --> 0