#4927 Avoid dog-piling repo refreshes NEEDS PRE-CLEANUP
closed
nobody
General
2015-08-20
2012-09-13
No
If multiple refreshes for the same repo run, it can unnecessarily block the task queue, especially if the repo is very large.
The task can check the repo status and abort if it says it's refreshing, then put a while loop around the refresh to check for any new unrecognized commits. One thing to note is that it has to execute at least once, even if no new commits, for the case of the user-started full refresh
Another case:
Due to bug [#5022], thousands of curl requests to
/auth/refresh_repo/p/projectname/code/were issued. That resulted in thousands of tasks created like:{ "_id" : ObjectId("50686e8971b75b10ebb20cb8"), "time_start" : null, "context" : { "project_id" : ObjectId("50682204b9363c76ee1d3aba"), "app_config_id" : ObjectId("50682204b9363c76ee1d3ae0"), "user_id" : null }, "time_stop" : null, "process" : null, "args" : [ [ "forgegit/model/git_repo/Repository#50682204b9363c76ee1d3ae6" ] ], "result_type" : "forget", "priority" : 10, "state" : "ready", "result" : null, "task_name" : "allura.tasks.index_tasks.add_artifacts", "kwargs" : { }, "time_queue" : ISODate("2012-09-30T16:08:41.579Z") }With the only variation being _id and time_queue
Related
Tickets:
#5022Created #185: [#4927] Avoid multiple repo refreshes (3cp)
Related
Tickets:
#4927What do you mean by this? Should we check for new commits when repo refresh is finished and run refresh again or what? Could you explain this in more detail, please?
Yes, that's more or less correct. The problem that we're having is that, if many changes are done to a repo in a short period or a refresh takes a long time to complete (which is relatively common), we'll end up with all the task workers trying to refresh a single repo on the same set of commits, thus blocking all other tasks.
We want to add a check in the
allura.tasks.repo_tasks.refresh()such that if the repo is currently in anything other than'ready'state, the task will abort and not double-queue the refresh.However, if the refresh takes a while, new commits may have come in that have not been processed and for which the corresponding refresh task has already aborted. So the task needs to check for new, unknown commits (see
allura.model.repo_refresh.unknown_commits()) and re-queue itself to pick up those missed changes.A while loop was just an initial thought; re-posting the task to the queue is a better implementation as it will allow other tasks to be processed during the break.
Thanks for clarifying.
Closed #185. Branch 42cc_4927.
We have ~ 17 refresh tasks currently in "busy" state, most of which were orphaned by hard restarts of taskd workers and need to be cleaned up or they will block the corresponding repos from being refreshed.
The command to find the "busy" refresh tasks is:
db.monq_task.find({state: 'busy', task_name: 'allura.tasks.repo_tasks.refresh'}, {process: 1})These will need to be compared to what the given workers are actually working on (obtained by sending a
USR1signal to the worker and watchingallura.logfor the response) to clear out any that are not actually busy.