#4914 Add graceful restart command to taskd

v1.0.0
closed
Dave Brondsema
stability (19) sf-1 (616)
General
Cory Johns
2015-08-20
2012-09-12
Dave Brondsema
No

Add a graceful restart to command taskd, so that new code can take effect for new tasks, but existing long-running tasks can keep running.

  1. send the command - a separate paster command that sends a OS signal? need to make sure it goes to the right proc when multiple taskd processes are being used
  2. taskd receives the orders and should continue its current task but then stop when done with it
  3. when done, it should restart itself, e.g. http://www.daniweb.com/software-development/python/code/260268/restart-your-python-program
  4. add appropriate logging at every step, so it is clear what happens

Discussion

  • Dave Brondsema

    Dave Brondsema - 2012-09-14
    • status: open --> in-progress
    • assigned_to: Dave Brondsema
    • milestone: forge-oct-05 --> forge-sep-21
     

  • Dave Brondsema

    Dave Brondsema - 2012-09-14

    As reference, apachectl uses signals USR1 for graceful and HUP for restart. http://httpd.apache.org/docs/2.2/stopping.html#graceful

     

  • Dave Brondsema

    Dave Brondsema - 2012-09-14

    allura:db/4914

    For easier testing on our sandboxes:

    • /etc/init.d/mod_wsgi_autoreload stop
    • set NUM_TASKS=1 in /etc/init.d/reactor

    Test:

    • tail allura.log
    • killall -s USR1 paster
    • killall -s USR2 paster
    • killall -s TRAP paster
    • test when idle and with a long-running task

    Here's an example of a task good for testing

    import logging
import time
log = logging.getLogger(__name__)

@task
def count():
    for i in range(1,5):
        log.info('count %s' % i)
        time.sleep(1)

    Create a lot of them with:

    allurapaste shell /var/local/config/production.ini
> from allura.tasks.event_tasks import count
> count.post()
> count.post()
> ...

    Then you can change the logging within the event, and do a graceful restart to confirm that the new code is used.

     

  • Dave Brondsema

    Dave Brondsema - 2012-09-14
    • status: in-progress --> code-review
    • qa: Cory Johns
    • size: --> 1
     

  • Cory Johns - 2012-09-14
    • status: code-review --> closed
     

  • Dave Brondsema

    Dave Brondsema - 2012-09-14
    • labels: --> stability
     

Log in to post a comment.