#7806 Create a docker image for Allura

v1.3.0
closed
General
2015-08-20
2014-12-05
No

http://blog.dscpl.com.au/2014/12/hosting-python-wsgi-applications-using.html has a good starting point.

Would be nice to support development config (supplanting our Vagrant image) as well as a production-ready config (for which we don't have any good docs/images currently)

Discussion

  • Dave Brondsema

    Dave Brondsema - 2014-12-11
    • status: open --> in-progress
    • assigned_to: Dave Brondsema
     
  • Dave Brondsema

    Dave Brondsema - 2014-12-15
    • labels: --> sf-current
     
  • Dave Brondsema

    Dave Brondsema - 2014-12-15
    • labels: sf-current --> sf-current, sf-2
     
  • Dave Brondsema

    Dave Brondsema - 2014-12-17

    We need to determine how we want to run the different services (mongo, solr, taskd, mail listener, webapp) since docker prefers one process per container. E.g. the default init scripts for mongo won't run inside docker. We could have a different container for each service, which would illustrate how they can be separate (good for production, complex for local). We might then need a tool like fig or decking to help orchestrate all those containers together. Or we could make it all run on a single host with one of the phusion base images or other workarounds.

     

    Last edit: Dave Brondsema 2014-12-17
  • Heith Seewald

    Heith Seewald - 2014-12-17

    Fig seems like a strong contender.
    I like the fact that it's python/yaml based and can aggregate logs.
    I also like that it's now an official docker project.

    panamax has also been on my radar.

     
  • Dave Brondsema

    Dave Brondsema - 2015-01-26
    • labels: sf-current, sf-2 --> getting-started
    • status: in-progress --> open
    • assigned_to: Dave Brondsema --> nobody
     
  • Dave Brondsema

    Dave Brondsema - 2015-04-14

    At PyCon I learned about supervisord as a way to run multiple services from one single command. This could be used if we want to have a single docker box run everything. https://docs.docker.com/articles/using_supervisord/

    However it sounds like the better way to use Docker is as-intended, one box per service. This would be needed anyway for us to have a realistic production deployment option.

    Fig has been superseded by Docker Compose, which is probably the way we should go.

     
  • Dave Brondsema

    Dave Brondsema - 2015-05-19
    • labels: getting-started --> getting-started, sf-current
     
  • Dave Brondsema

    Dave Brondsema - 2015-05-19

    I've pushed some earlier work to branch db/7806. This is pretty much a conversion of the INSTALL file and tries to do everything on one box, which is not how we want to do it. Notably service mongod start doesn't work since docker base ubuntu images have their services system disabled intentionally.

     
  • Igor Bondarenko

    Igor Bondarenko - 2015-05-20
    • status: open --> in-progress
    • assigned_to: Igor Bondarenko
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-05-30
    • status: in-progress --> review
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-05-30

    Closed #773, #778, #779, #780. ib/7806

    Created configuration suitable for development. See INSTALL-docker.markdown for details.

     
  • Dave Brondsema

    Dave Brondsema - 2015-06-01
    • labels: getting-started, sf-current --> getting-started, sf-current, sf-4
     
  • Dave Brondsema

    Dave Brondsema - 2015-06-08

    I had a few interruptions that caused me to rebuild the 'web' image. (Two were the Virtualbox VM got into "aborted" state, one was pip install locked up when my laptop hibernated). Running the 'build' step more than once seemed to take a while. Is there a way to save that image? Perhaps related: In docker-compose.yml there is a "allura_web" image referenced a few times. I assume that comes from the "web" declaration in same file?

    I got this error after 'web' was built but the 'pip' command was incomplete. The container was stopped and I couldn't start it, forcing me to rebuild. This might be a corner case, not sure, but happened to me.

    $ docker-compose up -d
    Recreating allura_outmail_1...
    Recreating allura_mongo_1...
    Recreating allura_solr_1...
    Creating allura_web_1...
    Creating allura_taskd_1...
    Cannot start container 9f27208cb8ed39ef0afe3ad153343dbaf2e09d6d2489298e9eea520a2ab4c4e5: [8] System error: exec: "paster": executable file not found in $PATH
    

    And then I got stuck on this error. I didn't try to hard to fix it, but this is where I left off my testing.

    Cannot start container 1dfff66d6f2fceadca9b7f6e257f5fb9bbea94076fa8b599bc312c42f4fc7bdf: Cannot link to a non running container: /allura_mongo_1 AS /allura_web_run_6/mongo
    
    mongo_1 | 2015-06-08T17:02:10.003+0000 [initandlisten] ERROR: Insufficient free space for journal files
    mongo_1 | 2015-06-08T17:02:10.003+0000 [initandlisten] Please make at least 3379MB available in /data/db/journal or use --smallfiles
    
     
    • Igor Bondarenko

      Igor Bondarenko - 2015-06-09

      In theory you don't need to run 'build' stage if it is successfully completed at least once. And docker saves intermediate images and reuses them on rebuild if command in Dockerfile is not changed, so second rebuild should be faster. This is not true for some commands, e.g. solr image is always wgets solr.

      I don't know what we can do regarding your first interruption, but the second one (pip install) implies that your image was already build, so you should be able to skip "build" stage in that case and just re-run pip install.

      Regarding "paster not found". Maybe deleting existing containers and then trying to run pip install again would help ($ docker-compose rm)? It should not require image rebuilding, you should have image in your cache already.

      That's right "allura_web" comes from the "web" image. Compose uses current directory + image name to give the name to the image, so if your allura code is not in the "allura" directory you should use -p allura for every docker-compose command. Unfortunately they don't provide a way to refer to current project name from config file, so "allura_web" just hardcoded there.

      One possibility is to upload prebuilt image to Docker Hub and use that in our docker-compose.yml, so that you fetch it once and reuse from cache whenever needed, but that's not really different from "build it once and reuse from cache whenever needed".

      Regarding mongo error. Do you have enough disk space available on your VirtualBox VM? Seems like mongo can't allocate files it needs.

       
  • Dave Brondsema

    Dave Brondsema - 2015-06-08
    • status: review --> in-progress
    • Reviewer: Dave Brondsema
     
  • Dave Brondsema

    Dave Brondsema - 2015-06-09

    I'm past the rebuild & "paster not found" issues so I won't try to reproduce them again now. Docker Hub will be good to use at some point I think, but I agree its not really needed now.

    For mongo: I've got a 20G boot2docker image. docker-compose run mongo df -h says:

    Filesystem      Size  Used Avail Use% Mounted on
    none             19G  1.7G   16G  10% /
    tmpfs          1003M     0 1003M   0% /dev
    shm              64M     0   64M   0% /dev/shm
    tmpfs           1.8G   95M  1.7G   6% /data/db
    /dev/sda1        19G  1.7G   16G  10% /etc/resolv.conf
    /dev/sda1        19G  1.7G   16G  10% /etc/hostname
    /dev/sda1        19G  1.7G   16G  10% /etc/hosts
    tmpfs          1003M     0 1003M   0% /proc/kcore
    tmpfs          1003M     0 1003M   0% /proc/timer_stats
    

    And mongo wants >3G on /data/db, and /data comes from the /allura-data volume right? Where is that set up?

     
    • Dave Brondsema

      Dave Brondsema - 2015-06-09

      This helped: command: mongod --smallfiles

      I'm still curious to learn where /allura-data is setup.

       
  • Igor Bondarenko

    Igor Bondarenko - 2015-06-10

    In docker-compsose.yml, under the "mongo" section.

      volumes:
        - /allura-data/mongo:/data/db
    

     
    It means /allura-data/mongo from host system (in your case boot2docker VM) is mounted to /data/db in the container. If host system path is not available it is created by Docker Compose automatically.

     
    • Dave Brondsema

      Dave Brondsema - 2015-06-10

      Thanks. Everything seems to be working well now.

      Should we put command: mongod --smallfiles in the "mongo" section of docker-compose.yml? It was necessary for me.

      The "paster": executable file not found in $PATH error I reproduced now. It happens if you try to run docker-compose up -d before running the 3 adhoc 'web' commands. Perhaps that could be handled a bit more gracefully, but it was an error on my part, due to things aborting and not being exactly sure how to get started again.

       
      • Igor Bondarenko

        Igor Bondarenko - 2015-06-11

        I think we should put mongo command to docker-compose.yml. I guess other boot2docker deployments will hit this issue too, so it is worth to prevent by default.

        The "paster" error is just due to python requirements not being installed inside docker. I'm not sure what we can do about that, maybe create some helper script that will run all three commands and check their return status?

         
  • Dave Brondsema

    Dave Brondsema - 2015-06-12
    • status: in-progress --> closed
     
  • Dave Brondsema

    Dave Brondsema - 2015-06-12

    Merged, including --smallfiles

     
  • Dave Brondsema

    Dave Brondsema - 2015-06-15
    • labels: getting-started, sf-current, sf-4 --> getting-started, sf-4
     
  • Igor Bondarenko

    Igor Bondarenko - 2015-06-18
    • Milestone: unreleased --> asf_release_1.3.0
     

Log in to post a comment.