Apache Allura™ / Tickets / #7806 Create a docker image for Allura

Dave Brondsema - 2014-12-11

status: open --> in-progress

assigned_to: Dave Brondsema
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2014-12-15

labels: --> sf-current
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2014-12-15

labels: sf-current --> sf-current, sf-2
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2014-12-17

We need to determine how we want to run the different services (mongo, solr, taskd, mail listener, webapp) since docker prefers one process per container. E.g. the default init scripts for mongo won't run inside docker. We could have a different container for each service, which would illustrate how they can be separate (good for production, complex for local). We might then need a tool like fig or decking to help orchestrate all those containers together. Or we could make it all run on a single host with one of the phusion base images or other workarounds.

Last edit: Dave Brondsema 2014-12-17

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Heith Seewald - 2014-12-17

Fig seems like a strong contender.
I like the fact that it's python/yaml based and can aggregate logs.
I also like that it's now an official docker project.

panamax has also been on my radar.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-01-26

labels: sf-current, sf-2 --> getting-started

status: in-progress --> open

assigned_to: Dave Brondsema --> nobody
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-04-14

At PyCon I learned about supervisord as a way to run multiple services from one single command. This could be used if we want to have a single docker box run everything. https://docs.docker.com/articles/using_supervisord/

However it sounds like the better way to use Docker is as-intended, one box per service. This would be needed anyway for us to have a realistic production deployment option.

Fig has been superseded by Docker Compose, which is probably the way we should go.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-05-19

labels: getting-started --> getting-started, sf-current
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-05-19

I've pushed some earlier work to branch db/7806. This is pretty much a conversion of the INSTALL file and tries to do everything on one box, which is not how we want to do it. Notably service mongod start doesn't work since docker base ubuntu images have their services system disabled intentionally.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Igor Bondarenko - 2015-05-20

status: open --> in-progress

assigned_to: Igor Bondarenko
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Igor Bondarenko - 2015-05-30

status: in-progress --> review
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Igor Bondarenko - 2015-05-30

Closed #773, #778, #779, #780. ib/7806

Created configuration suitable for development. See INSTALL-docker.markdown for details.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-06-01

labels: getting-started, sf-current --> getting-started, sf-current, sf-4
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-06-08

I had a few interruptions that caused me to rebuild the 'web' image. (Two were the Virtualbox VM got into "aborted" state, one was pip install locked up when my laptop hibernated). Running the 'build' step more than once seemed to take a while. Is there a way to save that image? Perhaps related: In docker-compose.yml there is a "allura_web" image referenced a few times. I assume that comes from the "web" declaration in same file?

I got this error after 'web' was built but the 'pip' command was incomplete. The container was stopped and I couldn't start it, forcing me to rebuild. This might be a corner case, not sure, but happened to me.

$ docker-compose up -d Recreating allura_outmail_1... Recreating allura_mongo_1... Recreating allura_solr_1... Creating allura_web_1... Creating allura_taskd_1... Cannot start container 9f27208cb8ed39ef0afe3ad153343dbaf2e09d6d2489298e9eea520a2ab4c4e5: [8] System error: exec: "paster": executable file not found in $PATH

And then I got stuck on this error. I didn't try to hard to fix it, but this is where I left off my testing.

Cannot start container 1dfff66d6f2fceadca9b7f6e257f5fb9bbea94076fa8b599bc312c42f4fc7bdf: Cannot link to a non running container: /allura_mongo_1 AS /allura_web_run_6/mongo mongo_1 | 2015-06-08T17:02:10.003+0000 [initandlisten] ERROR: Insufficient free space for journal files mongo_1 | 2015-06-08T17:02:10.003+0000 [initandlisten] Please make at least 3379MB available in /data/db/journal or use --smallfiles
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Igor Bondarenko - 2015-06-09
  
  In theory you don't need to run 'build' stage if it is successfully completed at least once. And docker saves intermediate images and reuses them on rebuild if command in Dockerfile is not changed, so second rebuild should be faster. This is not true for some commands, e.g. solr image is always wgets solr.
  
  I don't know what we can do regarding your first interruption, but the second one (pip install) implies that your image was already build, so you should be able to skip "build" stage in that case and just re-run pip install.
  
  Regarding "paster not found". Maybe deleting existing containers and then trying to run pip install again would help ($ docker-compose rm)? It should not require image rebuilding, you should have image in your cache already.
  
  That's right "allura_web" comes from the "web" image. Compose uses current directory + image name to give the name to the image, so if your allura code is not in the "allura" directory you should use -p allura for every docker-compose command. Unfortunately they don't provide a way to refer to current project name from config file, so "allura_web" just hardcoded there.
  
  One possibility is to upload prebuilt image to Docker Hub and use that in our docker-compose.yml, so that you fetch it once and reuse from cache whenever needed, but that's not really different from "build it once and reuse from cache whenever needed".
  
  Regarding mongo error. Do you have enough disk space available on your VirtualBox VM? Seems like mongo can't allocate files it needs.
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-06-08

status: review --> in-progress

Reviewer: Dave Brondsema
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-06-09

I'm past the rebuild & "paster not found" issues so I won't try to reproduce them again now. Docker Hub will be good to use at some point I think, but I agree its not really needed now.

For mongo: I've got a 20G boot2docker image. docker-compose run mongo df -h says:

Filesystem Size Used Avail Use% Mounted on none 19G 1.7G 16G 10% / tmpfs 1003M 0 1003M 0% /dev shm 64M 0 64M 0% /dev/shm tmpfs 1.8G 95M 1.7G 6% /data/db /dev/sda1 19G 1.7G 16G 10% /etc/resolv.conf /dev/sda1 19G 1.7G 16G 10% /etc/hostname /dev/sda1 19G 1.7G 16G 10% /etc/hosts tmpfs 1003M 0 1003M 0% /proc/kcore tmpfs 1003M 0 1003M 0% /proc/timer_stats

And mongo wants >3G on /data/db, and /data comes from the /allura-data volume right? Where is that set up?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Dave Brondsema - 2015-06-09
  
  This helped: command: mongod --smallfiles
  
  I'm still curious to learn where /allura-data is setup.
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Igor Bondarenko - 2015-06-10

In docker-compsose.yml, under the "mongo" section.

volumes: - /allura-data/mongo:/data/db

It means /allura-data/mongo from host system (in your case boot2docker VM) is mounted to /data/db in the container. If host system path is not available it is created by Docker Compose automatically.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Dave Brondsema - 2015-06-10
  
  Thanks. Everything seems to be working well now.
  
  Should we put command: mongod --smallfiles in the "mongo" section of docker-compose.yml? It was necessary for me.
  
  The "paster": executable file not found in $PATH error I reproduced now. It happens if you try to run docker-compose up -d before running the 3 adhoc 'web' commands. Perhaps that could be handled a bit more gracefully, but it was an error on my part, due to things aborting and not being exactly sure how to get started again.
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
  - Igor Bondarenko - 2015-06-11
    
    I think we should put mongo command to docker-compose.yml. I guess other boot2docker deployments will hit this issue too, so it is worth to prevent by default.
    
    The "paster" error is just due to python requirements not being installed inside docker. I'm not sure what we can do about that, maybe create some helper script that will run all three commands and check their return status?
    
    If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-06-12

status: in-progress --> closed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-06-12

Merged, including --smallfiles

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Dave Brondsema - 2015-06-15

labels: getting-started, sf-current, sf-4 --> getting-started, sf-4
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Igor Bondarenko - 2015-06-18

Milestone: unreleased --> asf_release_1.3.0
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Apache Allura™

Forge software for hosting software projects

Milestone

Searches

Help

#7806 Create a docker image for Allura

Discussion