Apache Allura™

Forge software for hosting software projects

Brought to you by: alexluberg, brondsem, ccruz, deshani, and 11 others

forge-allura hosting setup

This describes how forge-allura.apache.org itself is hosted. This is only relevant for the Apache Allura developers, and not for the general public.

Infrastructure
Directories & Files
Installed packages
Docker and initial app setup
Cron jobs
Mail routing
Migrating to a new host
Maintenance & General Administration
- If python version is changed

Infrastructure

https://infra.apache.org/vm-management.html

ASF infrastructure team is responsible for:

providing allura-vm host (currently Ubuntu 18.04)
controlling which committers can SSH in
- for sudo setup see https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=122916896
DNS records:
- A & MX record for forge-allura.apache.org
- MX records for **.forge-allura.apache.org (any subdomain, recursively)
- reverse DNS
routing inboud mail from apache mx server to allura-vm
outgoing mail routing
backups

Directories & Files

/allura-data holds configs, cron scripts, virtualenv and repo data

/var/local/allura has the app and is the docker root directory. Its a git checkout with some customizations (committed on top of normal master). Things like:

copying docker-compose-prod.yml to docker-compose.yml and further:
use /allura-data/forge-allura.ini for all command: lines
make sure all right ports are exposed or not exposed
allura_web image names instead of allura-web until newer Docker is used
some bot/spider blocks in scm_config/git-http/git-http.conf
serve url /docs/ from /allura-data/www-docs/
serve /allura-data/www-misc/ files

Code repos in /allura-data/scm/git/p/allura are for browsing, merge-requests etc. (separate from /var/local/allura used to run the site). Initially set up with:

git clone --mirror https://forge-allura.apache.org/git/p/allura/AlluraSite
git clone --mirror https://gitbox.apache.org/repos/asf/allura-site website.git
git clone --mirror https://gitbox.apache.org/repos/asf/allura git.git

And update-repos.sh keeps our mirrors up to date

Installed packages

docker-compose (and thus docker.io) and mongodb-clients installed with apt

https://docs.docker.com/engine/install/ubuntu/ has longer install options, but apt install seems to get new enough version.

Docker and initial app setup

Enabled at startup with: systemctl enable docker

/etc/docker/daemon.json is set to

{
  "log-driver": "syslog",
  "live-restore": true
}

Docker was set up based around normal instructions https://forge-allura.apache.org/docs/getting_started/installation.html#using-docker:

scripts/init-docker-dev.sh (might clobber www-misc/robots.txt, can run steps by hand)
docker-compose run --rm oneoff bash then:
- pip install -r requirements-dev.txt for sphinx for docs building
- pip install ForgePastebin
- pip install git+file:/allura-data/scm/git/p/allura/AlluraSite.git#egg=AlluraSite custom theme
- pip uninstall ForgeSVN
- No setup-app cmd

Cron jobs

The root cron file is currently:

MAILTO=""
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/bin/bash

# every few days restart of web services due to memory bloat
50 0 * * */2 /usr/bin/docker-compose -f /var/local/allura/docker-compose.yml restart web git-http

# check for new commits in repos
* * * * * /allura-data/update-repos.sh >>/tmp/cron-update-repos.log 2>&1

# TODO: /allura-data/update-allura.sh in cron

# every few days restart docker, seems necessary to help letsencrypt-nginx-proxy-companion connect to docker sometimes?
0 3 * * */5 service docker reload

Mail routing

Inbound email is received by postfix service, we need it routed the inmail docker container. Outbound mail goes from within docker (taskd) to postfix on main host. Edit /etc/postfix/main.cf:

append , forge-allura.apache.org, tickets.allura.p.forge-allura.apache.org, git.allura.p.forge-allura.apache.org, wiki.allura.p.forge-allura.apache.org to the mydestination line. Could do wildcards with https://serverfault.com/questions/133190/host-wildcard-subdomains-using-postfix
add local_recipient_maps = (blank) so that postfix doesn't try to validate 1234@ (ticket num) as a local user mailbox etc.
ensure inet_interfaces = all (so docker can reach out to it)
# 172.16+ private/internal network needed for docker hosts with addr like 172.18.0.9
append 172.16.0.0/12 to mynetworks = line
add transport_maps = hash:/etc/postfix/transport
Create /etc/postfix/transport as:

# send all email to our domain to the python handler
forge-allura.apache.org smtp:127.0.0.1:8825
.forge-allura.apache.org smtp:127.0.0.1:8825
*.forge-allura.apache.org smtp:127.0.0.1:8825

Run postmap /etc/postfix/transport and service postfix reload
Will infra's "puppet" config overwrite those? https://github.com/apache/infrastructure-p6/blob/production/data/nodes/allura-vm.apache.org.yaml has some of the customizations we need
/allura-data/forge-allura.ini has smtp_server set to 172.17.0.1 because that's result of ip addr show docker0 and will let a docker container reach the outer host

Testing outgoing mail:

apt install swaks
swaks --protocol ESMTPS --to myself@wherever.com --from noreply@forge-allura.apache.org --body "test message2" --server 172.17.0.1:25

And then also from within docker / the app.

Test incoming mail:

Run telnet localhost 25 and then see [Notes] for send mail. Check that outmail and taskd containers got it.

Migrating to a new host

From time to time, ASF infra will work with us to upgrade to newer hardware and/or newer OS setup. Here's some info specific to migration (in addition to the general setup noted above).

Copy data from one host (in this example vm2 is the old host). Probably need to tweak ssh config on old host so you can ssh into it as root (e.g. add pub file to /etc/ssh/ssh_keys/)

scp -r allura-vm2.apache.org:/allura-data/solr .
ssh allura-vm2.apache.org 'cd /allura-data/scm/git/ && tar zcf - u' | tar zxf -
ssh allura-vm2.apache.org 'cd /var/local/allura && docker-compose run --rm mongo mongodump --host mongo --gzip --archive' > /allura-data/mongoarchive.gz
docker-compose up -d mongo
mongorestore -v --gzip --archive=/allura-data/mongoarchive.gz --drop

When testing URL other than forge-allura.apache.org you'll have to edit VIRTUAL_HOST and LETSENCRYPT_HOST in docker-compose.yml and restart the nginx containers. Remember to set them back when all ready to go live under forge-allura DNS.

The direct hostname (allura-vm) is reachable at port 80 & 443 which isn't ideal (everyone should use forge-allura.apache.org). But the SSL cert shouldn't be valid and port 80 errors, so nobody will really use that even if they happen across it.

Remember to test that local repos (user forks mostly) serve ok. They use ApacheAccessHandler etc.

Maintenance & General Administration

Periodically we should run:

apt upgrade to keep system packages up to date. Reboot as needed.

/allura-data/update-allura.sh to update latest code & python packages. Occasionally manual steps need too. If docker-compose.yml for examples is changed, then local changes to that file will cause a conflict, and will have to be resolved. If Dockerfile is changed you'll have to run docker-compose build and stop/start services for it to take effect.

If python version is changed

This is controlled by PY_VERSION in Dockerfile

docker-compose stop
docker-compose build
mv /allura-data/virtualenv/ /allura-data/virtualenv-old
docker-compose run --rm -w /allura oneoff bash -c '$PYTHON_EXE -m venv /allura-data/virtualenv && /allura-data/virtualenv/bin/pip install -U pip wheel'
use "oneoff" commands to install requirements.txt AND extra packages, see Initial Setup section above with its pip install several commands
docker-compose up -d

Here's how to run some commands with docker:

cd /var/local/allura
docker-compose run --rm oneoff paster ensure_index /allura-data/forge-allura.ini
docker-compose run --rm oneoff paster script /allura-data/forge-allura.ini allura/scripts/whatever.py -- --options here
docker-compose logs -f --tail 10

Tickets: ~~#8366~~
Wiki: Home
Wiki: Notes