forge-allura hosting setup

There is a newer version of this page. You can find it here.

This describes how forge-allura.apache.org itself is hosted. This is only relevant for the Apache Allura developers, and not for the general public.

Infrastructure

ASF infrastructure team is responsible for:

  • providing allura-vm host
  • controlling which committers can SSH in
  • DNS records:
    • A & MX record for forge-allura.apache.org
    • MX records for **.forge-allura.apache.org (any subdomain, recursively)
    • reverse DNS
  • routing inboud mail from apache mx server to allura-vm
  • outgoing mail routing
  • backups

Directories & Files

/allura-data holds configs, cron scripts, virtualenv and repo data

/var/local/allura has the app and is the docker root directory. Its a git checkout with some minor customizations (committed on top of normal master). Things like:

  • copying docker-compose-prod.yml to docker-compose.yml and further:
  • use /allura-data/forge-allura.ini for all command: lines
  • make sure all right ports are exposed or not exposed
  • some bot/spider blocks in scm_config/git-http/git-http.conf
  • serve url /docs/ from /allura-data/www-docs/
  • serve /allura-data/www-misc/ files

Code repos in /allura-data/scm/git/p/allura are for browsing, merge-requests etc. (separate from /var/local/allura used to run the site). Initially set up with:

git clone --mirror https://forge-allura.apache.org/git/p/allura/AlluraSite
git clone --mirror https://gitbox.apache.org/repos/asf/allura-site website.git
git clone --mirror https://gitbox.apache.org/repos/asf/allura git.git

And update-repos.sh keeps our mirrors up to date

Installed packages

docker-compose and mongodb-clients installed with apt

https://docs.docker.com/engine/install/ubuntu/ has longer install options, but apt install seems to get new enough version.

Docker and initial app setup

Enabled at startup with: systemctl enable docker

/etc/docker/daemon.json is set to

{
  "log-driver": "syslog",
  "live-restore": true
}

Docker was set up based around normal instructions https://forge-allura.apache.org/docs/getting_started/installation.html#using-docker:

  • scripts/init-docker-dev.sh (might clobber www-misc/robots.txt, can run steps by hand)
  • docker-compose run --rm oneoff bash then:
    • pip install -r requirements-dev.txt for sphinx for docs building
    • pip install ForgePastebin
    • pip install git+file:/allura-data/scm/git/p/allura/AlluraSite.git#egg=AlluraSite custom theme
    • pip uninstall ForgeSVN or rm /allura-data/virtualenv/lib/python2.7/site-packages/ForgeSVN.egg-link
    • No setup-app cmd

Cron jobs

The root cron file is currently:

MAILTO=""
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/bin/bash

# every few days restart of web services due to memory bloat
50 0 * * */2 /usr/bin/docker-compose -f /var/local/allura/docker-compose.yml restart web git-http

# check for new commits in repos
* * * * * /allura-data/update-repos.sh >>/tmp/cron-update-repos.log 2>&1

# TODO: /allura-data/update-allura.sh in cron

# every few days restart docker, seems necessary to help letsencrypt-nginx-proxy-companion connect to docker sometimes?
0 3 * * */5 service docker reload

Mail routing

Inbound email is received by postfix service, we need it routed the inmail docker container. Outbound mail goes from within docker (taskd) to postfix on main host. Edit /etc/postfix/main.cf:

  • append , forge-allura.apache.org, tickets.allura.p.forge-allura.apache.org, git.allura.p.forge-allura.apache.org, wiki.allura.p.forge-allura.apache.org to the mydestination line. Could do wildcards with https://serverfault.com/questions/133190/host-wildcard-subdomains-using-postfix
  • add local_recipient_maps = (blank) so that postfix doesn't try to validate 1234@ (ticket num) as a local user mailbox etc.
  • ensure inet_interfaces = all (so docker can reach out to it)
  • # 172.16+ private/internal network needed for docker hosts with addr like 172.18.0.9
  • append 172.16.0.0/12 to mynetworks = line
  • add transport_maps = hash:/etc/postfix/transport
  • Create /etc/postfix/transport as:
# send all email to our domain to the python handler
forge-allura.apache.org smtp:127.0.0.1:8825
.forge-allura.apache.org smtp:127.0.0.1:8825
*.forge-allura.apache.org smtp:127.0.0.1:8825

Testing outgoing mail:

apt install swaks
swaks --protocol ESMTPS --to myself@wherever.com --from noreply@forge-allura.apache.org --body "test message2" --server 172.17.0.1:25

And then also from within docker / the app.

Test incoming mail:

Run telnet localhost 25 and then see [Notes] for send mail. Check that outmail and taskd containers got it.

Migrating to a new host

From time to time, ASF infra will work with us to upgrade to newer hardware and/or newer OS setup. Here's some info specific to migration (in addition to the general setup noted above).

Copy data from one host (in this example vm2 is the old host). Probably need to tweak ssh config on old host so you can ssh into it as root (e.g. add pub file to /etc/ssh/ssh_keys/)

scp -r allura-vm2.apache.org:/allura-data/solr .
ssh allura-vm2.apache.org 'cd /allura-data/scm/git/ && tar zcf - u' | tar zxf -
ssh allura-vm2.apache.org 'cd /var/local/allura && docker-compose run --rm mongo mongodump --host mongo --gzip --archive' > /allura-data/mongoarchive.gz
docker-compose up -d mongo
mongorestore -v --gzip --archive=/allura-data/mongoarchive.gz --drop

When testing URL other than forge-allura.apache.org you'll have to edit VIRTUAL_HOST and LETSENCRYPT_HOST in docker-compose.yml and restart the nginx containers. Remember to set them back when all ready to go live under forge-allura DNS.

The direct hostname (allura-vm) is reachable at port 80 & 443 which isn't ideal (everyone should use forge-allura.apache.org). But the SSL cert shouldn't be valid and port 80 errors, so nobody will really use that even if they happen across it.

Remember to test that local repos (user forks mostly) serve ok. They use ApacheAccessHandler etc.

Maintenance & General Administration

Periodically we should run:

/allura-data/update-allura.sh to update latest code & python packages. Occasionally manual steps need too.

apt upgrade to keep system packages up to date. Reboot as needed

Here's how to run some commands with docker:

cd /var/local/allura
docker-compose run --rm oneoff paster ensure_index /allura-data/forge-allura.ini
docker-compose run --rm oneoff paster script /allura-data/forge-allura.ini allura/scripts/whatever.py -- --options here
docker-compose logs -f --tail 10