Files
crosscode-platform/docs/ROLLOUT-1.0.md
T

5.8 KiB

Rollout: Hub 2.1, the agent and Events 2.3 on the real servers

The order to move the live servers onto the agent, so that installs, updates and uninstalls are run from the Hub. Each step can be stopped after without leaving anything half done. Tested end to end on a VM (WSL: a church server, a central server with frps, the Hub) on 2026-11-17; the real servers add FASTPANEL, real DNS and Let's Encrypt, which the VM didn't have, so step 6 starts with a throwaway church.

Releases: events@2.3.0, contracts@1.3.0, agent@1.0.0 (crosscode-platform), Hub v2.1.0, crosscode-site v1.3.0.

Before you start

  • A backup of the Hub's database and backend/keys/ (the licence key).
  • A backup of each church's database (pg_dump) and uploads/. adopt and deploy take their own backups too, but keep one that's yours.
  • Know your frp port range (allow_ports in frps.ini) and which ports your hand-written tunnels use.

1. The Hub (2.1.0)

Follow events-hub/deployment.md → "Upgrading to 2.1":

  • npm run generate-keys -- --job in backend/; back up keys/hub-job-key.pem with the licence key. Agents pin this key when they join: losing or replacing it means every server joins again.
  • .env: HUB_JOB_SIGNING_KEY_FILE=./keys/hub-job-key.pem and HUB_PUBLIC_URL=https://hub.crosscode.co.za.
  • bash scripts/deploy.sh v2.1.0.
  • Servers → Ports & defaults: the port range, the central server's IP, the install defaults. Mark every port a hand-written tunnel uses as taken.

2. The central server (frps)

  • frps.ini has proxy_bind_addr = 127.0.0.1 (tunnelled ports are then only reachable by nginx on that server). Restart frps and check every FASTPANEL site still answers.
  • Hub → Servers → Add server → Central; run the command it shows as root. It's read-only: it checks a new install's ports in frps and watches frps.
  • Within a minute the server shows on Servers with its stats.

3. Each church server

  • Hub → Servers → Add server → Church; run the command as root. It installs Node tools, the agent (/opt/crosscode/agent) and the crosscode script (/opt/crosscode/tools, from main), and starts crosscode-agent. The agent listens on nothing.
  • On the server's page: its configuration. TRUST_PROXY=loopback behind frp; the database: DB_HOST/DB_PORT/DB_ADMIN_URL for a separate PostgreSQL, or nothing for the local one.
  • systemctl status crosscode-agent is active; the server's stats show on the Hub.
  • The first time the agent changes frpc.ini it adds the admin lines and restarts frpc once (every tunnel on that server drops for a few seconds). Do the first install on a quiet evening. Later changes only reload frpc.

4. Existing sites (the demo site and the churches): adopt, don't reinstall

Moving a site you already run onto the monorepo is adopt. It keeps the same database, uploads, .env, ports and nginx site; it only swaps the code and PM2 underneath, with a backup and an undo.

A New install can't take over an existing database: it refuses a database or role that already exists, so that it can never overwrite a church's data. "Install new and point it at the old database" would therefore stop at that check.

For each site:

  • Servers → Installs → Link existing: the server, its folder name and its two ports.
  • Move onto the monorepo (an adopt job): planned at least 4 hours ahead; the church gets the email and banner. Start with the demo site.
  • After it: the site answers, crosscode status on the server shows it on events@2.3.0, its PM2 runs as pm2-<user> under systemd.
  • If something's wrong: the job's page has the log. crosscode adopt <church> --undo on the server puts the old install back (for 14 days).

5. Updates from then on

  • Updates → pick installs and a version → a time at least 4 hours ahead (or an emergency update: 15 minutes, a reason and your 2FA code). A batch runs one church at a time and stops at the first failure; a failed update rolls back by itself.
  • An agent that's offline when a job is due doesn't run it late: the job is missed and you plan it again (the church gets a new notice).

6. New churches

  • First, a throwaway church on a real server (a test domain): New install → the report email with the FASTPANEL steps → add the nginx site with its two ports → Proxy done → live. Uninstall it after (it exports first; check the download link works).
  • Then real churches the same way.

7. Decommissioning the old setup

Only after every site is adopted and has run on 2.3.0 for a while:

  • Each adopted site keeps its old folder 14 days for adopt --undo. After that, crosscode status warns about it and shows the rm -rf to remove it; once removed, undo isn't possible.
  • The archived hope-events repo stays read-only (its v… tags are the history).
  • Old PM2 processes or cron jobs from the hand-run deploys: check pm2 ls for each site user shows only <church>-events-backend and <church>-events-frontend.

If something goes wrong

Symptom Look at
A server doesn't show on the Hub journalctl -u crosscode-agent; the server can reach HUB_PUBLIC_URL
A job is refused by the agent The job key changed (the server must join again) or the server's clock is off
A new install's tunnels don't open journalctl -u frpc; Hub → Jobs → the central check-ports job
A site is down after an agent restart systemctl status pm2-<user> (sites run under their own unit, not the agent's)
An update failed Hub → Jobs → its log; it rolled back. A failed migration is marked rolled back, so the next update isn't blocked