From d985a4c8536359fa89f832ff223c2b7edf204efc Mon Sep 17 00:00:00 2001 From: Joshua Oosthuizen Date: Tue, 17 Nov 2026 22:29:06 +0200 Subject: [PATCH] docs: rollout checklist for Hub 2.1, the agent and Events 2.3 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Ls1vMdni7EfytGYqQTprME --- docs/ROLLOUT-1.0.md | 105 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 docs/ROLLOUT-1.0.md diff --git a/docs/ROLLOUT-1.0.md b/docs/ROLLOUT-1.0.md new file mode 100644 index 0000000..99d668f --- /dev/null +++ b/docs/ROLLOUT-1.0.md @@ -0,0 +1,105 @@ +# Rollout: Hub 2.1, the agent and Events 2.3 on the real servers + +The order to move the live servers onto the agent, so that installs, updates and uninstalls are run +from the Hub. Each step can be stopped after without leaving anything half done. Tested end to end +on a VM (WSL: a church server, a central server with frps, the Hub) on 2026-11-17; the real servers +add FASTPANEL, real DNS and Let's Encrypt, which the VM didn't have, so step 6 starts with a +throwaway church. + +Releases: `events@2.3.0`, `contracts@1.3.0`, `agent@1.0.0` (crosscode-platform), Hub `v2.1.0`, +crosscode-site `v1.3.0`. + +## Before you start + +- [ ] A backup of the Hub's database and `backend/keys/` (the licence key). +- [ ] A backup of each church's database (`pg_dump`) and `uploads/`. `adopt` and `deploy` take + their own backups too, but keep one that's yours. +- [ ] Know your frp port range (`allow_ports` in `frps.ini`) and which ports your hand-written + tunnels use. + +## 1. The Hub (2.1.0) + +Follow `events-hub/deployment.md` → "Upgrading to 2.1": + +- [ ] `npm run generate-keys -- --job` in `backend/`; back up `keys/hub-job-key.pem` with the + licence key. **Agents pin this key when they join**: losing or replacing it means every server + joins again. +- [ ] `.env`: `HUB_JOB_SIGNING_KEY_FILE=./keys/hub-job-key.pem` and + `HUB_PUBLIC_URL=https://hub.crosscode.co.za`. +- [ ] `bash scripts/deploy.sh v2.1.0`. +- [ ] Servers → Ports & defaults: the port range, the central server's IP, the install defaults. + Mark every port a hand-written tunnel uses as taken. + +## 2. The central server (frps) + +- [ ] `frps.ini` has `proxy_bind_addr = 127.0.0.1` (tunnelled ports are then only reachable by + nginx on that server). Restart frps and check every FASTPANEL site still answers. +- [ ] Hub → Servers → Add server → **Central**; run the command it shows as root. It's read-only: + it checks a new install's ports in frps and watches frps. +- [ ] Within a minute the server shows on Servers with its stats. + +## 3. Each church server + +- [ ] Hub → Servers → Add server → **Church**; run the command as root. It installs Node tools, + the agent (`/opt/crosscode/agent`) and the crosscode script (`/opt/crosscode/tools`, from + main), and starts `crosscode-agent`. The agent listens on nothing. +- [ ] On the server's page: its configuration. `TRUST_PROXY=loopback` behind frp; the database: + `DB_HOST`/`DB_PORT`/`DB_ADMIN_URL` for a separate PostgreSQL, or nothing for the local one. +- [ ] `systemctl status crosscode-agent` is active; the server's stats show on the Hub. +- [ ] The first time the agent changes `frpc.ini` it adds the admin lines and **restarts frpc once** + (every tunnel on that server drops for a few seconds). Do the first install on a quiet + evening. Later changes only reload frpc. + +## 4. Existing sites (the demo site and the churches): adopt, don't reinstall + +Moving a site you already run onto the monorepo is `adopt`. It keeps the same database, uploads, +`.env`, ports and nginx site; it only swaps the code and PM2 underneath, with a backup and an undo. + +A New install can't take over an existing database: it refuses a database or role that already +exists, so that it can never overwrite a church's data. "Install new and point it at the old +database" would therefore stop at that check. + +For each site: + +- [ ] Servers → Installs → **Link existing**: the server, its folder name and its two ports. +- [ ] **Move onto the monorepo** (an `adopt` job): planned at least 4 hours ahead; the church gets + the email and banner. Start with the demo site. +- [ ] After it: the site answers, `crosscode status` on the server shows it on `events@2.3.0`, its + PM2 runs as `pm2-` under systemd. +- [ ] If something's wrong: the job's page has the log. `crosscode adopt --undo` on the + server puts the old install back (for 14 days). + +## 5. Updates from then on + +- [ ] Updates → pick installs and a version → a time at least 4 hours ahead (or an emergency + update: 15 minutes, a reason and your 2FA code). A batch runs one church at a time and stops at + the first failure; a failed update rolls back by itself. +- [ ] An agent that's offline when a job is due doesn't run it late: the job is **missed** and + you plan it again (the church gets a new notice). + +## 6. New churches + +- [ ] First, a throwaway church on a real server (a test domain): New install → the report email + with the FASTPANEL steps → add the nginx site with its two ports → **Proxy done** → live. + Uninstall it after (it exports first; check the download link works). +- [ ] Then real churches the same way. + +## 7. Decommissioning the old setup + +Only after every site is adopted and has run on 2.3.0 for a while: + +- [ ] Each adopted site keeps its old folder 14 days for `adopt --undo`. After that, `crosscode + status` warns about it and shows the `rm -rf` to remove it; once removed, undo isn't possible. +- [ ] The archived `hope-events` repo stays read-only (its `v…` tags are the history). +- [ ] Old PM2 processes or cron jobs from the hand-run deploys: check `pm2 ls` for each site user + shows only `-events-backend` and `-events-frontend`. + +## If something goes wrong + +| Symptom | Look at | +|---|---| +| A server doesn't show on the Hub | `journalctl -u crosscode-agent`; the server can reach `HUB_PUBLIC_URL` | +| A job is refused by the agent | The job key changed (the server must join again) or the server's clock is off | +| A new install's tunnels don't open | `journalctl -u frpc`; Hub → Jobs → the central `check-ports` job | +| A site is down after an agent restart | `systemctl status pm2-` (sites run under their own unit, not the agent's) | +| An update failed | Hub → Jobs → its log; it rolled back. A failed migration is marked rolled back, so the next update isn't blocked |