docs: rollout checklist for Hub 2.1, the agent and Events 2.3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ls1vMdni7EfytGYqQTprME
This commit is contained in:
2026-11-17 22:29:06 +02:00
co-authored by Claude Opus 5.5
parent eb4e44a735
commit d985a4c853
+105
View File
@@ -0,0 +1,105 @@
# Rollout: Hub 2.1, the agent and Events 2.3 on the real servers
The order to move the live servers onto the agent, so that installs, updates and uninstalls are run
from the Hub. Each step can be stopped after without leaving anything half done. Tested end to end
on a VM (WSL: a church server, a central server with frps, the Hub) on 2026-11-17; the real servers
add FASTPANEL, real DNS and Let's Encrypt, which the VM didn't have, so step 6 starts with a
throwaway church.
Releases: `events@2.3.0`, `contracts@1.3.0`, `agent@1.0.0` (crosscode-platform), Hub `v2.1.0`,
crosscode-site `v1.3.0`.
## Before you start
- [ ] A backup of the Hub's database and `backend/keys/` (the licence key).
- [ ] A backup of each church's database (`pg_dump`) and `uploads/`. `adopt` and `deploy` take
their own backups too, but keep one that's yours.
- [ ] Know your frp port range (`allow_ports` in `frps.ini`) and which ports your hand-written
tunnels use.
## 1. The Hub (2.1.0)
Follow `events-hub/deployment.md` → "Upgrading to 2.1":
- [ ] `npm run generate-keys -- --job` in `backend/`; back up `keys/hub-job-key.pem` with the
licence key. **Agents pin this key when they join**: losing or replacing it means every server
joins again.
- [ ] `.env`: `HUB_JOB_SIGNING_KEY_FILE=./keys/hub-job-key.pem` and
`HUB_PUBLIC_URL=https://hub.crosscode.co.za`.
- [ ] `bash scripts/deploy.sh v2.1.0`.
- [ ] Servers → Ports & defaults: the port range, the central server's IP, the install defaults.
Mark every port a hand-written tunnel uses as taken.
## 2. The central server (frps)
- [ ] `frps.ini` has `proxy_bind_addr = 127.0.0.1` (tunnelled ports are then only reachable by
nginx on that server). Restart frps and check every FASTPANEL site still answers.
- [ ] Hub → Servers → Add server → **Central**; run the command it shows as root. It's read-only:
it checks a new install's ports in frps and watches frps.
- [ ] Within a minute the server shows on Servers with its stats.
## 3. Each church server
- [ ] Hub → Servers → Add server → **Church**; run the command as root. It installs Node tools,
the agent (`/opt/crosscode/agent`) and the crosscode script (`/opt/crosscode/tools`, from
main), and starts `crosscode-agent`. The agent listens on nothing.
- [ ] On the server's page: its configuration. `TRUST_PROXY=loopback` behind frp; the database:
`DB_HOST`/`DB_PORT`/`DB_ADMIN_URL` for a separate PostgreSQL, or nothing for the local one.
- [ ] `systemctl status crosscode-agent` is active; the server's stats show on the Hub.
- [ ] The first time the agent changes `frpc.ini` it adds the admin lines and **restarts frpc once**
(every tunnel on that server drops for a few seconds). Do the first install on a quiet
evening. Later changes only reload frpc.
## 4. Existing sites (the demo site and the churches): adopt, don't reinstall
Moving a site you already run onto the monorepo is `adopt`. It keeps the same database, uploads,
`.env`, ports and nginx site; it only swaps the code and PM2 underneath, with a backup and an undo.
A New install can't take over an existing database: it refuses a database or role that already
exists, so that it can never overwrite a church's data. "Install new and point it at the old
database" would therefore stop at that check.
For each site:
- [ ] Servers → Installs → **Link existing**: the server, its folder name and its two ports.
- [ ] **Move onto the monorepo** (an `adopt` job): planned at least 4 hours ahead; the church gets
the email and banner. Start with the demo site.
- [ ] After it: the site answers, `crosscode status` on the server shows it on `events@2.3.0`, its
PM2 runs as `pm2-<user>` under systemd.
- [ ] If something's wrong: the job's page has the log. `crosscode adopt <church> --undo` on the
server puts the old install back (for 14 days).
## 5. Updates from then on
- [ ] Updates → pick installs and a version → a time at least 4 hours ahead (or an emergency
update: 15 minutes, a reason and your 2FA code). A batch runs one church at a time and stops at
the first failure; a failed update rolls back by itself.
- [ ] An agent that's offline when a job is due doesn't run it late: the job is **missed** and
you plan it again (the church gets a new notice).
## 6. New churches
- [ ] First, a throwaway church on a real server (a test domain): New install → the report email
with the FASTPANEL steps → add the nginx site with its two ports → **Proxy done** → live.
Uninstall it after (it exports first; check the download link works).
- [ ] Then real churches the same way.
## 7. Decommissioning the old setup
Only after every site is adopted and has run on 2.3.0 for a while:
- [ ] Each adopted site keeps its old folder 14 days for `adopt --undo`. After that, `crosscode
status` warns about it and shows the `rm -rf` to remove it; once removed, undo isn't possible.
- [ ] The archived `hope-events` repo stays read-only (its `v…` tags are the history).
- [ ] Old PM2 processes or cron jobs from the hand-run deploys: check `pm2 ls` for each site user
shows only `<church>-events-backend` and `<church>-events-frontend`.
## If something goes wrong
| Symptom | Look at |
|---|---|
| A server doesn't show on the Hub | `journalctl -u crosscode-agent`; the server can reach `HUB_PUBLIC_URL` |
| A job is refused by the agent | The job key changed (the server must join again) or the server's clock is off |
| A new install's tunnels don't open | `journalctl -u frpc`; Hub → Jobs → the central `check-ports` job |
| A site is down after an agent restart | `systemctl status pm2-<user>` (sites run under their own unit, not the agent's) |
| An update failed | Hub → Jobs → its log; it rolled back. A failed migration is marked rolled back, so the next update isn't blocked |