Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ls1vMdni7EfytGYqQTprME
5.8 KiB
Rollout: Hub 2.1, the agent and Events 2.3 on the real servers
The order to move the live servers onto the agent, so that installs, updates and uninstalls are run from the Hub. Each step can be stopped after without leaving anything half done. Tested end to end on a VM (WSL: a church server, a central server with frps, the Hub) on 2026-11-17; the real servers add FASTPANEL, real DNS and Let's Encrypt, which the VM didn't have, so step 6 starts with a throwaway church.
Releases: events@2.3.0, contracts@1.3.0, agent@1.0.0 (crosscode-platform), Hub v2.1.0,
crosscode-site v1.3.0.
Before you start
- A backup of the Hub's database and
backend/keys/(the licence key). - A backup of each church's database (
pg_dump) anduploads/.adoptanddeploytake their own backups too, but keep one that's yours. - Know your frp port range (
allow_portsinfrps.ini) and which ports your hand-written tunnels use.
1. The Hub (2.1.0)
Follow events-hub/deployment.md → "Upgrading to 2.1":
npm run generate-keys -- --jobinbackend/; back upkeys/hub-job-key.pemwith the licence key. Agents pin this key when they join: losing or replacing it means every server joins again..env:HUB_JOB_SIGNING_KEY_FILE=./keys/hub-job-key.pemandHUB_PUBLIC_URL=https://hub.crosscode.co.za.bash scripts/deploy.sh v2.1.0.- Servers → Ports & defaults: the port range, the central server's IP, the install defaults. Mark every port a hand-written tunnel uses as taken.
2. The central server (frps)
frps.inihasproxy_bind_addr = 127.0.0.1(tunnelled ports are then only reachable by nginx on that server). Restart frps and check every FASTPANEL site still answers.- Hub → Servers → Add server → Central; run the command it shows as root. It's read-only: it checks a new install's ports in frps and watches frps.
- Within a minute the server shows on Servers with its stats.
3. Each church server
- Hub → Servers → Add server → Church; run the command as root. It installs Node tools,
the agent (
/opt/crosscode/agent) and the crosscode script (/opt/crosscode/tools, from main), and startscrosscode-agent. The agent listens on nothing. - On the server's page: its configuration.
TRUST_PROXY=loopbackbehind frp; the database:DB_HOST/DB_PORT/DB_ADMIN_URLfor a separate PostgreSQL, or nothing for the local one. systemctl status crosscode-agentis active; the server's stats show on the Hub.- The first time the agent changes
frpc.iniit adds the admin lines and restarts frpc once (every tunnel on that server drops for a few seconds). Do the first install on a quiet evening. Later changes only reload frpc.
4. Existing sites (the demo site and the churches): adopt, don't reinstall
Moving a site you already run onto the monorepo is adopt. It keeps the same database, uploads,
.env, ports and nginx site; it only swaps the code and PM2 underneath, with a backup and an undo.
A New install can't take over an existing database: it refuses a database or role that already exists, so that it can never overwrite a church's data. "Install new and point it at the old database" would therefore stop at that check.
For each site:
- Servers → Installs → Link existing: the server, its folder name and its two ports.
- Move onto the monorepo (an
adoptjob): planned at least 4 hours ahead; the church gets the email and banner. Start with the demo site. - After it: the site answers,
crosscode statuson the server shows it onevents@2.3.0, its PM2 runs aspm2-<user>under systemd. - If something's wrong: the job's page has the log.
crosscode adopt <church> --undoon the server puts the old install back (for 14 days).
5. Updates from then on
- Updates → pick installs and a version → a time at least 4 hours ahead (or an emergency update: 15 minutes, a reason and your 2FA code). A batch runs one church at a time and stops at the first failure; a failed update rolls back by itself.
- An agent that's offline when a job is due doesn't run it late: the job is missed and you plan it again (the church gets a new notice).
6. New churches
- First, a throwaway church on a real server (a test domain): New install → the report email with the FASTPANEL steps → add the nginx site with its two ports → Proxy done → live. Uninstall it after (it exports first; check the download link works).
- Then real churches the same way.
7. Decommissioning the old setup
Only after every site is adopted and has run on 2.3.0 for a while:
- Each adopted site keeps its old folder 14 days for
adopt --undo. After that,crosscode statuswarns about it and shows therm -rfto remove it; once removed, undo isn't possible. - The archived
hope-eventsrepo stays read-only (itsv…tags are the history). - Old PM2 processes or cron jobs from the hand-run deploys: check
pm2 lsfor each site user shows only<church>-events-backendand<church>-events-frontend.
If something goes wrong
| Symptom | Look at |
|---|---|
| A server doesn't show on the Hub | journalctl -u crosscode-agent; the server can reach HUB_PUBLIC_URL |
| A job is refused by the agent | The job key changed (the server must join again) or the server's clock is off |
| A new install's tunnels don't open | journalctl -u frpc; Hub → Jobs → the central check-ports job |
| A site is down after an agent restart | systemctl status pm2-<user> (sites run under their own unit, not the agent's) |
| An update failed | Hub → Jobs → its log; it rolled back. A failed migration is marked rolled back, so the next update isn't blocked |