Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ls1vMdni7EfytGYqQTprME
106 lines
5.8 KiB
Markdown
106 lines
5.8 KiB
Markdown
# Rollout: Hub 2.1, the agent and Events 2.3 on the real servers
|
|
|
|
The order to move the live servers onto the agent, so that installs, updates and uninstalls are run
|
|
from the Hub. Each step can be stopped after without leaving anything half done. Tested end to end
|
|
on a VM (WSL: a church server, a central server with frps, the Hub) on 2026-11-17; the real servers
|
|
add FASTPANEL, real DNS and Let's Encrypt, which the VM didn't have, so step 6 starts with a
|
|
throwaway church.
|
|
|
|
Releases: `events@2.3.0`, `contracts@1.3.0`, `agent@1.0.0` (crosscode-platform), Hub `v2.1.0`,
|
|
crosscode-site `v1.3.0`.
|
|
|
|
## Before you start
|
|
|
|
- [ ] A backup of the Hub's database and `backend/keys/` (the licence key).
|
|
- [ ] A backup of each church's database (`pg_dump`) and `uploads/`. `adopt` and `deploy` take
|
|
their own backups too, but keep one that's yours.
|
|
- [ ] Know your frp port range (`allow_ports` in `frps.ini`) and which ports your hand-written
|
|
tunnels use.
|
|
|
|
## 1. The Hub (2.1.0)
|
|
|
|
Follow `events-hub/deployment.md` → "Upgrading to 2.1":
|
|
|
|
- [ ] `npm run generate-keys -- --job` in `backend/`; back up `keys/hub-job-key.pem` with the
|
|
licence key. **Agents pin this key when they join**: losing or replacing it means every server
|
|
joins again.
|
|
- [ ] `.env`: `HUB_JOB_SIGNING_KEY_FILE=./keys/hub-job-key.pem` and
|
|
`HUB_PUBLIC_URL=https://hub.crosscode.co.za`.
|
|
- [ ] `bash scripts/deploy.sh v2.1.0`.
|
|
- [ ] Servers → Ports & defaults: the port range, the central server's IP, the install defaults.
|
|
Mark every port a hand-written tunnel uses as taken.
|
|
|
|
## 2. The central server (frps)
|
|
|
|
- [ ] `frps.ini` has `proxy_bind_addr = 127.0.0.1` (tunnelled ports are then only reachable by
|
|
nginx on that server). Restart frps and check every FASTPANEL site still answers.
|
|
- [ ] Hub → Servers → Add server → **Central**; run the command it shows as root. It's read-only:
|
|
it checks a new install's ports in frps and watches frps.
|
|
- [ ] Within a minute the server shows on Servers with its stats.
|
|
|
|
## 3. Each church server
|
|
|
|
- [ ] Hub → Servers → Add server → **Church**; run the command as root. It installs Node tools,
|
|
the agent (`/opt/crosscode/agent`) and the crosscode script (`/opt/crosscode/tools`, from
|
|
main), and starts `crosscode-agent`. The agent listens on nothing.
|
|
- [ ] On the server's page: its configuration. `TRUST_PROXY=loopback` behind frp; the database:
|
|
`DB_HOST`/`DB_PORT`/`DB_ADMIN_URL` for a separate PostgreSQL, or nothing for the local one.
|
|
- [ ] `systemctl status crosscode-agent` is active; the server's stats show on the Hub.
|
|
- [ ] The first time the agent changes `frpc.ini` it adds the admin lines and **restarts frpc once**
|
|
(every tunnel on that server drops for a few seconds). Do the first install on a quiet
|
|
evening. Later changes only reload frpc.
|
|
|
|
## 4. Existing sites (the demo site and the churches): adopt, don't reinstall
|
|
|
|
Moving a site you already run onto the monorepo is `adopt`. It keeps the same database, uploads,
|
|
`.env`, ports and nginx site; it only swaps the code and PM2 underneath, with a backup and an undo.
|
|
|
|
A New install can't take over an existing database: it refuses a database or role that already
|
|
exists, so that it can never overwrite a church's data. "Install new and point it at the old
|
|
database" would therefore stop at that check.
|
|
|
|
For each site:
|
|
|
|
- [ ] Servers → Installs → **Link existing**: the server, its folder name and its two ports.
|
|
- [ ] **Move onto the monorepo** (an `adopt` job): planned at least 4 hours ahead; the church gets
|
|
the email and banner. Start with the demo site.
|
|
- [ ] After it: the site answers, `crosscode status` on the server shows it on `events@2.3.0`, its
|
|
PM2 runs as `pm2-<user>` under systemd.
|
|
- [ ] If something's wrong: the job's page has the log. `crosscode adopt <church> --undo` on the
|
|
server puts the old install back (for 14 days).
|
|
|
|
## 5. Updates from then on
|
|
|
|
- [ ] Updates → pick installs and a version → a time at least 4 hours ahead (or an emergency
|
|
update: 15 minutes, a reason and your 2FA code). A batch runs one church at a time and stops at
|
|
the first failure; a failed update rolls back by itself.
|
|
- [ ] An agent that's offline when a job is due doesn't run it late: the job is **missed** and
|
|
you plan it again (the church gets a new notice).
|
|
|
|
## 6. New churches
|
|
|
|
- [ ] First, a throwaway church on a real server (a test domain): New install → the report email
|
|
with the FASTPANEL steps → add the nginx site with its two ports → **Proxy done** → live.
|
|
Uninstall it after (it exports first; check the download link works).
|
|
- [ ] Then real churches the same way.
|
|
|
|
## 7. Decommissioning the old setup
|
|
|
|
Only after every site is adopted and has run on 2.3.0 for a while:
|
|
|
|
- [ ] Each adopted site keeps its old folder 14 days for `adopt --undo`. After that, `crosscode
|
|
status` warns about it and shows the `rm -rf` to remove it; once removed, undo isn't possible.
|
|
- [ ] The archived `hope-events` repo stays read-only (its `v…` tags are the history).
|
|
- [ ] Old PM2 processes or cron jobs from the hand-run deploys: check `pm2 ls` for each site user
|
|
shows only `<church>-events-backend` and `<church>-events-frontend`.
|
|
|
|
## If something goes wrong
|
|
|
|
| Symptom | Look at |
|
|
|---|---|
|
|
| A server doesn't show on the Hub | `journalctl -u crosscode-agent`; the server can reach `HUB_PUBLIC_URL` |
|
|
| A job is refused by the agent | The job key changed (the server must join again) or the server's clock is off |
|
|
| A new install's tunnels don't open | `journalctl -u frpc`; Hub → Jobs → the central `check-ports` job |
|
|
| A site is down after an agent restart | `systemctl status pm2-<user>` (sites run under their own unit, not the agent's) |
|
|
| An update failed | Hub → Jobs → its log; it rolled back. A failed migration is marked rolled back, so the next update isn't blocked |
|