Files
crosscode-platform/docs/AGENT-1.0-PLAN.md
T
joshuaandClaude Opus 5.5 c398e8071d Docs: server agent and Hub 2.1.0 plan (phase 1d, decisions 27-37)
Agent on every server runs installs, scheduled/emergency updates and
uninstalls on signed jobs from the Hub, polls only (no listening port),
appends to frpc.ini in marked blocks, builds .env files from per-server
config in the Hub. FastPanel and DNS stay manual (emailed instructions).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:26:51 +02:00

478 lines
29 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Server agent 1.0.0 + Hub 2.1.0 — installs, updates and uninstalls from the Hub
Planned 2026-11-12 (ROADMAP-v3 phase **1d**, decisions 27–37). Goal: when a new church signs up,
Joshua picks a server in the Hub and the rest happens by itself, with a report by email. Updates
are scheduled from the Hub with notice to the church, and every server's health is visible there.
This replaces the old rule "the Hub never deploys" (decisions 10 and 23). The part that matters
stays: **no site depends on the Hub to keep running**, and everything still goes through the
`crosscode` script (EVENTS-2.2-PLAN §4), which keeps working by hand over SSH.
## Contents
1. [The shape](#1-the-shape)
2. [Rules](#2-rules)
3. [The agent](#3-the-agent)
4. [Jobs and their security](#4-jobs-and-their-security)
5. [Configuration: server, defaults, per install](#5-configuration-server-defaults-per-install)
6. [frp and the central server](#6-frp-and-the-central-server)
7. [New install](#7-new-install)
8. [Updates: scheduled, emergency, batch](#8-updates-scheduled-emergency-batch)
9. [Uninstall](#9-uninstall)
10. [Servers page and alerts](#10-servers-page-and-alerts)
11. [Reports and emails](#11-reports-and-emails)
12. [What changes in each repo](#12-what-changes-in-each-repo)
13. [Testing](#13-testing)
14. [Rolling it out](#14-rolling-it-out)
15. [Decisions](#decisions)
---
## 1. The shape
```
Cross Code Hub (hub.crosscode.co.za)
servers · jobs · schedules · port list · server config · reports
▲ agents poll: "any jobs?" + stats + job results (outbound HTTPS only)
│
┌────────┴─────────────┐ ┌──────────────────────┐ ┌─────────────────────────────┐
│ Church server A │ │ Church server B │ │ Central server (FastPanel) │
│ crosscode-agent │ │ crosscode-agent │ │ crosscode-agent --central │
│ └ runs `crosscode` │ │ └ runs `crosscode` │ │ read-only: frps ports, │
│ PM2 sites, Postgres │ │ … │ │ allow_ports, bind address │
│ frpc ═══tunnel═══════╪═══╪══════════════════════╪══▶│ frps ◀─ nginx (FastPanel, │
└──────────────────────┘ └──────────────────────┘ │ set up by Joshua) │
└─────────────────────────────┘
```
## 2. Rules
1. **The agent listens on nothing.** No HTTP server, no open port, no URL, not even on
`127.0.0.1`. It only makes outgoing HTTPS requests to the Hub (poll, stats, results). A test
checks that the running agent has no listening socket.
2. **The agent only runs the `crosscode` script**, with arguments built from a fixed list of job
types (§4). There is no "run this command" job.
3. **Jobs are signed by the Hub** and checked by the agent against the Hub key pinned when the
server joined. Unsigned, expired or replayed jobs are refused and reported.
4. **No site depends on the Hub or the agent.** If either is down, sites keep running and
`crosscode` still works by hand. Scheduled jobs that can't be delivered are reported as missed,
never run late without a fresh notice (§8).
5. **The agent never touches FastPanel or DNS.** FastPanel setup is an email to Joshua (§7, §11).
6. **Files the agent edits are only ever added to, never replaced**: `/etc/frp/frpc.ini` and each
install's `.env` (§5, §6). Anything written by hand is never changed.
7. **Secrets generated for an install stay on its server** (`JWT_SECRET`, the database password).
They're never sent to the Hub, never put in a report or email.
## 3. The agent
- **Where:** `apps/agent` in this monorepo, Node 24 + TypeScript, tagged `agent@1.0.0`. Installed
in `/opt/crosscode/agent` with its config in `/etc/crosscode/agent.json` (mode 600, root).
- **Run by systemd** (`crosscode-agent.service`, restarts on failure), not PM2, because it manages
PM2. Runs as root, as `crosscode install` needs (PostgreSQL, PM2 for the app user, frpc).
- **Joining:** Hub → Servers → **Add server** gives a one-time token (valid 1 hour). On the server:
`curl … | bash -s -- --hub https://hub.crosscode.co.za --token …` installs Node if needed, the
agent and `/opt/crosscode/tools`, then the agent:
1. makes an **Ed25519 key pair** (signs its requests to the Hub) and an **X25519 key pair**
(receives encrypted config, §5); the private keys never leave the server,
2. sends the token and both public keys; the Hub replies with the server id and **the Hub's
job-signing public key**, which the agent pins.
- **Polling:** every 30 s `POST /api/agent/v1/poll` (signed) with a short status; the reply holds
any jobs due. Every 60 s the stats (§10). Requests back off when the Hub doesn't answer.
- **Modes:** `church` (default) and `central` (§6). The central agent can't run any job but
`check-ports` and `status`.
- **Updating itself:** an `update-agent` job checks out the newest `agent@…` tag, builds it next
to the running copy, switches the systemd unit over and reports its new version. If the new one
doesn't poll within 2 minutes, a systemd timer starts the previous one again.
- **One job at a time per server**, in the order the Hub gives them. A job has a time limit
(install 30 min, deploy 20 min); a job past it is stopped and reported as failed.
## 4. Jobs and their security
**Job types** (and the `crosscode` command each runs):
| Job | Runs | Needs 2FA again in the Hub |
|-----|------|:-:|
| `install` | `crosscode install <church> --products … --json` | no |
| `add-product` | `crosscode add <church> <product> --json` | no |
| `deploy` | `crosscode deploy <church> [products] [--version …] --json` | emergency only |
| `adopt` / `adopt-undo` | `crosscode adopt …` | no |
| `restart` | `crosscode restart <church> [product]` (PM2 reload) | no |
| `export` | `crosscode export <church> --json` (§9) | no |
| `uninstall` | `crosscode uninstall <church> --json` | **yes**, and typing the short name |
| `purge` | `crosscode purge <church>` (deletes the 30-day keep, §9) | no (scheduled at uninstall) |
| `update-agent` | §3 | no |
| `check-ports` | central only (§6) | no |
| `status` | `crosscode status --json` | no |
**A signed job** (`crosscode.agent-job/1`, in `packages/contracts`):
```json
{
"schema": "crosscode.agent-job/1",
"id": "job_…",
"serverId": "srv_…",
"type": "deploy",
"args": { "church": "hope", "products": ["events"], "version": "events@2.4.1" },
"config": { "alg": "x25519-xsalsa20-poly1305", "ciphertext": "…" },
"notBefore": "2026-11-20T12:00:00Z",
"expiresAt": "2026-11-20T12:15:00Z",
"signature": "…"
}
```
- The agent checks: the signature (pinned Hub key), `serverId` is its own, it's within
`notBefore`–`expiresAt`, the id hasn't been seen (it keeps the ids of the last 90 days), the
type is known and every `args` value passes that type's schema (short names
`^[a-z][a-z0-9-]{1,30}$`, products from the enum, versions as `<product>@x.y.z`). Arguments are
passed to `crosscode` as an argument array, never through a shell.
- `config` holds whatever the job needs from §5, encrypted to **this server's** X25519 key, so
job rows and logs in the Hub never hold it in plain text.
- **Results** (`crosscode.agent-report/1`): the job id, status (`succeeded`, `failed`,
`rolled-back`, `refused`, `expired`), each step with its start, end and outcome, and the last
200 log lines. The script's `--json` output is that report; the agent adds the log lines and
sends it, signed with its own key. Logs are filtered for anything that looks like a secret
before they leave the server.
- **The Hub side:** job-signing private key in the Hub's `.env` (not in the database), its own
key separate from the licence-signing key. 2FA re-confirmation for uninstall and emergency
deploys. Every job is in the audit log with who made it.
## 5. Configuration: server, defaults, per install
An install's `.env` files are built from four layers. Each variable comes from exactly one layer.
| Layer | Set where | Examples | Sent how |
|-------|-----------|----------|----------|
| **Server** | Hub → Servers → *server* → Configuration, **once per server** (editable) | Database host, port and an admin role that can create databases and roles (or "local PostgreSQL, through `sudo -u postgres`"), `TRUST_PROXY`, app user, backup folder | Stored AES-GCM encrypted in the Hub (as install secrets are today); sent to the agent encrypted to its key with each job that needs it. The agent doesn't keep it on disk |
| **Defaults** | Hub → Settings → Install defaults, once | `HUB_URL`, `APP_TIMEZONE` (Africa/Johannesburg), job switches (`DAILY_SUMMARY_ENABLED=true`, `BACKUP_ENABLED=true`, …), `HUB_HEARTBEAT_ENABLED=true`, optional platform `SENTRY_DSN` | With the job, encrypted the same way |
| **Install** | The New install form (§7) | Domain → `FRONTEND_URL`, `APP_BASE_URL`, `BACKEND_URL` and the frontend's `NEXT_PUBLIC_API_URL` (all `https://<domain>`, since FastPanel sends `/api` to the backend on the same domain); timezone if not the default; the licence key; ports from the port list (§6) → `PORT` | With the job |
| **Generated** | On the server by `crosscode install` | `JWT_SECRET`, the database password (so `DATABASE_URL`), `NODE_ENV=production` | Never leaves the server (rule 7) |
**Per-product env template.** Each product has `apps/<product>/env.template.json` in the
monorepo, listing every variable with its layer, whether it's required, and a default:
```json
{ "schema": "crosscode.env-template/1", "product": "events", "files": {
"backend/.env": [
{ "name": "DATABASE_URL", "layer": "generated", "required": true },
{ "name": "FRONTEND_URL", "layer": "install", "from": "https://{domain}", "required": true },
{ "name": "TRUST_PROXY", "layer": "server", "required": true },
{ "name": "BACKUP_ENABLED", "layer": "defaults", "default": "true" }
],
"frontend/.env.production": [
{ "name": "NEXT_PUBLIC_API_URL", "layer": "install", "from": "https://{domain}", "required": true }
] } }
```
- The Hub reads the template of the release being installed (synced with the release, §5 of the
roadmap) and **refuses to create the job if a required server or defaults value is missing**,
saying which, before anything runs on the server.
- **Install:** `crosscode install` writes each `.env` from the template (mode 600, app user).
- **Updates:** before building, `crosscode deploy` compares the new release's template with the
`.env`. A new variable is **appended** with its value (and a comment with the date and
release); existing lines are never changed or removed. A new *required* variable with no value
in any layer stops the deploy at the checks, before anything is touched.
- **Changing a server value later** (e.g. a new database host) is a `reconfigure` step of a
deploy: the agent shows which lines would change in each install, and only changes them when
Joshua confirms in the Hub. This is the one exception to "only append", and it never runs by
itself.
- **URLs are checked before and after.** Before: the form checks that the domain's DNS points at
the central server and warns if not (the install still runs). After Joshua marks the proxy done:
the Hub checks `https://<domain>/` (frontend answers), `https://<domain>/health` (backend
answers with the right version) and that `/health` reports the `FRONTEND_URL` it was given.
**For Joshua to set per server before the first install:** the database settings and
`TRUST_PROXY`. Behind frp the app sees every connection from frpc on `127.0.0.1`, and the
visitor's address arrives in `X-Forwarded-For` from FastPanel's nginx, so the expected value is
`loopback` (the Hub suggests it).
## 6. frp and the central server
**Today:** `/etc/frp/frpc.ini` on each church server, `tcp` tunnels, frontend and backend each
with their own `remote_port`. FastPanel on the central server proxies a domain to
`127.0.0.1:<remote_port>` with Let's Encrypt.
**Port list in the Hub.** frps opens every `remote_port` on the one central server, so they must
be unique across every church server. The Hub hands them out from a range (default
**20000–29999**, set in Hub settings to match `allow_ports`), two per product install (web, api),
and **uses the same number for `local_port`**, so it's also free on the church server and there's
one number per tunnel to think about. Ports already in use (today's `3000`/`5000` and anything
the central agent sees listening) are recorded as taken. Ports are released 30 days after an
uninstall.
**Tunnel names** must also be unique on frps: `<church>-<product>-web` and
`<church>-<product>-api` (e.g. `hope-events-web`). Existing entries such as `[events-frontend]`
stay as they are.
**What the agent adds to frpc.ini** (one block per product install, appended at the end):
```ini
# >>> crosscode:hope:events (added by crosscode-agent 2026-11-20, do not edit by hand)
[hope-events-web]
type = tcp
local_ip = 127.0.0.1
local_port = 20010
remote_port = 20010
[hope-events-api]
type = tcp
local_ip = 127.0.0.1
local_port = 20011
remote_port = 20011
# <<< crosscode:hope:events
```
- **Only appended to, never replaced.** Before a change: a copy to
`/etc/frp/backups/frpc.ini.<timestamp>`; write to a temporary file; `frpc verify -c`; then move
it into place. If the check fails, nothing is changed. Uninstall removes only the lines between
that install's own markers; nothing outside a marked block is ever changed.
- **Reload without dropping other sites.** Once per server (agreed 2026-11-12) the agent adds to
`[common]`, if they're not there:
```ini
admin_addr = 127.0.0.1
admin_port = 7400
```
so `frpc reload -c /etc/frp/frpc.ini` picks up new tunnels without restarting frpc. This is the
only change ever made in `[common]`, and it's reported. (`admin_addr` is loopback only, so it
adds no way in from outside.)
- The `.ini` format is deprecated in newer frp versions in favour of TOML. The agent reads the
format the server has; moving to TOML is a separate job for later.
**The central agent** (`--central`, on the FastPanel server, read-only) answers `check-ports`:
- the port is inside `allow_ports` in `frps.ini`,
- frps has the tunnel open (the port is listening; frps's dashboard API is used if it's enabled
on `127.0.0.1`),
- **`proxy_bind_addr = 127.0.0.1` is set**, so tunnels are only reachable through FastPanel's
nginx, not straight from the internet on `<central IP>:<port>` (which would bypass HTTPS and
expose the backend). If it's missing, every report and the Servers page show a warning.
It changes nothing on the central server: no frps edits, no firewall changes, no FastPanel.
## 7. New install
**Hub → Customers → *customer* → New install** (or a new customer first):
1. **Form:** products (Events now; Accounts / Website / App as they ship), **short name**
(`hope`, `cross`, `bridge`; checked unique across all servers and used for folders, PM2 names,
database names and tunnel names), address (own domain or Cross Code address; per product the
domain is shown, e.g. `events.hope.co.za`), timezone, **site admin name and email**, and the
**server**: a list showing each server's free RAM, free disk, CPU load over the last day, sites
on it and agent status, with servers below the free-RAM threshold (default 1.5 GB per product
install) marked as not recommended.
2. **Checks in the Hub** before the job exists: the server agent is online, server config and
defaults are complete for the release's env template (§5), ports are free, the short name is
free, DNS (warning only).
3. The Hub **creates the licence(s)**, assigns ports, and queues the `install` job.
4. **On the server** (`crosscode install`, each step in the report):
checks (disk, RAM, Node, PostgreSQL reachable) → folder `/var/www/sites/<church>/<product>`
→ checkout of the release tag → `.env` files (§5) → database and role → install dependencies
→ migrations → build → PM2 processes on the given ports, `pm2 save` → `church.env` → frpc
block and reload (§6) → local `/health` answers with the right version → the site registers
with the Hub using its licence.
5. **On failure** anything this job created is removed again (PM2 processes, frpc block, database
and role, folder), in reverse order, and the report says what failed and what was cleaned up.
6. **Central check:** the Hub asks the central agent to `check-ports` for both ports.
7. **Emails to Joshua:** the install report and, if everything passed, the **FastPanel email**
(§11). The install shows **Waiting for proxy** in the Hub.
8. Joshua sets up FastPanel by hand and clicks **Proxy done**. The Hub checks the public URLs
(§5) and marks it **Live**, or says what didn't answer. When Live, the site admin gets the
welcome email with the setup-wizard link.
## 8. Updates: scheduled, emergency, batch
**Scheduled** (Hub → Install → Schedule update, or Updates → Schedule):
- Joshua picks the release and the time. The time must be **at least 4 hours away**; the Hub
refuses anything sooner. A warning (not a block) for Sunday 06:00–13:00 and for days with an
event of that church running (from the stats).
- Straight away: an email to the site admin(s) ("Cross Events will be updated on Thursday 20
November at 14:00; the site will be unavailable for a few minutes", plus the what's-new
summary), and a **banner** through the existing Notices, starting now and ending after the
update. A reminder email 1 hour before.
- At the time the Hub releases the `deploy` job (`notBefore` = the time, `expiresAt` = 15 min
later). If the agent doesn't pick it up in that window the update is **missed**: nothing runs,
Joshua is emailed, and a new time needs a new notice.
- Afterwards: the report to Joshua; to the site admin "Updated to 2.4.1" with what's new, or
"The update didn't go through; the site is running the previous version" if it rolled back.
- Cancelling or moving it updates the banner and emails the site admin.
**Emergency:** the same, with at least **15 minutes'** notice, a required reason (shown in the
email and banner), and 2FA again. Meant for security fixes and broken releases.
**Batch:** "every Events install to 2.4.1" (or a selection), at one time with one notice. The
Hub runs them **one at a time** (on each server, and across servers) and **stops the batch at the
first failure**, emailing Joshua; the remaining installs get a "postponed" email and their banner
is removed.
**Rules carried over:** the release's requirements (`— requires accounts 1.2.0`, decision 24)
are checked in the Hub before scheduling and again by the script; each product rolls back on its
own (§8 of the roadmap). A church's **Request update** now shows as "Requested" next to
Schedule update. Moving a 2.1.x install to 2.2.0 is an `adopt` job, scheduled the same way.
## 9. Uninstall
Hub → Install → **Uninstall** (2FA again, type the short name):
1. Joshua picks the date (at least 4 hours away, default 7 days) and the site admin is emailed
and gets a banner, as for an update.
2. **Export first** (`crosscode export`): the church's data as **JSON in a format of our own**,
`crosscode.church-export/1` (defined in `packages/contracts`): events, sessions, ticket types,
registrations and attendees, people (name, email, phone, consent), payments (amounts, dates,
references), check-ins, rosters, settings, and the uploads as files next to it. Written in
plain terms with our own field names; **no database tables, Prisma models or internal ids**.
One ZIP: `export.json`, `uploads/`, `README.txt` explaining the fields.
3. The agent **encrypts the ZIP with a random key** and uploads it to the Hub. The key is only in
the link emailed to the site admin (`…/download/<id>#<key>`; the part after `#` is never sent
to the Hub), so the Hub stores the file but can't read it, and the roadmap's "the Hub only sees
aggregate stats" still holds. The download page decrypts it in the browser. Available for
**30 days**.
4. **Uninstall:** final database backup and an archive of the uploads to
`/var/www/sites/.removed/<church>-<date>/` (root only) → PM2 processes removed, `pm2 save` →
the frpc block removed and reloaded → database and role dropped → the site folder removed →
the licence released in the Hub.
5. **Emails:** to the site admin, the download link and the date it expires; to Joshua, the report
and "remove the FastPanel site `events.hope.co.za` (ports 20010, 20011)".
6. **After 30 days:** a `purge` job deletes the kept backup and archive, and the Hub deletes the
encrypted export. Joshua is emailed **3 days before**, and can extend it once by 30 days. The
Hub keeps a record that it was deleted, and when (for POPIA).
## 10. Servers page and alerts
**Hub → Servers:** one card per server (church servers and the central one).
- **Every 60 s** the agent sends: RAM (total, used, available, swap), CPU (% and 1/5/15-minute
load), disk per mount (used, free), uptime, versions (OS, Node, PostgreSQL, frpc, agent), frpc
running, and **per PM2 process** (church, product, memory, CPU, restarts, uptime).
- **Kept:** per minute for 7 days, per hour (average and peak) for 90 days.
- **Server page:** RAM and CPU charts for 24 h / 7 days / 90 days with **spikes marked** (peaks
above 85%), disk bars, the sites on it with each process's memory, recent jobs, the frpc blocks
it manages and the ports they use.
- **Alerts** (email to Joshua, through the existing alerts, once per incident and again when it
clears): agent not heard from for 5 minutes; disk over 85%; RAM over 90% for 5 minutes; a PM2
process restarting more than 5 times in 10 minutes; frpc not running; on the central server,
`proxy_bind_addr` missing or a managed tunnel not listening.
## 11. Reports and emails
All in the Hub (Jobs page, with the full report) and by email:
| Email | To | When |
|-------|----|------|
| Install report | Joshua | Every install, success or failure: each step with its time, ports, the central check, the log tail on failure, what was cleaned up |
| **FastPanel setup** | Joshua | After a successful install. What *he* does in FastPanel; the agent does none of it (below) |
| Welcome | Site admin | When the install is Live |
| Update planned / reminder / done / didn't go through | Site admin | §8 |
| Update report | Joshua | Every update (and batch summary) |
| Missed job | Joshua | A scheduled job wasn't picked up in time |
| Uninstall planned / export ready | Site admin | §9 |
| Uninstall report, purge in 3 days | Joshua | §9 |
| Alerts | Joshua | §10 |
**The FastPanel email** (an instruction to Joshua, nothing is set up automatically):
```
Set up in FastPanel: events.hope.co.za (Hope Family Church, Cross Events)
1. Add the site events.hope.co.za.
2. Proxy:
/api, /uploads, /health, /docs → http://127.0.0.1:20011 (backend)
everything else → http://127.0.0.1:20010 (frontend)
Pass the Host header and X-Forwarded-For / X-Forwarded-Proto: https.
Allow uploads up to 25 MB.
3. Turn on Let's Encrypt for the site.
4. DNS: events.hope.co.za → <central server IP> (checked: ✅ points there / ⚠ not yet)
5. Then click "Proxy done" in the Hub: <link>
Central check: both ports are open in frps and only on 127.0.0.1.
```
## 12. What changes in each repo
**`crosscode-platform`**
- `apps/agent` (new, `agent@1.0.0`) with its install script and systemd unit.
- `scripts/` (the `crosscode` script): `--json` reports on every command, `uninstall`, `export`,
`purge`, `restart`, the frpc block handling (lib/frpc.sh), the env template handling
(lib/env.sh), ports given on the command line instead of picked.
- `apps/events`: `env.template.json`, and the export (`crosscode export` calls a backend script
that writes `crosscode.church-export/1`).
- `packages/contracts` **1.2.0**: `crosscode.agent-job/1`, `crosscode.agent-report/1`,
`crosscode.agent-stats/1`, `crosscode.env-template/1`, `crosscode.church-export/1`.
**`events-hub` (Hub 2.1.0)**
- Models: `Server` (name, mode, public keys, encrypted config, last seen, versions), `ServerStat`
(minute and hourly), `Job` (type, args, status, schedule, who, report), `ScheduledUpdate` /
batch, `Port` (number, server, install, released at), `Export` (encrypted blob, expires).
`Install` gets `serverId`, `shortName`, `status` (installing, waiting-for-proxy, live,
uninstalling, removed).
- Agent API `/api/agent/v1/*` (join, poll, stats, report, export upload), signed by the agent.
- Pages: Servers, Server, New install, Jobs, Updates (schedule, batch), Install defaults, the
public export download page.
- Emails in §11; alerts in §10; the job-signing key in `.env`.
**`crosscode-site`:** nothing (this changes how Joshua works, not what churches buy).
## 13. Testing
- **Agent:** unit tests for job checking (bad signature, wrong server, expired, replayed,
unknown type, an argument with shell characters), and **a test that the running agent has no
listening socket** (`ss -ltnp` shows nothing for its process).
- **Script:** bats tests for the frpc blocks (append to a file with hand-written entries and
comments, remove one block, the other blocks and the hand-written lines unchanged byte for byte;
a failed `frpc verify` leaves the file untouched; `[common]` admin lines added only once), the
env template (new variable appended, existing lines untouched, required one missing → refused),
and the export (a demo church's export validates against `crosscode.church-export/1` and has no
internal ids).
- **On VMs** (multipass): a "central" VM with frps (with `proxy_bind_addr = 127.0.0.1`) and nginx
in place of FastPanel, two church VMs with frpc and a hand-written tunnel, the Hub running
locally:
1. Join all three. Stats show up.
2. New install on church VM 1 → report, FastPanel email, ports open on the central VM, the
hand-written tunnel still works. Add the nginx site by hand, Proxy done → Live.
3. A second church on VM 2 → different ports, unique tunnel names.
4. Schedule an update 4 h ahead (time moved forward in the test) → emails, banner, update,
report. Try 3 h → refused. Emergency with 15 min → needs 2FA.
5. A broken release → rolled back, "didn't go through" email, batch stopped.
6. Hub stopped during a scheduled update → missed, nothing runs, sites keep running.
7. Uninstall → export downloads and decrypts in the browser, everything removed, the other
church's tunnel and the hand-written one untouched. Purge after 30 days (clock moved).
8. A forged job (wrong key) and a replayed job → refused and reported.
## 14. Rolling it out
1. Contracts 1.2.0 → the `crosscode` script changes (with Events 2.2.0 out first) → agent 1.0.0
→ Hub 2.1.0, each released as usual (branch → merge → tag → push).
2. On the central server: add `proxy_bind_addr = 127.0.0.1` to `frps.ini` if it's missing (check
that each FastPanel site still answers), then join the central agent.
3. Join each church server; set its server config in the Hub. Existing installs are linked to
their server and their current ports recorded in the port list (read from `church.env` and
`frpc.ini`; nothing is changed).
4. First real use: the next new church. Existing churches get their next update through a
scheduled update.
5. Docs: CHANGELOGs, HANDOVER.md (1d done), `events-hub/deployment.md` (the job-signing key),
a server how-to in `apps/agent/README.md`.
## Decisions
Agreed with Joshua on 2026-11-12 (ROADMAP-v3 decisions 27–37):
1. The Hub can install, update and uninstall through an agent on each server, with **signed jobs
of fixed types**; the agent only runs the `crosscode` script; uninstall and emergency updates
need 2FA again.
2. **The agent has no URL or listening port**; it only polls the Hub.
3. **Scheduled updates at least 4 hours ahead, emergency updates at least 15 minutes**, with an
email and banner to the site admin; Joshua picks the time.
4. **Uninstall:** the church gets a **JSON export in our own format** (no internal data
structures), encrypted so the Hub can't read it, downloadable for 30 days; backups kept 30
days, then deleted.
5. All church servers are behind frp to **one central FastPanel server**. The Hub keeps the port
list; **frpc.ini is only appended to**; the `[common]` admin lines are added once.
6. **FastPanel and DNS are never automated**; Joshua gets an email with what to set up.
7. **Database and other server settings are set once per server in the Hub** and sent encrypted
with each job; `.env` files are built from a per-product template and only appended to on
updates; generated secrets stay on the server.
8. A read-only agent on the central server checks ports and `proxy_bind_addr`.
## Still open
Nothing.