Docs: server agent and Hub 2.1.0 plan (phase 1d, decisions 27-37)

Agent on every server runs installs, scheduled/emergency updates and
uninstalls on signed jobs from the Hub, polls only (no listening port),
appends to frpc.ini in marked blocks, builds .env files from per-server
config in the Hub. FastPanel and DNS stay manual (emailed instructions).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-12 16:26:51 +02:00
co-authored by Claude Opus 5.5
parent 934f7a9eed
commit c398e8071d
3 changed files with 554 additions and 11 deletions
+477
View File
@@ -0,0 +1,477 @@
# Server agent 1.0.0 + Hub 2.1.0 — installs, updates and uninstalls from the Hub
Planned 2026-11-12 (ROADMAP-v3 phase **1d**, decisions 27–37). Goal: when a new church signs up,
Joshua picks a server in the Hub and the rest happens by itself, with a report by email. Updates
are scheduled from the Hub with notice to the church, and every server's health is visible there.
This replaces the old rule "the Hub never deploys" (decisions 10 and 23). The part that matters
stays: **no site depends on the Hub to keep running**, and everything still goes through the
`crosscode` script (EVENTS-2.2-PLAN §4), which keeps working by hand over SSH.
## Contents
1. [The shape](#1-the-shape)
2. [Rules](#2-rules)
3. [The agent](#3-the-agent)
4. [Jobs and their security](#4-jobs-and-their-security)
5. [Configuration: server, defaults, per install](#5-configuration-server-defaults-per-install)
6. [frp and the central server](#6-frp-and-the-central-server)
7. [New install](#7-new-install)
8. [Updates: scheduled, emergency, batch](#8-updates-scheduled-emergency-batch)
9. [Uninstall](#9-uninstall)
10. [Servers page and alerts](#10-servers-page-and-alerts)
11. [Reports and emails](#11-reports-and-emails)
12. [What changes in each repo](#12-what-changes-in-each-repo)
13. [Testing](#13-testing)
14. [Rolling it out](#14-rolling-it-out)
15. [Decisions](#decisions)
---
## 1. The shape
```
Cross Code Hub (hub.crosscode.co.za)
servers · jobs · schedules · port list · server config · reports
▲ agents poll: "any jobs?" + stats + job results (outbound HTTPS only)
│
┌────────┴─────────────┐ ┌──────────────────────┐ ┌─────────────────────────────┐
│ Church server A │ │ Church server B │ │ Central server (FastPanel) │
│ crosscode-agent │ │ crosscode-agent │ │ crosscode-agent --central │
│ └ runs `crosscode` │ │ └ runs `crosscode` │ │ read-only: frps ports, │
│ PM2 sites, Postgres │ │ … │ │ allow_ports, bind address │
│ frpc ═══tunnel═══════╪═══╪══════════════════════╪══▶│ frps ◀─ nginx (FastPanel, │
└──────────────────────┘ └──────────────────────┘ │ set up by Joshua) │
└─────────────────────────────┘
```
## 2. Rules
1. **The agent listens on nothing.** No HTTP server, no open port, no URL, not even on
`127.0.0.1`. It only makes outgoing HTTPS requests to the Hub (poll, stats, results). A test
checks that the running agent has no listening socket.
2. **The agent only runs the `crosscode` script**, with arguments built from a fixed list of job
types (§4). There is no "run this command" job.
3. **Jobs are signed by the Hub** and checked by the agent against the Hub key pinned when the
server joined. Unsigned, expired or replayed jobs are refused and reported.
4. **No site depends on the Hub or the agent.** If either is down, sites keep running and
`crosscode` still works by hand. Scheduled jobs that can't be delivered are reported as missed,
never run late without a fresh notice (§8).
5. **The agent never touches FastPanel or DNS.** FastPanel setup is an email to Joshua (§7, §11).
6. **Files the agent edits are only ever added to, never replaced**: `/etc/frp/frpc.ini` and each
install's `.env` (§5, §6). Anything written by hand is never changed.
7. **Secrets generated for an install stay on its server** (`JWT_SECRET`, the database password).
They're never sent to the Hub, never put in a report or email.
## 3. The agent
- **Where:** `apps/agent` in this monorepo, Node 24 + TypeScript, tagged `agent@1.0.0`. Installed
in `/opt/crosscode/agent` with its config in `/etc/crosscode/agent.json` (mode 600, root).
- **Run by systemd** (`crosscode-agent.service`, restarts on failure), not PM2, because it manages
PM2. Runs as root, as `crosscode install` needs (PostgreSQL, PM2 for the app user, frpc).
- **Joining:** Hub → Servers → **Add server** gives a one-time token (valid 1 hour). On the server:
`curl … | bash -s -- --hub https://hub.crosscode.co.za --token …` installs Node if needed, the
agent and `/opt/crosscode/tools`, then the agent:
1. makes an **Ed25519 key pair** (signs its requests to the Hub) and an **X25519 key pair**
(receives encrypted config, §5); the private keys never leave the server,
2. sends the token and both public keys; the Hub replies with the server id and **the Hub's
job-signing public key**, which the agent pins.
- **Polling:** every 30 s `POST /api/agent/v1/poll` (signed) with a short status; the reply holds
any jobs due. Every 60 s the stats (§10). Requests back off when the Hub doesn't answer.
- **Modes:** `church` (default) and `central` (§6). The central agent can't run any job but
`check-ports` and `status`.
- **Updating itself:** an `update-agent` job checks out the newest `agent@…` tag, builds it next
to the running copy, switches the systemd unit over and reports its new version. If the new one
doesn't poll within 2 minutes, a systemd timer starts the previous one again.
- **One job at a time per server**, in the order the Hub gives them. A job has a time limit
(install 30 min, deploy 20 min); a job past it is stopped and reported as failed.
## 4. Jobs and their security
**Job types** (and the `crosscode` command each runs):
| Job | Runs | Needs 2FA again in the Hub |
|-----|------|:-:|
| `install` | `crosscode install <church> --products … --json` | no |
| `add-product` | `crosscode add <church> <product> --json` | no |
| `deploy` | `crosscode deploy <church> [products] [--version …] --json` | emergency only |
| `adopt` / `adopt-undo` | `crosscode adopt …` | no |
| `restart` | `crosscode restart <church> [product]` (PM2 reload) | no |
| `export` | `crosscode export <church> --json` (§9) | no |
| `uninstall` | `crosscode uninstall <church> --json` | **yes**, and typing the short name |
| `purge` | `crosscode purge <church>` (deletes the 30-day keep, §9) | no (scheduled at uninstall) |
| `update-agent` | §3 | no |
| `check-ports` | central only (§6) | no |
| `status` | `crosscode status --json` | no |
**A signed job** (`crosscode.agent-job/1`, in `packages/contracts`):
```json
{
"schema": "crosscode.agent-job/1",
"id": "job_…",
"serverId": "srv_…",
"type": "deploy",
"args": { "church": "hope", "products": ["events"], "version": "events@2.4.1" },
"config": { "alg": "x25519-xsalsa20-poly1305", "ciphertext": "…" },
"notBefore": "2026-11-20T12:00:00Z",
"expiresAt": "2026-11-20T12:15:00Z",
"signature": "…"
}
```
- The agent checks: the signature (pinned Hub key), `serverId` is its own, it's within
`notBefore`–`expiresAt`, the id hasn't been seen (it keeps the ids of the last 90 days), the
type is known and every `args` value passes that type's schema (short names
`^[a-z][a-z0-9-]{1,30}$`, products from the enum, versions as `<product>@x.y.z`). Arguments are
passed to `crosscode` as an argument array, never through a shell.
- `config` holds whatever the job needs from §5, encrypted to **this server's** X25519 key, so
job rows and logs in the Hub never hold it in plain text.
- **Results** (`crosscode.agent-report/1`): the job id, status (`succeeded`, `failed`,
`rolled-back`, `refused`, `expired`), each step with its start, end and outcome, and the last
200 log lines. The script's `--json` output is that report; the agent adds the log lines and
sends it, signed with its own key. Logs are filtered for anything that looks like a secret
before they leave the server.
- **The Hub side:** job-signing private key in the Hub's `.env` (not in the database), its own
key separate from the licence-signing key. 2FA re-confirmation for uninstall and emergency
deploys. Every job is in the audit log with who made it.
## 5. Configuration: server, defaults, per install
An install's `.env` files are built from four layers. Each variable comes from exactly one layer.
| Layer | Set where | Examples | Sent how |
|-------|-----------|----------|----------|
| **Server** | Hub → Servers → *server* → Configuration, **once per server** (editable) | Database host, port and an admin role that can create databases and roles (or "local PostgreSQL, through `sudo -u postgres`"), `TRUST_PROXY`, app user, backup folder | Stored AES-GCM encrypted in the Hub (as install secrets are today); sent to the agent encrypted to its key with each job that needs it. The agent doesn't keep it on disk |
| **Defaults** | Hub → Settings → Install defaults, once | `HUB_URL`, `APP_TIMEZONE` (Africa/Johannesburg), job switches (`DAILY_SUMMARY_ENABLED=true`, `BACKUP_ENABLED=true`, …), `HUB_HEARTBEAT_ENABLED=true`, optional platform `SENTRY_DSN` | With the job, encrypted the same way |
| **Install** | The New install form (§7) | Domain → `FRONTEND_URL`, `APP_BASE_URL`, `BACKEND_URL` and the frontend's `NEXT_PUBLIC_API_URL` (all `https://<domain>`, since FastPanel sends `/api` to the backend on the same domain); timezone if not the default; the licence key; ports from the port list (§6) → `PORT` | With the job |
| **Generated** | On the server by `crosscode install` | `JWT_SECRET`, the database password (so `DATABASE_URL`), `NODE_ENV=production` | Never leaves the server (rule 7) |
**Per-product env template.** Each product has `apps/<product>/env.template.json` in the
monorepo, listing every variable with its layer, whether it's required, and a default:
```json
{ "schema": "crosscode.env-template/1", "product": "events", "files": {
"backend/.env": [
{ "name": "DATABASE_URL", "layer": "generated", "required": true },
{ "name": "FRONTEND_URL", "layer": "install", "from": "https://{domain}", "required": true },
{ "name": "TRUST_PROXY", "layer": "server", "required": true },
{ "name": "BACKUP_ENABLED", "layer": "defaults", "default": "true" }
],
"frontend/.env.production": [
{ "name": "NEXT_PUBLIC_API_URL", "layer": "install", "from": "https://{domain}", "required": true }
] } }
```
- The Hub reads the template of the release being installed (synced with the release, §5 of the
roadmap) and **refuses to create the job if a required server or defaults value is missing**,
saying which, before anything runs on the server.
- **Install:** `crosscode install` writes each `.env` from the template (mode 600, app user).
- **Updates:** before building, `crosscode deploy` compares the new release's template with the
`.env`. A new variable is **appended** with its value (and a comment with the date and
release); existing lines are never changed or removed. A new *required* variable with no value
in any layer stops the deploy at the checks, before anything is touched.
- **Changing a server value later** (e.g. a new database host) is a `reconfigure` step of a
deploy: the agent shows which lines would change in each install, and only changes them when
Joshua confirms in the Hub. This is the one exception to "only append", and it never runs by
itself.
- **URLs are checked before and after.** Before: the form checks that the domain's DNS points at
the central server and warns if not (the install still runs). After Joshua marks the proxy done:
the Hub checks `https://<domain>/` (frontend answers), `https://<domain>/health` (backend
answers with the right version) and that `/health` reports the `FRONTEND_URL` it was given.
**For Joshua to set per server before the first install:** the database settings and
`TRUST_PROXY`. Behind frp the app sees every connection from frpc on `127.0.0.1`, and the
visitor's address arrives in `X-Forwarded-For` from FastPanel's nginx, so the expected value is
`loopback` (the Hub suggests it).
## 6. frp and the central server
**Today:** `/etc/frp/frpc.ini` on each church server, `tcp` tunnels, frontend and backend each
with their own `remote_port`. FastPanel on the central server proxies a domain to
`127.0.0.1:<remote_port>` with Let's Encrypt.
**Port list in the Hub.** frps opens every `remote_port` on the one central server, so they must
be unique across every church server. The Hub hands them out from a range (default
**20000–29999**, set in Hub settings to match `allow_ports`), two per product install (web, api),
and **uses the same number for `local_port`**, so it's also free on the church server and there's
one number per tunnel to think about. Ports already in use (today's `3000`/`5000` and anything
the central agent sees listening) are recorded as taken. Ports are released 30 days after an
uninstall.
**Tunnel names** must also be unique on frps: `<church>-<product>-web` and
`<church>-<product>-api` (e.g. `hope-events-web`). Existing entries such as `[events-frontend]`
stay as they are.
**What the agent adds to frpc.ini** (one block per product install, appended at the end):
```ini
# >>> crosscode:hope:events (added by crosscode-agent 2026-11-20, do not edit by hand)
[hope-events-web]
type = tcp
local_ip = 127.0.0.1
local_port = 20010
remote_port = 20010
[hope-events-api]
type = tcp
local_ip = 127.0.0.1
local_port = 20011
remote_port = 20011
# <<< crosscode:hope:events
```
- **Only appended to, never replaced.** Before a change: a copy to
`/etc/frp/backups/frpc.ini.<timestamp>`; write to a temporary file; `frpc verify -c`; then move
it into place. If the check fails, nothing is changed. Uninstall removes only the lines between
that install's own markers; nothing outside a marked block is ever changed.
- **Reload without dropping other sites.** Once per server (agreed 2026-11-12) the agent adds to
`[common]`, if they're not there:
```ini
admin_addr = 127.0.0.1
admin_port = 7400
```
so `frpc reload -c /etc/frp/frpc.ini` picks up new tunnels without restarting frpc. This is the
only change ever made in `[common]`, and it's reported. (`admin_addr` is loopback only, so it
adds no way in from outside.)
- The `.ini` format is deprecated in newer frp versions in favour of TOML. The agent reads the
format the server has; moving to TOML is a separate job for later.
**The central agent** (`--central`, on the FastPanel server, read-only) answers `check-ports`:
- the port is inside `allow_ports` in `frps.ini`,
- frps has the tunnel open (the port is listening; frps's dashboard API is used if it's enabled
on `127.0.0.1`),
- **`proxy_bind_addr = 127.0.0.1` is set**, so tunnels are only reachable through FastPanel's
nginx, not straight from the internet on `<central IP>:<port>` (which would bypass HTTPS and
expose the backend). If it's missing, every report and the Servers page show a warning.
It changes nothing on the central server: no frps edits, no firewall changes, no FastPanel.
## 7. New install
**Hub → Customers → *customer* → New install** (or a new customer first):
1. **Form:** products (Events now; Accounts / Website / App as they ship), **short name**
(`hope`, `cross`, `bridge`; checked unique across all servers and used for folders, PM2 names,
database names and tunnel names), address (own domain or Cross Code address; per product the
domain is shown, e.g. `events.hope.co.za`), timezone, **site admin name and email**, and the
**server**: a list showing each server's free RAM, free disk, CPU load over the last day, sites
on it and agent status, with servers below the free-RAM threshold (default 1.5 GB per product
install) marked as not recommended.
2. **Checks in the Hub** before the job exists: the server agent is online, server config and
defaults are complete for the release's env template (§5), ports are free, the short name is
free, DNS (warning only).
3. The Hub **creates the licence(s)**, assigns ports, and queues the `install` job.
4. **On the server** (`crosscode install`, each step in the report):
checks (disk, RAM, Node, PostgreSQL reachable) → folder `/var/www/sites/<church>/<product>`
→ checkout of the release tag → `.env` files (§5) → database and role → install dependencies
→ migrations → build → PM2 processes on the given ports, `pm2 save` → `church.env` → frpc
block and reload (§6) → local `/health` answers with the right version → the site registers
with the Hub using its licence.
5. **On failure** anything this job created is removed again (PM2 processes, frpc block, database
and role, folder), in reverse order, and the report says what failed and what was cleaned up.
6. **Central check:** the Hub asks the central agent to `check-ports` for both ports.
7. **Emails to Joshua:** the install report and, if everything passed, the **FastPanel email**
(§11). The install shows **Waiting for proxy** in the Hub.
8. Joshua sets up FastPanel by hand and clicks **Proxy done**. The Hub checks the public URLs
(§5) and marks it **Live**, or says what didn't answer. When Live, the site admin gets the
welcome email with the setup-wizard link.
## 8. Updates: scheduled, emergency, batch
**Scheduled** (Hub → Install → Schedule update, or Updates → Schedule):
- Joshua picks the release and the time. The time must be **at least 4 hours away**; the Hub
refuses anything sooner. A warning (not a block) for Sunday 06:00–13:00 and for days with an
event of that church running (from the stats).
- Straight away: an email to the site admin(s) ("Cross Events will be updated on Thursday 20
November at 14:00; the site will be unavailable for a few minutes", plus the what's-new
summary), and a **banner** through the existing Notices, starting now and ending after the
update. A reminder email 1 hour before.
- At the time the Hub releases the `deploy` job (`notBefore` = the time, `expiresAt` = 15 min
later). If the agent doesn't pick it up in that window the update is **missed**: nothing runs,
Joshua is emailed, and a new time needs a new notice.
- Afterwards: the report to Joshua; to the site admin "Updated to 2.4.1" with what's new, or
"The update didn't go through; the site is running the previous version" if it rolled back.
- Cancelling or moving it updates the banner and emails the site admin.
**Emergency:** the same, with at least **15 minutes'** notice, a required reason (shown in the
email and banner), and 2FA again. Meant for security fixes and broken releases.
**Batch:** "every Events install to 2.4.1" (or a selection), at one time with one notice. The
Hub runs them **one at a time** (on each server, and across servers) and **stops the batch at the
first failure**, emailing Joshua; the remaining installs get a "postponed" email and their banner
is removed.
**Rules carried over:** the release's requirements (`— requires accounts 1.2.0`, decision 24)
are checked in the Hub before scheduling and again by the script; each product rolls back on its
own (§8 of the roadmap). A church's **Request update** now shows as "Requested" next to
Schedule update. Moving a 2.1.x install to 2.2.0 is an `adopt` job, scheduled the same way.
## 9. Uninstall
Hub → Install → **Uninstall** (2FA again, type the short name):
1. Joshua picks the date (at least 4 hours away, default 7 days) and the site admin is emailed
and gets a banner, as for an update.
2. **Export first** (`crosscode export`): the church's data as **JSON in a format of our own**,
`crosscode.church-export/1` (defined in `packages/contracts`): events, sessions, ticket types,
registrations and attendees, people (name, email, phone, consent), payments (amounts, dates,
references), check-ins, rosters, settings, and the uploads as files next to it. Written in
plain terms with our own field names; **no database tables, Prisma models or internal ids**.
One ZIP: `export.json`, `uploads/`, `README.txt` explaining the fields.
3. The agent **encrypts the ZIP with a random key** and uploads it to the Hub. The key is only in
the link emailed to the site admin (`…/download/<id>#<key>`; the part after `#` is never sent
to the Hub), so the Hub stores the file but can't read it, and the roadmap's "the Hub only sees
aggregate stats" still holds. The download page decrypts it in the browser. Available for
**30 days**.
4. **Uninstall:** final database backup and an archive of the uploads to
`/var/www/sites/.removed/<church>-<date>/` (root only) → PM2 processes removed, `pm2 save` →
the frpc block removed and reloaded → database and role dropped → the site folder removed →
the licence released in the Hub.
5. **Emails:** to the site admin, the download link and the date it expires; to Joshua, the report
and "remove the FastPanel site `events.hope.co.za` (ports 20010, 20011)".
6. **After 30 days:** a `purge` job deletes the kept backup and archive, and the Hub deletes the
encrypted export. Joshua is emailed **3 days before**, and can extend it once by 30 days. The
Hub keeps a record that it was deleted, and when (for POPIA).
## 10. Servers page and alerts
**Hub → Servers:** one card per server (church servers and the central one).
- **Every 60 s** the agent sends: RAM (total, used, available, swap), CPU (% and 1/5/15-minute
load), disk per mount (used, free), uptime, versions (OS, Node, PostgreSQL, frpc, agent), frpc
running, and **per PM2 process** (church, product, memory, CPU, restarts, uptime).
- **Kept:** per minute for 7 days, per hour (average and peak) for 90 days.
- **Server page:** RAM and CPU charts for 24 h / 7 days / 90 days with **spikes marked** (peaks
above 85%), disk bars, the sites on it with each process's memory, recent jobs, the frpc blocks
it manages and the ports they use.
- **Alerts** (email to Joshua, through the existing alerts, once per incident and again when it
clears): agent not heard from for 5 minutes; disk over 85%; RAM over 90% for 5 minutes; a PM2
process restarting more than 5 times in 10 minutes; frpc not running; on the central server,
`proxy_bind_addr` missing or a managed tunnel not listening.
## 11. Reports and emails
All in the Hub (Jobs page, with the full report) and by email:
| Email | To | When |
|-------|----|------|
| Install report | Joshua | Every install, success or failure: each step with its time, ports, the central check, the log tail on failure, what was cleaned up |
| **FastPanel setup** | Joshua | After a successful install. What *he* does in FastPanel; the agent does none of it (below) |
| Welcome | Site admin | When the install is Live |
| Update planned / reminder / done / didn't go through | Site admin | §8 |
| Update report | Joshua | Every update (and batch summary) |
| Missed job | Joshua | A scheduled job wasn't picked up in time |
| Uninstall planned / export ready | Site admin | §9 |
| Uninstall report, purge in 3 days | Joshua | §9 |
| Alerts | Joshua | §10 |
**The FastPanel email** (an instruction to Joshua, nothing is set up automatically):
```
Set up in FastPanel: events.hope.co.za (Hope Family Church, Cross Events)
1. Add the site events.hope.co.za.
2. Proxy:
/api, /uploads, /health, /docs → http://127.0.0.1:20011 (backend)
everything else → http://127.0.0.1:20010 (frontend)
Pass the Host header and X-Forwarded-For / X-Forwarded-Proto: https.
Allow uploads up to 25 MB.
3. Turn on Let's Encrypt for the site.
4. DNS: events.hope.co.za → <central server IP> (checked: ✅ points there / ⚠ not yet)
5. Then click "Proxy done" in the Hub: <link>
Central check: both ports are open in frps and only on 127.0.0.1.
```
## 12. What changes in each repo
**`crosscode-platform`**
- `apps/agent` (new, `agent@1.0.0`) with its install script and systemd unit.
- `scripts/` (the `crosscode` script): `--json` reports on every command, `uninstall`, `export`,
`purge`, `restart`, the frpc block handling (lib/frpc.sh), the env template handling
(lib/env.sh), ports given on the command line instead of picked.
- `apps/events`: `env.template.json`, and the export (`crosscode export` calls a backend script
that writes `crosscode.church-export/1`).
- `packages/contracts` **1.2.0**: `crosscode.agent-job/1`, `crosscode.agent-report/1`,
`crosscode.agent-stats/1`, `crosscode.env-template/1`, `crosscode.church-export/1`.
**`events-hub` (Hub 2.1.0)**
- Models: `Server` (name, mode, public keys, encrypted config, last seen, versions), `ServerStat`
(minute and hourly), `Job` (type, args, status, schedule, who, report), `ScheduledUpdate` /
batch, `Port` (number, server, install, released at), `Export` (encrypted blob, expires).
`Install` gets `serverId`, `shortName`, `status` (installing, waiting-for-proxy, live,
uninstalling, removed).
- Agent API `/api/agent/v1/*` (join, poll, stats, report, export upload), signed by the agent.
- Pages: Servers, Server, New install, Jobs, Updates (schedule, batch), Install defaults, the
public export download page.
- Emails in §11; alerts in §10; the job-signing key in `.env`.
**`crosscode-site`:** nothing (this changes how Joshua works, not what churches buy).
## 13. Testing
- **Agent:** unit tests for job checking (bad signature, wrong server, expired, replayed,
unknown type, an argument with shell characters), and **a test that the running agent has no
listening socket** (`ss -ltnp` shows nothing for its process).
- **Script:** bats tests for the frpc blocks (append to a file with hand-written entries and
comments, remove one block, the other blocks and the hand-written lines unchanged byte for byte;
a failed `frpc verify` leaves the file untouched; `[common]` admin lines added only once), the
env template (new variable appended, existing lines untouched, required one missing → refused),
and the export (a demo church's export validates against `crosscode.church-export/1` and has no
internal ids).
- **On VMs** (multipass): a "central" VM with frps (with `proxy_bind_addr = 127.0.0.1`) and nginx
in place of FastPanel, two church VMs with frpc and a hand-written tunnel, the Hub running
locally:
1. Join all three. Stats show up.
2. New install on church VM 1 → report, FastPanel email, ports open on the central VM, the
hand-written tunnel still works. Add the nginx site by hand, Proxy done → Live.
3. A second church on VM 2 → different ports, unique tunnel names.
4. Schedule an update 4 h ahead (time moved forward in the test) → emails, banner, update,
report. Try 3 h → refused. Emergency with 15 min → needs 2FA.
5. A broken release → rolled back, "didn't go through" email, batch stopped.
6. Hub stopped during a scheduled update → missed, nothing runs, sites keep running.
7. Uninstall → export downloads and decrypts in the browser, everything removed, the other
church's tunnel and the hand-written one untouched. Purge after 30 days (clock moved).
8. A forged job (wrong key) and a replayed job → refused and reported.
## 14. Rolling it out
1. Contracts 1.2.0 → the `crosscode` script changes (with Events 2.2.0 out first) → agent 1.0.0
→ Hub 2.1.0, each released as usual (branch → merge → tag → push).
2. On the central server: add `proxy_bind_addr = 127.0.0.1` to `frps.ini` if it's missing (check
that each FastPanel site still answers), then join the central agent.
3. Join each church server; set its server config in the Hub. Existing installs are linked to
their server and their current ports recorded in the port list (read from `church.env` and
`frpc.ini`; nothing is changed).
4. First real use: the next new church. Existing churches get their next update through a
scheduled update.
5. Docs: CHANGELOGs, HANDOVER.md (1d done), `events-hub/deployment.md` (the job-signing key),
a server how-to in `apps/agent/README.md`.
## Decisions
Agreed with Joshua on 2026-11-12 (ROADMAP-v3 decisions 27–37):
1. The Hub can install, update and uninstall through an agent on each server, with **signed jobs
of fixed types**; the agent only runs the `crosscode` script; uninstall and emergency updates
need 2FA again.
2. **The agent has no URL or listening port**; it only polls the Hub.
3. **Scheduled updates at least 4 hours ahead, emergency updates at least 15 minutes**, with an
email and banner to the site admin; Joshua picks the time.
4. **Uninstall:** the church gets a **JSON export in our own format** (no internal data
structures), encrypted so the Hub can't read it, downloadable for 30 days; backups kept 30
days, then deleted.
5. All church servers are behind frp to **one central FastPanel server**. The Hub keeps the port
list; **frpc.ini is only appended to**; the `[common]` admin lines are added once.
6. **FastPanel and DNS are never automated**; Joshua gets an email with what to set up.
7. **Database and other server settings are set once per server in the Hub** and sent encrypted
with each job; `.env` files are built from a per-product template and only appended to on
updates; generated secrets stay on the server.
8. A read-only agent on the central server checks ports and `proxy_bind_addr`.
## Still open
Nothing.
+8 -3
View File
@@ -1,6 +1,6 @@
# Handover — Cross Code, all products
Last updated: 2026-11-05. **The plan is [ROADMAP-v3.md](./ROADMAP-v3.md)** (decided 2026-11-03,
Last updated: 2026-11-12. **The plan is [ROADMAP-v3.md](./ROADMAP-v3.md)** (decided 2026-11-03,
decisions table near the end). This file says where each phase stands and what comes next.
Product-specific notes (where the code lives, known gaps) stay with each product: Cross Events'
are in `hope-events/docs/HANDOVER.md` (`apps/events/docs/HANDOVER.md` from Events 2.2.0), and
@@ -14,7 +14,7 @@ for each phase. They moved here from `hope-events/docs/` on 2026-11-05.
| Repo | What |
|------|------|
| `joshua/crosscode-platform` (this one) | Contracts, the apps (Accounts, Website, App, and Events from 2.2.0), shared packages, the `crosscode` script (from Events 2.2.0), and these docs |
| `joshua/crosscode-platform` (this one) | Contracts, the apps (Accounts, Website, App, and Events from 2.2.0), shared packages, the `crosscode` script (from Events 2.2.0), the server agent (`apps/agent`, phase 1d), and these docs |
| `joshua/hope-events` | Cross Events up to 2.1.x. Archived after the 2.2.0 move |
| `joshua/events-hub` | Cross Code Hub. Its own server, its own `scripts/deploy.sh` |
| `joshua/crosscode-site` | crosscode.co.za. Updated at the end of every phase that changes what it says (ROADMAP §9) |
@@ -27,6 +27,7 @@ for each phase. They moved here from `hope-events/docs/` on 2026-11-05.
| 1 | Hub 2.0.0 | **Released** 2026-11-05 (`events-hub` `v2.0.0`; `docs/HUB-2.0-PLAN.md` there). **Not deployed yet:** `events-hub/deployment.md` → "Upgrading to 2.0" (run `check-hub2` first). crosscode-site's address prices are merged, **not deployed yet** |
| 1b | Cross Events 2.2.0: moves here as `apps/events`, `crosscode` script | **Next.** Planned and decided in [EVENTS-2.2-PLAN.md](./EVENTS-2.2-PLAN.md) |
| 1c | Cross Events 2.3.0: authenticator app 2FA (TOTP + recovery codes) | Not started (`TwoFactorMethod` enum in Events' `prisma/schema.prisma`) |
| 1d | Agent 1.0.0 + Hub 2.1.0 (+ contracts 1.2.0): server agent, installs / scheduled updates / uninstalls from the Hub, server stats | **Planned** 2026-11-12 in [AGENT-1.0-PLAN.md](./AGENT-1.0-PLAN.md) (ROADMAP decisions 27–37). Starts after 1b; can run alongside 1c |
| 2 | Cross Events 2.4.0: protocol v2, Linked products, event feed, `packages/seo` | Not started. Events keeps speaking protocol 1 until then |
| 3–7 | Accounts 1.0.0 → Website 1.0.0 → Events 3.0.0 → App 1.0.0 → App 1.x | Not started |
@@ -37,7 +38,11 @@ for each phase. They moved here from `hope-events/docs/` on 2026-11-05.
2. **Cross Events 2.2.0**: [EVENTS-2.2-PLAN.md](./EVENTS-2.2-PLAN.md). Decided 2026-11-05:
`adopt` builds next to the running install and switches over, and `adopt --undo` is built
(ROADMAP decisions 25–26).
3. **Cross Events 2.3.0** (authenticator app 2FA), then 2.4.0, as in ROADMAP §9.
3. **Agent 1.0.0 + Hub 2.1.0** (phase 1d): [AGENT-1.0-PLAN.md](./AGENT-1.0-PLAN.md). Before the
first install through it: check `proxy_bind_addr = 127.0.0.1` in `frps.ini` on the central
server (worth doing now: without it, tunnel ports such as `:5000` may be reachable straight from
the internet), and set each server's database settings and `TRUST_PROXY` in the Hub.
4. **Cross Events 2.3.0** (authenticator app 2FA), then 2.4.0, as in ROADMAP §9.
## Things to remember across phases
+69 -8
View File
@@ -20,6 +20,12 @@ updates"), the Hub tracks releases per product (§5), and the contracts reach th
committed to its repo instead of through a package registry (§2). Cross Events versions in §9
moved up by one (the move is 2.2.0). See decisions 21–24.
**Updated 2026-11-12:** a **server agent** on each server installs, updates and uninstalls
church sites on signed jobs from the Hub (phase 1d, [AGENT-1.0-PLAN.md](./AGENT-1.0-PLAN.md)).
The Hub can now trigger changes, but no site depends on it to keep running and the agent only
polls (it has no URL or open port). Rule 2 (§1), §5, §8 and §9 are updated; decisions 27–37
change decisions 10 and 23.
---
## Contents
@@ -61,8 +67,10 @@ own process and its own database. The Hub only sees aggregate stats and licences
**Rules**
1. **No shared databases, ever.** Products only talk over HTTPS using documented contracts (§2).
2. **The Hub is never on the runtime path.** It stays read-only toward installs: installs push to
it, it never reaches into them, and a Hub outage switches nobody off.
2. **The Hub is never on the runtime path.** Installs push to it, it never connects to them, and
a Hub outage switches nobody off. **Changes to a site are made by the server agent** on its
server (from phase 1d): it polls the Hub for **signed jobs of fixed types**, only ever runs the
`crosscode` script, and has no URL or listening port of its own (§8, decisions 27–37).
3. **One maintained codebase per product.** Each church runs a tagged release of it. Nothing is
built specially for one church.
4. **The origin of a record owns it.** Copies elsewhere are read-only and link back to the origin.
@@ -317,7 +325,14 @@ product's customers, licences, stats, AI usage and billing.
before it deploys (§8). By default products never need to update together: the contract rules
(§2) and protocol v1 + v2 make that possible, so requirements are only for real breaking moves
such as Events 3.0.
- **Request update** works for every product. The Hub still never runs anything on an install.
- **Request update** works for every product. From Hub 2.1.0 Joshua answers it with a
**scheduled update** (at least 4 hours' notice to the site admin, emergency at least 15
minutes), carried out by the server agent (§8).
- **Servers (Hub 2.1.0):** every server's agent, its stats (RAM, CPU, disk, spikes, PM2
processes per church), alerts, the frp port list, server configuration (database and other
per-server `.env` values, stored encrypted), new installs with a choice of server, jobs with
emailed reports, and uninstalls with a 30-day encrypted export for the church
([AGENT-1.0-PLAN.md](./AGENT-1.0-PLAN.md)).
- **Domain:** `hub.crosscode.co.za`. `events-hub.crosscode.co.za` stays as an alias so existing
installs keep working.
- Dashboard: installs per product, versions behind latest, health, licence status, monthly
@@ -463,7 +478,7 @@ browsed without an account. Personal actions need sign-in through Accounts.
| Repo | What |
|------|------|
| `joshua/crosscode-platform` | **Every product a church installs**, and what they share: `apps/events` (Cross Events, moved here in 2.2.0), `apps/accounts`, `apps/website`, `apps/app`, `packages/contracts`, `packages/ui`, `packages/seo` (§11), and the `crosscode` script (`scripts/`) |
| `joshua/crosscode-platform` | **Every product a church installs**, and what they share: `apps/events` (Cross Events, moved here in 2.2.0), `apps/accounts`, `apps/website`, `apps/app`, `packages/contracts`, `packages/ui`, `packages/seo` (§11), the `crosscode` script (`scripts/`) and the server agent (`apps/agent`, phase 1d) |
| `joshua/hope-events` | Cross Events up to 2.1.x. **Archived** after the move (read-only, README points to the monorepo), and kept so installs that haven't moved yet can still deploy or roll back a `v2.1.x` tag |
| `joshua/events-hub` | Cross Code Hub (renamed in the UI; repo name can stay). Stays on its own: it runs on its own server, is never installed for a church, and is deployed with its own `scripts/deploy.sh`. Gets the contracts as a committed file (§2) |
| `joshua/crosscode-site` | The marketing site (crosscode.co.za). Updated at the end of every phase that changes what's on sale or what a product does (see §9, "Updating crosscode-site") |
@@ -473,8 +488,10 @@ Playwright, Jest), so code and habits carry over.
### Releases and updates
**What doesn't change:** git is how code reaches a server, the Hub only records releases and
passes on update requests (it never deploys, §1 rule 2), and Joshua runs the deploys.
**What doesn't change:** git is how code reaches a server, and every install, update and
uninstall goes through the `crosscode` script. Until phase 1d Joshua runs it by hand; from
phase 1d the **server agent** runs it on signed jobs from the Hub (below), and it still works by
hand over SSH.
**Releasing.** Each product is tagged on its own in the monorepo: `events@2.2.0`,
`accounts@1.0.0`, `website@1.0.0`, `app@1.0.0` (and `contracts@1.0.0`, which nobody installs).
@@ -564,6 +581,30 @@ would be down for the whole build. The steps are in
The PM2 names, nginx config (it only proxies to ports), database and its name stay the same.
Installs that haven't moved yet keep working with `deploy.sh` on the archived repo.
### Server agent (phase 1d)
Planned in [AGENT-1.0-PLAN.md](./AGENT-1.0-PLAN.md). In short:
- **`crosscode-agent`** (`apps/agent`, systemd) on every church server. It **only polls** the Hub
(outgoing HTTPS; no URL, no listening port), sends server stats every minute, and runs
**signed jobs of fixed types** (install, add product, deploy, adopt, restart, export,
uninstall), each by calling the `crosscode` script. Uninstall and emergency updates need 2FA
again in the Hub.
- **Servers are behind frp** to one central FastPanel server. The Hub hands out the tunnel ports
(unique across all servers). The agent **only appends** marked blocks to `/etc/frp/frpc.ini` and
removes only its own blocks. A read-only agent on the central server checks the ports are open
and only bound to `127.0.0.1`. **FastPanel and DNS are never automated:** Joshua gets an email
with what to set up.
- **`.env` files** are built from a per-product template: database and other server values set
**once per server in the Hub** (sent encrypted), platform defaults, the install's domain and
ports, and secrets generated on the server (never sent anywhere). Updates only append new
variables.
- **Updates** are scheduled at least 4 hours ahead (emergency 15 minutes) with an email and banner
to the site admin, and run one install at a time. **Uninstall** gives the church a JSON export
in our own format (encrypted, 30 days) and keeps backups for 30 days.
- New `crosscode` commands for it: `uninstall`, `export`, `purge`, `restart`, and `--json`
reports on every command.
---
## 9. Build order
@@ -574,6 +615,7 @@ Installs that haven't moved yet keep working with `deploy.sh` on the archived re
| 1 | **Hub 2.0.0** | Cross Code Hub: products, licence per product, per-customer billing with bundles, protocol v2 (v1 still accepted), sibling discovery, AI usage per product, **releases per product with requirements** (§5), contracts as a committed file (§2). **Released 2026-11-05** (crosscode-site address prices merged) | Must exist before a second product is sold, and before Events releases come from the monorepo |
| 1b | **Cross Events 2.2.0** | **Moves into the monorepo** as `apps/events`, with its git history (merged in, not copied); its packages become workspaces; the **`crosscode` script** (`deploy`, `status`, `install`, `adopt`; §8) replaces `scripts/deploy.sh` and `scripts/install.sh`; every install moved with `crosscode adopt`; `hope-events` archived. No feature changes, so a problem is clearly the move | Before more products, so every product is released and deployed the same way from the start |
| 1c | **Cross Events 2.3.0** | **Authenticator app 2FA** (TOTP) next to email and WhatsApp, with recovery codes; admins can see which method each staff account uses | Independent and useful now; the Accounts migration later carries it over |
| 1d | **Agent 1.0.0 + Hub 2.1.0** (contracts 1.2.0) | **Server agent** on every server (polls only, signed jobs, runs the `crosscode` script); Hub: Servers page with stats and alerts, server configuration, frp port list, New install with a choice of server, scheduled / emergency / batch updates with notice to the church, uninstall with an encrypted JSON export, emailed reports. [AGENT-1.0-PLAN.md](./AGENT-1.0-PLAN.md) | Right after 1b (it runs the `crosscode` script), so every new church from then on is installed from the Hub. Can run alongside 1c |
| 2 | **Cross Events 2.4.0** | Reports `product: "events"` (protocol v2), **Linked products**, the event feed (publish + show events from siblings), `packages/seo` (§11) with canonical links for imported events | Small; ready for when the Website exists |
| 3 | **Accounts 1.0.0** | OIDC provider, sign-in, 2FA (email, WhatsApp, authenticator app), password reset, consent and notice, export / delete fan-out, admin, install script | The Website is built on it from day one |
| 4 | **Church Website 1.0.0** | Public site with the **four templates** (§6), admin, simple events + feed, SEO (§11), Hope Family Church as the first church, then a second test church on a different template to prove it repeats | More churches need a website than an app; it also covers the "website only" case |
@@ -596,6 +638,7 @@ At the end of each phase, update it in the same sitting and deploy it (its `DEPL
| 1 Hub 2.0.0 | Bundle discounts (15% / 25%) and the address add-on (own domain R0 · Cross Code address R10 · domain included R25) in `src/data/pricing.mjs`, worded so it's clear the Website and App are still coming. Legal pages if billing terms change |
| 1b Events 2.2.0 | Nothing (the move doesn't change what Cross Events does) |
| 1c Events 2.3.0 | Authenticator app 2FA in the Cross Events feature list (`src/data/cross-events.mjs`) and the security / POPIA wording |
| 1d Agent 1.0.0 + Hub 2.1.0 | Nothing (it changes how installs are run, not what churches buy) |
| 2 Events 2.4.0 | "Works with the Church Website and App" (event feed, Linked products) on the Cross Events page, as "coming" until those ship |
| 3 Accounts 1.0.0 | "One sign-in for all your church's products" on the Products page. Accounts isn't sold, so no page or price of its own |
| 4 Website 1.0.0 | **Church Website goes on sale:** `status: "available"` and its own page in `src/data/products.mjs` / `src/pages/products/`, R99 + storage add-on in `pricing.mjs`, **a showcase of the four templates** (a page per template with screenshots of the same demo church in each), screenshots from a demo church (fictional people only), FAQ, home page, contact form product list, `SoftwareApplication` JSON-LD (§11) |
@@ -620,6 +663,8 @@ Hub never disagree. Run the site's tests and build before deploying it.
- crosscode-site shows the product as available, with its page, prices and screenshots.
- The §11 SEO checks pass (Playwright and Lighthouse), for the Website in all four templates.
- A database backup is restored successfully.
- (From phase 1d) The product installs, updates and uninstalls through the server agent from the
Hub, with its `.env` template complete and its uninstall export in `crosscode.church-export/1`.
---
@@ -757,7 +802,7 @@ ranking.
| 7 | Hub | Becomes **Cross Code Hub**: every product's licences, stats, AI and billing in one place, still read-only toward installs |
| 8 | Stack | Next.js + Express + Prisma + PostgreSQL + pg-boss, as Cross Events (not the specs' React + Vite, Redis + BullMQ, Socket.IO) |
| 9 | Pricing | Income bands per product, bundles 15% (two) / 25% (three); every App module for every church (not the website spec's Starter / Growth / Pro tiers) |
| 10 | Hub provisioning | Scripts Joshua runs; the Hub records installs but never deploys or rolls back (not the specs' automated provisioning) |
| 10 | Hub provisioning | Scripts Joshua runs; the Hub records installs but never deploys or rolls back (not the specs' automated provisioning). **Changed 2026-11-12** by decisions 27–37: the server agent runs the script on signed jobs from the Hub |
| 11 | Installs per product | **One install of each product per customer**; the Hub refuses a second licence for the same product |
| 12 | House of Prayer defaults | 1-hour slots, book up to 12 months ahead, recurring bookings for up to 12 months from the day of booking, no cancelling within 8 hours of the slot |
| 13 | App notifications | **Email and WhatsApp only.** No push notifications |
@@ -775,7 +820,7 @@ ranking.
|---|----------|----------|
| 21 | Repos | **Cross Events moves into `crosscode-platform`** (`apps/events`, Events 2.2.0, history kept); `hope-events` is archived. The Hub stays in its own repo (own server, never installed for a church) |
| 22 | Contracts for the Hub | A packed release **committed to the Hub's repo** (`file:vendor/…`), not Gitea's package registry, so no deploy depends on a registry or a token (§2) |
| 23 | Deploying | **One `crosscode` script** for every product, with every church under `/var/www/sites/<church>/<product>`: `crosscode deploy <church> [events] [website] [app]` always updates Accounts and then the products named, in a fixed order, each rolled back on its own if it fails. Still git checkouts of tags, still run by Joshua; the Hub never deploys (§8) |
| 23 | Deploying | **One `crosscode` script** for every product, with every church under `/var/www/sites/<church>/<product>`: `crosscode deploy <church> [events] [website] [app]` always updates Accounts and then the products named, in a fixed order, each rolled back on its own if it fails. Still git checkouts of tags, still run by Joshua; the Hub never deploys (§8). **Changed 2026-11-12** by decision 27: from phase 1d the server agent runs it on jobs from the Hub |
| 24 | Product requirements | A release can require a minimum version of a sibling (`— requires accounts 1.2.0` in `WHATS-NEW.md`). The Hub **disables Request update** until it's met, and the script refuses to deploy without it (§5, §8) |
**Added 2026-11-05**
@@ -785,6 +830,22 @@ ranking.
| 25 | Moving existing Events installs | `crosscode adopt` **builds the new checkout next to the running install and switches PM2 over** (seconds of downtime) instead of moving the running folder; the old folder is untouched, so a failure just starts it again (§8) |
| 26 | Undoing a move | **`crosscode adopt <church> --undo`** goes back to the old folder and its 2.1.x while it's kept (14 days) (§8) |
**Added 2026-11-12** (phase 1d, [AGENT-1.0-PLAN.md](./AGENT-1.0-PLAN.md))
| # | Question | Decision |
|---|----------|----------|
| 27 | Deploying from the Hub | A **server agent** on every server runs installs, updates and uninstalls on **signed jobs of fixed types** from the Hub, always through the `crosscode` script (which still works by hand). Changes decisions 10 and 23; rule 2 (§1) reworded |
| 28 | The agent's exposure | **It has no URL or listening port.** It only polls the Hub over outgoing HTTPS |
| 29 | Confirming risky jobs | Uninstall and emergency updates need **2FA again** in the Hub; uninstall also needs the short name typed |
| 30 | Notice before updates | **Scheduled: at least 4 hours**; **emergency: at least 15 minutes**, with a reason. The site admin gets an email and a banner straight away. Joshua picks the time. A job the agent misses doesn't run late |
| 31 | Uninstall | The church gets a **JSON export in our own format** (`crosscode.church-export/1`, no internal tables or ids), encrypted so the Hub can't read it, downloadable for **30 days**. The final backup is kept 30 days, then deleted (Joshua emailed 3 days before) |
| 32 | frp | Every church server is behind frp to **one central FastPanel server**. The Hub keeps the **port list** (unique across servers, same number local and remote), tunnels are named `<church>-<product>-web/-api` |
| 33 | frpc.ini | **Only appended to**, in marked blocks, checked with `frpc verify`, with a backup first; uninstall removes only its own block. `admin_addr = 127.0.0.1` / `admin_port = 7400` added once to `[common]` so frpc reloads without dropping other sites |
| 34 | FastPanel and DNS | **Never automated.** Joshua gets an email with the proxy rules, Let's Encrypt and DNS to set up, then clicks *Proxy done*; the Hub checks the public URLs |
| 35 | Central server | A **read-only** agent on the central server checks the ports are allowed and open and that frps binds tunnels to `127.0.0.1` (`proxy_bind_addr`). It changes nothing there |
| 36 | `.env` configuration | **Database and other server values are set once per server in the Hub**, stored encrypted and sent encrypted to that server's agent with each job. `.env` files come from a per-product template; updates only append new variables; secrets are generated on the server and never leave it |
| 37 | Server stats | Every agent reports RAM, CPU, disk and PM2 processes every minute; the Hub keeps minutes for 7 days and hours for 90 days, shows spikes, and emails alerts |
## Still open
Nothing. crosscode-site is updated at the end of each phase (§9, "Updating crosscode-site"); the