Agent on every server runs installs, scheduled/emergency updates and uninstalls on signed jobs from the Hub, polls only (no listening port), appends to frpc.ini in marked blocks, builds .env files from per-server config in the Hub. FastPanel and DNS stay manual (emailed instructions). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
29 KiB
Server agent 1.0.0 + Hub 2.1.0 — installs, updates and uninstalls from the Hub
Planned 2026-11-12 (ROADMAP-v3 phase 1d, decisions 27–37). Goal: when a new church signs up, Joshua picks a server in the Hub and the rest happens by itself, with a report by email. Updates are scheduled from the Hub with notice to the church, and every server's health is visible there.
This replaces the old rule "the Hub never deploys" (decisions 10 and 23). The part that matters
stays: no site depends on the Hub to keep running, and everything still goes through the
crosscode script (EVENTS-2.2-PLAN §4), which keeps working by hand over SSH.
Contents
- The shape
- Rules
- The agent
- Jobs and their security
- Configuration: server, defaults, per install
- frp and the central server
- New install
- Updates: scheduled, emergency, batch
- Uninstall
- Servers page and alerts
- Reports and emails
- What changes in each repo
- Testing
- Rolling it out
- Decisions
1. The shape
Cross Code Hub (hub.crosscode.co.za)
servers · jobs · schedules · port list · server config · reports
▲ agents poll: "any jobs?" + stats + job results (outbound HTTPS only)
│
┌────────┴─────────────┐ ┌──────────────────────┐ ┌─────────────────────────────┐
│ Church server A │ │ Church server B │ │ Central server (FastPanel) │
│ crosscode-agent │ │ crosscode-agent │ │ crosscode-agent --central │
│ └ runs `crosscode` │ │ └ runs `crosscode` │ │ read-only: frps ports, │
│ PM2 sites, Postgres │ │ … │ │ allow_ports, bind address │
│ frpc ═══tunnel═══════╪═══╪══════════════════════╪══▶│ frps ◀─ nginx (FastPanel, │
└──────────────────────┘ └──────────────────────┘ │ set up by Joshua) │
└─────────────────────────────┘
2. Rules
- The agent listens on nothing. No HTTP server, no open port, no URL, not even on
127.0.0.1. It only makes outgoing HTTPS requests to the Hub (poll, stats, results). A test checks that the running agent has no listening socket. - The agent only runs the
crosscodescript, with arguments built from a fixed list of job types (§4). There is no "run this command" job. - Jobs are signed by the Hub and checked by the agent against the Hub key pinned when the server joined. Unsigned, expired or replayed jobs are refused and reported.
- No site depends on the Hub or the agent. If either is down, sites keep running and
crosscodestill works by hand. Scheduled jobs that can't be delivered are reported as missed, never run late without a fresh notice (§8). - The agent never touches FastPanel or DNS. FastPanel setup is an email to Joshua (§7, §11).
- Files the agent edits are only ever added to, never replaced:
/etc/frp/frpc.iniand each install's.env(§5, §6). Anything written by hand is never changed. - Secrets generated for an install stay on its server (
JWT_SECRET, the database password). They're never sent to the Hub, never put in a report or email.
3. The agent
- Where:
apps/agentin this monorepo, Node 24 + TypeScript, taggedagent@1.0.0. Installed in/opt/crosscode/agentwith its config in/etc/crosscode/agent.json(mode 600, root). - Run by systemd (
crosscode-agent.service, restarts on failure), not PM2, because it manages PM2. Runs as root, ascrosscode installneeds (PostgreSQL, PM2 for the app user, frpc). - Joining: Hub → Servers → Add server gives a one-time token (valid 1 hour). On the server:
curl … | bash -s -- --hub https://hub.crosscode.co.za --token …installs Node if needed, the agent and/opt/crosscode/tools, then the agent:- makes an Ed25519 key pair (signs its requests to the Hub) and an X25519 key pair (receives encrypted config, §5); the private keys never leave the server,
- sends the token and both public keys; the Hub replies with the server id and the Hub's job-signing public key, which the agent pins.
- Polling: every 30 s
POST /api/agent/v1/poll(signed) with a short status; the reply holds any jobs due. Every 60 s the stats (§10). Requests back off when the Hub doesn't answer. - Modes:
church(default) andcentral(§6). The central agent can't run any job butcheck-portsandstatus. - Updating itself: an
update-agentjob checks out the newestagent@…tag, builds it next to the running copy, switches the systemd unit over and reports its new version. If the new one doesn't poll within 2 minutes, a systemd timer starts the previous one again. - One job at a time per server, in the order the Hub gives them. A job has a time limit (install 30 min, deploy 20 min); a job past it is stopped and reported as failed.
4. Jobs and their security
Job types (and the crosscode command each runs):
| Job | Runs | Needs 2FA again in the Hub |
|---|---|---|
install |
crosscode install <church> --products … --json |
no |
add-product |
crosscode add <church> <product> --json |
no |
deploy |
crosscode deploy <church> [products] [--version …] --json |
emergency only |
adopt / adopt-undo |
crosscode adopt … |
no |
restart |
crosscode restart <church> [product] (PM2 reload) |
no |
export |
crosscode export <church> --json (§9) |
no |
uninstall |
crosscode uninstall <church> --json |
yes, and typing the short name |
purge |
crosscode purge <church> (deletes the 30-day keep, §9) |
no (scheduled at uninstall) |
update-agent |
§3 | no |
check-ports |
central only (§6) | no |
status |
crosscode status --json |
no |
A signed job (crosscode.agent-job/1, in packages/contracts):
{
"schema": "crosscode.agent-job/1",
"id": "job_…",
"serverId": "srv_…",
"type": "deploy",
"args": { "church": "hope", "products": ["events"], "version": "events@2.4.1" },
"config": { "alg": "x25519-xsalsa20-poly1305", "ciphertext": "…" },
"notBefore": "2026-11-20T12:00:00Z",
"expiresAt": "2026-11-20T12:15:00Z",
"signature": "…"
}
- The agent checks: the signature (pinned Hub key),
serverIdis its own, it's withinnotBefore–expiresAt, the id hasn't been seen (it keeps the ids of the last 90 days), the type is known and everyargsvalue passes that type's schema (short names^[a-z][a-z0-9-]{1,30}$, products from the enum, versions as<product>@x.y.z). Arguments are passed tocrosscodeas an argument array, never through a shell. configholds whatever the job needs from §5, encrypted to this server's X25519 key, so job rows and logs in the Hub never hold it in plain text.- Results (
crosscode.agent-report/1): the job id, status (succeeded,failed,rolled-back,refused,expired), each step with its start, end and outcome, and the last 200 log lines. The script's--jsonoutput is that report; the agent adds the log lines and sends it, signed with its own key. Logs are filtered for anything that looks like a secret before they leave the server. - The Hub side: job-signing private key in the Hub's
.env(not in the database), its own key separate from the licence-signing key. 2FA re-confirmation for uninstall and emergency deploys. Every job is in the audit log with who made it.
5. Configuration: server, defaults, per install
An install's .env files are built from four layers. Each variable comes from exactly one layer.
| Layer | Set where | Examples | Sent how |
|---|---|---|---|
| Server | Hub → Servers → server → Configuration, once per server (editable) | Database host, port and an admin role that can create databases and roles (or "local PostgreSQL, through sudo -u postgres"), TRUST_PROXY, app user, backup folder |
Stored AES-GCM encrypted in the Hub (as install secrets are today); sent to the agent encrypted to its key with each job that needs it. The agent doesn't keep it on disk |
| Defaults | Hub → Settings → Install defaults, once | HUB_URL, APP_TIMEZONE (Africa/Johannesburg), job switches (DAILY_SUMMARY_ENABLED=true, BACKUP_ENABLED=true, …), HUB_HEARTBEAT_ENABLED=true, optional platform SENTRY_DSN |
With the job, encrypted the same way |
| Install | The New install form (§7) | Domain → FRONTEND_URL, APP_BASE_URL, BACKEND_URL and the frontend's NEXT_PUBLIC_API_URL (all https://<domain>, since FastPanel sends /api to the backend on the same domain); timezone if not the default; the licence key; ports from the port list (§6) → PORT |
With the job |
| Generated | On the server by crosscode install |
JWT_SECRET, the database password (so DATABASE_URL), NODE_ENV=production |
Never leaves the server (rule 7) |
Per-product env template. Each product has apps/<product>/env.template.json in the
monorepo, listing every variable with its layer, whether it's required, and a default:
{ "schema": "crosscode.env-template/1", "product": "events", "files": {
"backend/.env": [
{ "name": "DATABASE_URL", "layer": "generated", "required": true },
{ "name": "FRONTEND_URL", "layer": "install", "from": "https://{domain}", "required": true },
{ "name": "TRUST_PROXY", "layer": "server", "required": true },
{ "name": "BACKUP_ENABLED", "layer": "defaults", "default": "true" }
],
"frontend/.env.production": [
{ "name": "NEXT_PUBLIC_API_URL", "layer": "install", "from": "https://{domain}", "required": true }
] } }
- The Hub reads the template of the release being installed (synced with the release, §5 of the roadmap) and refuses to create the job if a required server or defaults value is missing, saying which, before anything runs on the server.
- Install:
crosscode installwrites each.envfrom the template (mode 600, app user). - Updates: before building,
crosscode deploycompares the new release's template with the.env. A new variable is appended with its value (and a comment with the date and release); existing lines are never changed or removed. A new required variable with no value in any layer stops the deploy at the checks, before anything is touched. - Changing a server value later (e.g. a new database host) is a
reconfigurestep of a deploy: the agent shows which lines would change in each install, and only changes them when Joshua confirms in the Hub. This is the one exception to "only append", and it never runs by itself. - URLs are checked before and after. Before: the form checks that the domain's DNS points at
the central server and warns if not (the install still runs). After Joshua marks the proxy done:
the Hub checks
https://<domain>/(frontend answers),https://<domain>/health(backend answers with the right version) and that/healthreports theFRONTEND_URLit was given.
For Joshua to set per server before the first install: the database settings and
TRUST_PROXY. Behind frp the app sees every connection from frpc on 127.0.0.1, and the
visitor's address arrives in X-Forwarded-For from FastPanel's nginx, so the expected value is
loopback (the Hub suggests it).
6. frp and the central server
Today: /etc/frp/frpc.ini on each church server, tcp tunnels, frontend and backend each
with their own remote_port. FastPanel on the central server proxies a domain to
127.0.0.1:<remote_port> with Let's Encrypt.
Port list in the Hub. frps opens every remote_port on the one central server, so they must
be unique across every church server. The Hub hands them out from a range (default
20000–29999, set in Hub settings to match allow_ports), two per product install (web, api),
and uses the same number for local_port, so it's also free on the church server and there's
one number per tunnel to think about. Ports already in use (today's 3000/5000 and anything
the central agent sees listening) are recorded as taken. Ports are released 30 days after an
uninstall.
Tunnel names must also be unique on frps: <church>-<product>-web and
<church>-<product>-api (e.g. hope-events-web). Existing entries such as [events-frontend]
stay as they are.
What the agent adds to frpc.ini (one block per product install, appended at the end):
# >>> crosscode:hope:events (added by crosscode-agent 2026-11-20, do not edit by hand)
[hope-events-web]
type = tcp
local_ip = 127.0.0.1
local_port = 20010
remote_port = 20010
[hope-events-api]
type = tcp
local_ip = 127.0.0.1
local_port = 20011
remote_port = 20011
# <<< crosscode:hope:events
- Only appended to, never replaced. Before a change: a copy to
/etc/frp/backups/frpc.ini.<timestamp>; write to a temporary file;frpc verify -c; then move it into place. If the check fails, nothing is changed. Uninstall removes only the lines between that install's own markers; nothing outside a marked block is ever changed. - Reload without dropping other sites. Once per server (agreed 2026-11-12) the agent adds to
[common], if they're not there:soadmin_addr = 127.0.0.1 admin_port = 7400frpc reload -c /etc/frp/frpc.inipicks up new tunnels without restarting frpc. This is the only change ever made in[common], and it's reported. (admin_addris loopback only, so it adds no way in from outside.) - The
.iniformat is deprecated in newer frp versions in favour of TOML. The agent reads the format the server has; moving to TOML is a separate job for later.
The central agent (--central, on the FastPanel server, read-only) answers check-ports:
- the port is inside
allow_portsinfrps.ini, - frps has the tunnel open (the port is listening; frps's dashboard API is used if it's enabled
on
127.0.0.1), proxy_bind_addr = 127.0.0.1is set, so tunnels are only reachable through FastPanel's nginx, not straight from the internet on<central IP>:<port>(which would bypass HTTPS and expose the backend). If it's missing, every report and the Servers page show a warning.
It changes nothing on the central server: no frps edits, no firewall changes, no FastPanel.
7. New install
Hub → Customers → customer → New install (or a new customer first):
- Form: products (Events now; Accounts / Website / App as they ship), short name
(
hope,cross,bridge; checked unique across all servers and used for folders, PM2 names, database names and tunnel names), address (own domain or Cross Code address; per product the domain is shown, e.g.events.hope.co.za), timezone, site admin name and email, and the server: a list showing each server's free RAM, free disk, CPU load over the last day, sites on it and agent status, with servers below the free-RAM threshold (default 1.5 GB per product install) marked as not recommended. - Checks in the Hub before the job exists: the server agent is online, server config and defaults are complete for the release's env template (§5), ports are free, the short name is free, DNS (warning only).
- The Hub creates the licence(s), assigns ports, and queues the
installjob. - On the server (
crosscode install, each step in the report): checks (disk, RAM, Node, PostgreSQL reachable) → folder/var/www/sites/<church>/<product>→ checkout of the release tag →.envfiles (§5) → database and role → install dependencies → migrations → build → PM2 processes on the given ports,pm2 save→church.env→ frpc block and reload (§6) → local/healthanswers with the right version → the site registers with the Hub using its licence. - On failure anything this job created is removed again (PM2 processes, frpc block, database and role, folder), in reverse order, and the report says what failed and what was cleaned up.
- Central check: the Hub asks the central agent to
check-portsfor both ports. - Emails to Joshua: the install report and, if everything passed, the FastPanel email (§11). The install shows Waiting for proxy in the Hub.
- Joshua sets up FastPanel by hand and clicks Proxy done. The Hub checks the public URLs (§5) and marks it Live, or says what didn't answer. When Live, the site admin gets the welcome email with the setup-wizard link.
8. Updates: scheduled, emergency, batch
Scheduled (Hub → Install → Schedule update, or Updates → Schedule):
- Joshua picks the release and the time. The time must be at least 4 hours away; the Hub refuses anything sooner. A warning (not a block) for Sunday 06:00–13:00 and for days with an event of that church running (from the stats).
- Straight away: an email to the site admin(s) ("Cross Events will be updated on Thursday 20 November at 14:00; the site will be unavailable for a few minutes", plus the what's-new summary), and a banner through the existing Notices, starting now and ending after the update. A reminder email 1 hour before.
- At the time the Hub releases the
deployjob (notBefore= the time,expiresAt= 15 min later). If the agent doesn't pick it up in that window the update is missed: nothing runs, Joshua is emailed, and a new time needs a new notice. - Afterwards: the report to Joshua; to the site admin "Updated to 2.4.1" with what's new, or "The update didn't go through; the site is running the previous version" if it rolled back.
- Cancelling or moving it updates the banner and emails the site admin.
Emergency: the same, with at least 15 minutes' notice, a required reason (shown in the email and banner), and 2FA again. Meant for security fixes and broken releases.
Batch: "every Events install to 2.4.1" (or a selection), at one time with one notice. The Hub runs them one at a time (on each server, and across servers) and stops the batch at the first failure, emailing Joshua; the remaining installs get a "postponed" email and their banner is removed.
Rules carried over: the release's requirements (— requires accounts 1.2.0, decision 24)
are checked in the Hub before scheduling and again by the script; each product rolls back on its
own (§8 of the roadmap). A church's Request update now shows as "Requested" next to
Schedule update. Moving a 2.1.x install to 2.2.0 is an adopt job, scheduled the same way.
9. Uninstall
Hub → Install → Uninstall (2FA again, type the short name):
- Joshua picks the date (at least 4 hours away, default 7 days) and the site admin is emailed and gets a banner, as for an update.
- Export first (
crosscode export): the church's data as JSON in a format of our own,crosscode.church-export/1(defined inpackages/contracts): events, sessions, ticket types, registrations and attendees, people (name, email, phone, consent), payments (amounts, dates, references), check-ins, rosters, settings, and the uploads as files next to it. Written in plain terms with our own field names; no database tables, Prisma models or internal ids. One ZIP:export.json,uploads/,README.txtexplaining the fields. - The agent encrypts the ZIP with a random key and uploads it to the Hub. The key is only in
the link emailed to the site admin (
…/download/<id>#<key>; the part after#is never sent to the Hub), so the Hub stores the file but can't read it, and the roadmap's "the Hub only sees aggregate stats" still holds. The download page decrypts it in the browser. Available for 30 days. - Uninstall: final database backup and an archive of the uploads to
/var/www/sites/.removed/<church>-<date>/(root only) → PM2 processes removed,pm2 save→ the frpc block removed and reloaded → database and role dropped → the site folder removed → the licence released in the Hub. - Emails: to the site admin, the download link and the date it expires; to Joshua, the report
and "remove the FastPanel site
events.hope.co.za(ports 20010, 20011)". - After 30 days: a
purgejob deletes the kept backup and archive, and the Hub deletes the encrypted export. Joshua is emailed 3 days before, and can extend it once by 30 days. The Hub keeps a record that it was deleted, and when (for POPIA).
10. Servers page and alerts
Hub → Servers: one card per server (church servers and the central one).
- Every 60 s the agent sends: RAM (total, used, available, swap), CPU (% and 1/5/15-minute load), disk per mount (used, free), uptime, versions (OS, Node, PostgreSQL, frpc, agent), frpc running, and per PM2 process (church, product, memory, CPU, restarts, uptime).
- Kept: per minute for 7 days, per hour (average and peak) for 90 days.
- Server page: RAM and CPU charts for 24 h / 7 days / 90 days with spikes marked (peaks above 85%), disk bars, the sites on it with each process's memory, recent jobs, the frpc blocks it manages and the ports they use.
- Alerts (email to Joshua, through the existing alerts, once per incident and again when it
clears): agent not heard from for 5 minutes; disk over 85%; RAM over 90% for 5 minutes; a PM2
process restarting more than 5 times in 10 minutes; frpc not running; on the central server,
proxy_bind_addrmissing or a managed tunnel not listening.
11. Reports and emails
All in the Hub (Jobs page, with the full report) and by email:
| To | When | |
|---|---|---|
| Install report | Joshua | Every install, success or failure: each step with its time, ports, the central check, the log tail on failure, what was cleaned up |
| FastPanel setup | Joshua | After a successful install. What he does in FastPanel; the agent does none of it (below) |
| Welcome | Site admin | When the install is Live |
| Update planned / reminder / done / didn't go through | Site admin | §8 |
| Update report | Joshua | Every update (and batch summary) |
| Missed job | Joshua | A scheduled job wasn't picked up in time |
| Uninstall planned / export ready | Site admin | §9 |
| Uninstall report, purge in 3 days | Joshua | §9 |
| Alerts | Joshua | §10 |
The FastPanel email (an instruction to Joshua, nothing is set up automatically):
Set up in FastPanel: events.hope.co.za (Hope Family Church, Cross Events)
1. Add the site events.hope.co.za.
2. Proxy:
/api, /uploads, /health, /docs → http://127.0.0.1:20011 (backend)
everything else → http://127.0.0.1:20010 (frontend)
Pass the Host header and X-Forwarded-For / X-Forwarded-Proto: https.
Allow uploads up to 25 MB.
3. Turn on Let's Encrypt for the site.
4. DNS: events.hope.co.za → <central server IP> (checked: ✅ points there / ⚠ not yet)
5. Then click "Proxy done" in the Hub: <link>
Central check: both ports are open in frps and only on 127.0.0.1.
12. What changes in each repo
crosscode-platform
apps/agent(new,agent@1.0.0) with its install script and systemd unit.scripts/(thecrosscodescript):--jsonreports on every command,uninstall,export,purge,restart, the frpc block handling (lib/frpc.sh), the env template handling (lib/env.sh), ports given on the command line instead of picked.apps/events:env.template.json, and the export (crosscode exportcalls a backend script that writescrosscode.church-export/1).packages/contracts1.2.0:crosscode.agent-job/1,crosscode.agent-report/1,crosscode.agent-stats/1,crosscode.env-template/1,crosscode.church-export/1.
events-hub (Hub 2.1.0)
- Models:
Server(name, mode, public keys, encrypted config, last seen, versions),ServerStat(minute and hourly),Job(type, args, status, schedule, who, report),ScheduledUpdate/ batch,Port(number, server, install, released at),Export(encrypted blob, expires).InstallgetsserverId,shortName,status(installing, waiting-for-proxy, live, uninstalling, removed). - Agent API
/api/agent/v1/*(join, poll, stats, report, export upload), signed by the agent. - Pages: Servers, Server, New install, Jobs, Updates (schedule, batch), Install defaults, the public export download page.
- Emails in §11; alerts in §10; the job-signing key in
.env.
crosscode-site: nothing (this changes how Joshua works, not what churches buy).
13. Testing
- Agent: unit tests for job checking (bad signature, wrong server, expired, replayed,
unknown type, an argument with shell characters), and a test that the running agent has no
listening socket (
ss -ltnpshows nothing for its process). - Script: bats tests for the frpc blocks (append to a file with hand-written entries and
comments, remove one block, the other blocks and the hand-written lines unchanged byte for byte;
a failed
frpc verifyleaves the file untouched;[common]admin lines added only once), the env template (new variable appended, existing lines untouched, required one missing → refused), and the export (a demo church's export validates againstcrosscode.church-export/1and has no internal ids). - On VMs (multipass): a "central" VM with frps (with
proxy_bind_addr = 127.0.0.1) and nginx in place of FastPanel, two church VMs with frpc and a hand-written tunnel, the Hub running locally:- Join all three. Stats show up.
- New install on church VM 1 → report, FastPanel email, ports open on the central VM, the hand-written tunnel still works. Add the nginx site by hand, Proxy done → Live.
- A second church on VM 2 → different ports, unique tunnel names.
- Schedule an update 4 h ahead (time moved forward in the test) → emails, banner, update, report. Try 3 h → refused. Emergency with 15 min → needs 2FA.
- A broken release → rolled back, "didn't go through" email, batch stopped.
- Hub stopped during a scheduled update → missed, nothing runs, sites keep running.
- Uninstall → export downloads and decrypts in the browser, everything removed, the other church's tunnel and the hand-written one untouched. Purge after 30 days (clock moved).
- A forged job (wrong key) and a replayed job → refused and reported.
14. Rolling it out
- Contracts 1.2.0 → the
crosscodescript changes (with Events 2.2.0 out first) → agent 1.0.0 → Hub 2.1.0, each released as usual (branch → merge → tag → push). - On the central server: add
proxy_bind_addr = 127.0.0.1tofrps.iniif it's missing (check that each FastPanel site still answers), then join the central agent. - Join each church server; set its server config in the Hub. Existing installs are linked to
their server and their current ports recorded in the port list (read from
church.envandfrpc.ini; nothing is changed). - First real use: the next new church. Existing churches get their next update through a scheduled update.
- Docs: CHANGELOGs, HANDOVER.md (1d done),
events-hub/deployment.md(the job-signing key), a server how-to inapps/agent/README.md.
Decisions
Agreed with Joshua on 2026-11-12 (ROADMAP-v3 decisions 27–37):
- The Hub can install, update and uninstall through an agent on each server, with signed jobs
of fixed types; the agent only runs the
crosscodescript; uninstall and emergency updates need 2FA again. - The agent has no URL or listening port; it only polls the Hub.
- Scheduled updates at least 4 hours ahead, emergency updates at least 15 minutes, with an email and banner to the site admin; Joshua picks the time.
- Uninstall: the church gets a JSON export in our own format (no internal data structures), encrypted so the Hub can't read it, downloadable for 30 days; backups kept 30 days, then deleted.
- All church servers are behind frp to one central FastPanel server. The Hub keeps the port
list; frpc.ini is only appended to; the
[common]admin lines are added once. - FastPanel and DNS are never automated; Joshua gets an email with what to set up.
- Database and other server settings are set once per server in the Hub and sent encrypted
with each job;
.envfiles are built from a per-product template and only appended to on updates; generated secrets stay on the server. - A read-only agent on the central server checks ports and
proxy_bind_addr.
Still open
Nothing.