Agent on every server runs installs, scheduled/emergency updates and uninstalls on signed jobs from the Hub, polls only (no listening port), appends to frpc.ini in marked blocks, builds .env files from per-server config in the Hub. FastPanel and DNS stay manual (emailed instructions). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
478 lines
29 KiB
Markdown
478 lines
29 KiB
Markdown
# Server agent 1.0.0 + Hub 2.1.0 — installs, updates and uninstalls from the Hub
|
||
|
||
Planned 2026-11-12 (ROADMAP-v3 phase **1d**, decisions 27–37). Goal: when a new church signs up,
|
||
Joshua picks a server in the Hub and the rest happens by itself, with a report by email. Updates
|
||
are scheduled from the Hub with notice to the church, and every server's health is visible there.
|
||
|
||
This replaces the old rule "the Hub never deploys" (decisions 10 and 23). The part that matters
|
||
stays: **no site depends on the Hub to keep running**, and everything still goes through the
|
||
`crosscode` script (EVENTS-2.2-PLAN §4), which keeps working by hand over SSH.
|
||
|
||
## Contents
|
||
|
||
1. [The shape](#1-the-shape)
|
||
2. [Rules](#2-rules)
|
||
3. [The agent](#3-the-agent)
|
||
4. [Jobs and their security](#4-jobs-and-their-security)
|
||
5. [Configuration: server, defaults, per install](#5-configuration-server-defaults-per-install)
|
||
6. [frp and the central server](#6-frp-and-the-central-server)
|
||
7. [New install](#7-new-install)
|
||
8. [Updates: scheduled, emergency, batch](#8-updates-scheduled-emergency-batch)
|
||
9. [Uninstall](#9-uninstall)
|
||
10. [Servers page and alerts](#10-servers-page-and-alerts)
|
||
11. [Reports and emails](#11-reports-and-emails)
|
||
12. [What changes in each repo](#12-what-changes-in-each-repo)
|
||
13. [Testing](#13-testing)
|
||
14. [Rolling it out](#14-rolling-it-out)
|
||
15. [Decisions](#decisions)
|
||
|
||
---
|
||
|
||
## 1. The shape
|
||
|
||
```
|
||
Cross Code Hub (hub.crosscode.co.za)
|
||
servers · jobs · schedules · port list · server config · reports
|
||
▲ agents poll: "any jobs?" + stats + job results (outbound HTTPS only)
|
||
│
|
||
┌────────┴─────────────┐ ┌──────────────────────┐ ┌─────────────────────────────┐
|
||
│ Church server A │ │ Church server B │ │ Central server (FastPanel) │
|
||
│ crosscode-agent │ │ crosscode-agent │ │ crosscode-agent --central │
|
||
│ └ runs `crosscode` │ │ └ runs `crosscode` │ │ read-only: frps ports, │
|
||
│ PM2 sites, Postgres │ │ … │ │ allow_ports, bind address │
|
||
│ frpc ═══tunnel═══════╪═══╪══════════════════════╪══▶│ frps ◀─ nginx (FastPanel, │
|
||
└──────────────────────┘ └──────────────────────┘ │ set up by Joshua) │
|
||
└─────────────────────────────┘
|
||
```
|
||
|
||
## 2. Rules
|
||
|
||
1. **The agent listens on nothing.** No HTTP server, no open port, no URL, not even on
|
||
`127.0.0.1`. It only makes outgoing HTTPS requests to the Hub (poll, stats, results). A test
|
||
checks that the running agent has no listening socket.
|
||
2. **The agent only runs the `crosscode` script**, with arguments built from a fixed list of job
|
||
types (§4). There is no "run this command" job.
|
||
3. **Jobs are signed by the Hub** and checked by the agent against the Hub key pinned when the
|
||
server joined. Unsigned, expired or replayed jobs are refused and reported.
|
||
4. **No site depends on the Hub or the agent.** If either is down, sites keep running and
|
||
`crosscode` still works by hand. Scheduled jobs that can't be delivered are reported as missed,
|
||
never run late without a fresh notice (§8).
|
||
5. **The agent never touches FastPanel or DNS.** FastPanel setup is an email to Joshua (§7, §11).
|
||
6. **Files the agent edits are only ever added to, never replaced**: `/etc/frp/frpc.ini` and each
|
||
install's `.env` (§5, §6). Anything written by hand is never changed.
|
||
7. **Secrets generated for an install stay on its server** (`JWT_SECRET`, the database password).
|
||
They're never sent to the Hub, never put in a report or email.
|
||
|
||
## 3. The agent
|
||
|
||
- **Where:** `apps/agent` in this monorepo, Node 24 + TypeScript, tagged `agent@1.0.0`. Installed
|
||
in `/opt/crosscode/agent` with its config in `/etc/crosscode/agent.json` (mode 600, root).
|
||
- **Run by systemd** (`crosscode-agent.service`, restarts on failure), not PM2, because it manages
|
||
PM2. Runs as root, as `crosscode install` needs (PostgreSQL, PM2 for the app user, frpc).
|
||
- **Joining:** Hub → Servers → **Add server** gives a one-time token (valid 1 hour). On the server:
|
||
`curl … | bash -s -- --hub https://hub.crosscode.co.za --token …` installs Node if needed, the
|
||
agent and `/opt/crosscode/tools`, then the agent:
|
||
1. makes an **Ed25519 key pair** (signs its requests to the Hub) and an **X25519 key pair**
|
||
(receives encrypted config, §5); the private keys never leave the server,
|
||
2. sends the token and both public keys; the Hub replies with the server id and **the Hub's
|
||
job-signing public key**, which the agent pins.
|
||
- **Polling:** every 30 s `POST /api/agent/v1/poll` (signed) with a short status; the reply holds
|
||
any jobs due. Every 60 s the stats (§10). Requests back off when the Hub doesn't answer.
|
||
- **Modes:** `church` (default) and `central` (§6). The central agent can't run any job but
|
||
`check-ports` and `status`.
|
||
- **Updating itself:** an `update-agent` job checks out the newest `agent@…` tag, builds it next
|
||
to the running copy, switches the systemd unit over and reports its new version. If the new one
|
||
doesn't poll within 2 minutes, a systemd timer starts the previous one again.
|
||
- **One job at a time per server**, in the order the Hub gives them. A job has a time limit
|
||
(install 30 min, deploy 20 min); a job past it is stopped and reported as failed.
|
||
|
||
## 4. Jobs and their security
|
||
|
||
**Job types** (and the `crosscode` command each runs):
|
||
|
||
| Job | Runs | Needs 2FA again in the Hub |
|
||
|-----|------|:-:|
|
||
| `install` | `crosscode install <church> --products … --json` | no |
|
||
| `add-product` | `crosscode add <church> <product> --json` | no |
|
||
| `deploy` | `crosscode deploy <church> [products] [--version …] --json` | emergency only |
|
||
| `adopt` / `adopt-undo` | `crosscode adopt …` | no |
|
||
| `restart` | `crosscode restart <church> [product]` (PM2 reload) | no |
|
||
| `export` | `crosscode export <church> --json` (§9) | no |
|
||
| `uninstall` | `crosscode uninstall <church> --json` | **yes**, and typing the short name |
|
||
| `purge` | `crosscode purge <church>` (deletes the 30-day keep, §9) | no (scheduled at uninstall) |
|
||
| `update-agent` | §3 | no |
|
||
| `check-ports` | central only (§6) | no |
|
||
| `status` | `crosscode status --json` | no |
|
||
|
||
**A signed job** (`crosscode.agent-job/1`, in `packages/contracts`):
|
||
|
||
```json
|
||
{
|
||
"schema": "crosscode.agent-job/1",
|
||
"id": "job_…",
|
||
"serverId": "srv_…",
|
||
"type": "deploy",
|
||
"args": { "church": "hope", "products": ["events"], "version": "events@2.4.1" },
|
||
"config": { "alg": "x25519-xsalsa20-poly1305", "ciphertext": "…" },
|
||
"notBefore": "2026-11-20T12:00:00Z",
|
||
"expiresAt": "2026-11-20T12:15:00Z",
|
||
"signature": "…"
|
||
}
|
||
```
|
||
|
||
- The agent checks: the signature (pinned Hub key), `serverId` is its own, it's within
|
||
`notBefore`–`expiresAt`, the id hasn't been seen (it keeps the ids of the last 90 days), the
|
||
type is known and every `args` value passes that type's schema (short names
|
||
`^[a-z][a-z0-9-]{1,30}$`, products from the enum, versions as `<product>@x.y.z`). Arguments are
|
||
passed to `crosscode` as an argument array, never through a shell.
|
||
- `config` holds whatever the job needs from §5, encrypted to **this server's** X25519 key, so
|
||
job rows and logs in the Hub never hold it in plain text.
|
||
- **Results** (`crosscode.agent-report/1`): the job id, status (`succeeded`, `failed`,
|
||
`rolled-back`, `refused`, `expired`), each step with its start, end and outcome, and the last
|
||
200 log lines. The script's `--json` output is that report; the agent adds the log lines and
|
||
sends it, signed with its own key. Logs are filtered for anything that looks like a secret
|
||
before they leave the server.
|
||
- **The Hub side:** job-signing private key in the Hub's `.env` (not in the database), its own
|
||
key separate from the licence-signing key. 2FA re-confirmation for uninstall and emergency
|
||
deploys. Every job is in the audit log with who made it.
|
||
|
||
## 5. Configuration: server, defaults, per install
|
||
|
||
An install's `.env` files are built from four layers. Each variable comes from exactly one layer.
|
||
|
||
| Layer | Set where | Examples | Sent how |
|
||
|-------|-----------|----------|----------|
|
||
| **Server** | Hub → Servers → *server* → Configuration, **once per server** (editable) | Database host, port and an admin role that can create databases and roles (or "local PostgreSQL, through `sudo -u postgres`"), `TRUST_PROXY`, app user, backup folder | Stored AES-GCM encrypted in the Hub (as install secrets are today); sent to the agent encrypted to its key with each job that needs it. The agent doesn't keep it on disk |
|
||
| **Defaults** | Hub → Settings → Install defaults, once | `HUB_URL`, `APP_TIMEZONE` (Africa/Johannesburg), job switches (`DAILY_SUMMARY_ENABLED=true`, `BACKUP_ENABLED=true`, …), `HUB_HEARTBEAT_ENABLED=true`, optional platform `SENTRY_DSN` | With the job, encrypted the same way |
|
||
| **Install** | The New install form (§7) | Domain → `FRONTEND_URL`, `APP_BASE_URL`, `BACKEND_URL` and the frontend's `NEXT_PUBLIC_API_URL` (all `https://<domain>`, since FastPanel sends `/api` to the backend on the same domain); timezone if not the default; the licence key; ports from the port list (§6) → `PORT` | With the job |
|
||
| **Generated** | On the server by `crosscode install` | `JWT_SECRET`, the database password (so `DATABASE_URL`), `NODE_ENV=production` | Never leaves the server (rule 7) |
|
||
|
||
**Per-product env template.** Each product has `apps/<product>/env.template.json` in the
|
||
monorepo, listing every variable with its layer, whether it's required, and a default:
|
||
|
||
```json
|
||
{ "schema": "crosscode.env-template/1", "product": "events", "files": {
|
||
"backend/.env": [
|
||
{ "name": "DATABASE_URL", "layer": "generated", "required": true },
|
||
{ "name": "FRONTEND_URL", "layer": "install", "from": "https://{domain}", "required": true },
|
||
{ "name": "TRUST_PROXY", "layer": "server", "required": true },
|
||
{ "name": "BACKUP_ENABLED", "layer": "defaults", "default": "true" }
|
||
],
|
||
"frontend/.env.production": [
|
||
{ "name": "NEXT_PUBLIC_API_URL", "layer": "install", "from": "https://{domain}", "required": true }
|
||
] } }
|
||
```
|
||
|
||
- The Hub reads the template of the release being installed (synced with the release, §5 of the
|
||
roadmap) and **refuses to create the job if a required server or defaults value is missing**,
|
||
saying which, before anything runs on the server.
|
||
- **Install:** `crosscode install` writes each `.env` from the template (mode 600, app user).
|
||
- **Updates:** before building, `crosscode deploy` compares the new release's template with the
|
||
`.env`. A new variable is **appended** with its value (and a comment with the date and
|
||
release); existing lines are never changed or removed. A new *required* variable with no value
|
||
in any layer stops the deploy at the checks, before anything is touched.
|
||
- **Changing a server value later** (e.g. a new database host) is a `reconfigure` step of a
|
||
deploy: the agent shows which lines would change in each install, and only changes them when
|
||
Joshua confirms in the Hub. This is the one exception to "only append", and it never runs by
|
||
itself.
|
||
- **URLs are checked before and after.** Before: the form checks that the domain's DNS points at
|
||
the central server and warns if not (the install still runs). After Joshua marks the proxy done:
|
||
the Hub checks `https://<domain>/` (frontend answers), `https://<domain>/health` (backend
|
||
answers with the right version) and that `/health` reports the `FRONTEND_URL` it was given.
|
||
|
||
**For Joshua to set per server before the first install:** the database settings and
|
||
`TRUST_PROXY`. Behind frp the app sees every connection from frpc on `127.0.0.1`, and the
|
||
visitor's address arrives in `X-Forwarded-For` from FastPanel's nginx, so the expected value is
|
||
`loopback` (the Hub suggests it).
|
||
|
||
## 6. frp and the central server
|
||
|
||
**Today:** `/etc/frp/frpc.ini` on each church server, `tcp` tunnels, frontend and backend each
|
||
with their own `remote_port`. FastPanel on the central server proxies a domain to
|
||
`127.0.0.1:<remote_port>` with Let's Encrypt.
|
||
|
||
**Port list in the Hub.** frps opens every `remote_port` on the one central server, so they must
|
||
be unique across every church server. The Hub hands them out from a range (default
|
||
**20000–29999**, set in Hub settings to match `allow_ports`), two per product install (web, api),
|
||
and **uses the same number for `local_port`**, so it's also free on the church server and there's
|
||
one number per tunnel to think about. Ports already in use (today's `3000`/`5000` and anything
|
||
the central agent sees listening) are recorded as taken. Ports are released 30 days after an
|
||
uninstall.
|
||
|
||
**Tunnel names** must also be unique on frps: `<church>-<product>-web` and
|
||
`<church>-<product>-api` (e.g. `hope-events-web`). Existing entries such as `[events-frontend]`
|
||
stay as they are.
|
||
|
||
**What the agent adds to frpc.ini** (one block per product install, appended at the end):
|
||
|
||
```ini
|
||
# >>> crosscode:hope:events (added by crosscode-agent 2026-11-20, do not edit by hand)
|
||
[hope-events-web]
|
||
type = tcp
|
||
local_ip = 127.0.0.1
|
||
local_port = 20010
|
||
remote_port = 20010
|
||
|
||
[hope-events-api]
|
||
type = tcp
|
||
local_ip = 127.0.0.1
|
||
local_port = 20011
|
||
remote_port = 20011
|
||
# <<< crosscode:hope:events
|
||
```
|
||
|
||
- **Only appended to, never replaced.** Before a change: a copy to
|
||
`/etc/frp/backups/frpc.ini.<timestamp>`; write to a temporary file; `frpc verify -c`; then move
|
||
it into place. If the check fails, nothing is changed. Uninstall removes only the lines between
|
||
that install's own markers; nothing outside a marked block is ever changed.
|
||
- **Reload without dropping other sites.** Once per server (agreed 2026-11-12) the agent adds to
|
||
`[common]`, if they're not there:
|
||
```ini
|
||
admin_addr = 127.0.0.1
|
||
admin_port = 7400
|
||
```
|
||
so `frpc reload -c /etc/frp/frpc.ini` picks up new tunnels without restarting frpc. This is the
|
||
only change ever made in `[common]`, and it's reported. (`admin_addr` is loopback only, so it
|
||
adds no way in from outside.)
|
||
- The `.ini` format is deprecated in newer frp versions in favour of TOML. The agent reads the
|
||
format the server has; moving to TOML is a separate job for later.
|
||
|
||
**The central agent** (`--central`, on the FastPanel server, read-only) answers `check-ports`:
|
||
|
||
- the port is inside `allow_ports` in `frps.ini`,
|
||
- frps has the tunnel open (the port is listening; frps's dashboard API is used if it's enabled
|
||
on `127.0.0.1`),
|
||
- **`proxy_bind_addr = 127.0.0.1` is set**, so tunnels are only reachable through FastPanel's
|
||
nginx, not straight from the internet on `<central IP>:<port>` (which would bypass HTTPS and
|
||
expose the backend). If it's missing, every report and the Servers page show a warning.
|
||
|
||
It changes nothing on the central server: no frps edits, no firewall changes, no FastPanel.
|
||
|
||
## 7. New install
|
||
|
||
**Hub → Customers → *customer* → New install** (or a new customer first):
|
||
|
||
1. **Form:** products (Events now; Accounts / Website / App as they ship), **short name**
|
||
(`hope`, `cross`, `bridge`; checked unique across all servers and used for folders, PM2 names,
|
||
database names and tunnel names), address (own domain or Cross Code address; per product the
|
||
domain is shown, e.g. `events.hope.co.za`), timezone, **site admin name and email**, and the
|
||
**server**: a list showing each server's free RAM, free disk, CPU load over the last day, sites
|
||
on it and agent status, with servers below the free-RAM threshold (default 1.5 GB per product
|
||
install) marked as not recommended.
|
||
2. **Checks in the Hub** before the job exists: the server agent is online, server config and
|
||
defaults are complete for the release's env template (§5), ports are free, the short name is
|
||
free, DNS (warning only).
|
||
3. The Hub **creates the licence(s)**, assigns ports, and queues the `install` job.
|
||
4. **On the server** (`crosscode install`, each step in the report):
|
||
checks (disk, RAM, Node, PostgreSQL reachable) → folder `/var/www/sites/<church>/<product>`
|
||
→ checkout of the release tag → `.env` files (§5) → database and role → install dependencies
|
||
→ migrations → build → PM2 processes on the given ports, `pm2 save` → `church.env` → frpc
|
||
block and reload (§6) → local `/health` answers with the right version → the site registers
|
||
with the Hub using its licence.
|
||
5. **On failure** anything this job created is removed again (PM2 processes, frpc block, database
|
||
and role, folder), in reverse order, and the report says what failed and what was cleaned up.
|
||
6. **Central check:** the Hub asks the central agent to `check-ports` for both ports.
|
||
7. **Emails to Joshua:** the install report and, if everything passed, the **FastPanel email**
|
||
(§11). The install shows **Waiting for proxy** in the Hub.
|
||
8. Joshua sets up FastPanel by hand and clicks **Proxy done**. The Hub checks the public URLs
|
||
(§5) and marks it **Live**, or says what didn't answer. When Live, the site admin gets the
|
||
welcome email with the setup-wizard link.
|
||
|
||
## 8. Updates: scheduled, emergency, batch
|
||
|
||
**Scheduled** (Hub → Install → Schedule update, or Updates → Schedule):
|
||
|
||
- Joshua picks the release and the time. The time must be **at least 4 hours away**; the Hub
|
||
refuses anything sooner. A warning (not a block) for Sunday 06:00–13:00 and for days with an
|
||
event of that church running (from the stats).
|
||
- Straight away: an email to the site admin(s) ("Cross Events will be updated on Thursday 20
|
||
November at 14:00; the site will be unavailable for a few minutes", plus the what's-new
|
||
summary), and a **banner** through the existing Notices, starting now and ending after the
|
||
update. A reminder email 1 hour before.
|
||
- At the time the Hub releases the `deploy` job (`notBefore` = the time, `expiresAt` = 15 min
|
||
later). If the agent doesn't pick it up in that window the update is **missed**: nothing runs,
|
||
Joshua is emailed, and a new time needs a new notice.
|
||
- Afterwards: the report to Joshua; to the site admin "Updated to 2.4.1" with what's new, or
|
||
"The update didn't go through; the site is running the previous version" if it rolled back.
|
||
- Cancelling or moving it updates the banner and emails the site admin.
|
||
|
||
**Emergency:** the same, with at least **15 minutes'** notice, a required reason (shown in the
|
||
email and banner), and 2FA again. Meant for security fixes and broken releases.
|
||
|
||
**Batch:** "every Events install to 2.4.1" (or a selection), at one time with one notice. The
|
||
Hub runs them **one at a time** (on each server, and across servers) and **stops the batch at the
|
||
first failure**, emailing Joshua; the remaining installs get a "postponed" email and their banner
|
||
is removed.
|
||
|
||
**Rules carried over:** the release's requirements (`— requires accounts 1.2.0`, decision 24)
|
||
are checked in the Hub before scheduling and again by the script; each product rolls back on its
|
||
own (§8 of the roadmap). A church's **Request update** now shows as "Requested" next to
|
||
Schedule update. Moving a 2.1.x install to 2.2.0 is an `adopt` job, scheduled the same way.
|
||
|
||
## 9. Uninstall
|
||
|
||
Hub → Install → **Uninstall** (2FA again, type the short name):
|
||
|
||
1. Joshua picks the date (at least 4 hours away, default 7 days) and the site admin is emailed
|
||
and gets a banner, as for an update.
|
||
2. **Export first** (`crosscode export`): the church's data as **JSON in a format of our own**,
|
||
`crosscode.church-export/1` (defined in `packages/contracts`): events, sessions, ticket types,
|
||
registrations and attendees, people (name, email, phone, consent), payments (amounts, dates,
|
||
references), check-ins, rosters, settings, and the uploads as files next to it. Written in
|
||
plain terms with our own field names; **no database tables, Prisma models or internal ids**.
|
||
One ZIP: `export.json`, `uploads/`, `README.txt` explaining the fields.
|
||
3. The agent **encrypts the ZIP with a random key** and uploads it to the Hub. The key is only in
|
||
the link emailed to the site admin (`…/download/<id>#<key>`; the part after `#` is never sent
|
||
to the Hub), so the Hub stores the file but can't read it, and the roadmap's "the Hub only sees
|
||
aggregate stats" still holds. The download page decrypts it in the browser. Available for
|
||
**30 days**.
|
||
4. **Uninstall:** final database backup and an archive of the uploads to
|
||
`/var/www/sites/.removed/<church>-<date>/` (root only) → PM2 processes removed, `pm2 save` →
|
||
the frpc block removed and reloaded → database and role dropped → the site folder removed →
|
||
the licence released in the Hub.
|
||
5. **Emails:** to the site admin, the download link and the date it expires; to Joshua, the report
|
||
and "remove the FastPanel site `events.hope.co.za` (ports 20010, 20011)".
|
||
6. **After 30 days:** a `purge` job deletes the kept backup and archive, and the Hub deletes the
|
||
encrypted export. Joshua is emailed **3 days before**, and can extend it once by 30 days. The
|
||
Hub keeps a record that it was deleted, and when (for POPIA).
|
||
|
||
## 10. Servers page and alerts
|
||
|
||
**Hub → Servers:** one card per server (church servers and the central one).
|
||
|
||
- **Every 60 s** the agent sends: RAM (total, used, available, swap), CPU (% and 1/5/15-minute
|
||
load), disk per mount (used, free), uptime, versions (OS, Node, PostgreSQL, frpc, agent), frpc
|
||
running, and **per PM2 process** (church, product, memory, CPU, restarts, uptime).
|
||
- **Kept:** per minute for 7 days, per hour (average and peak) for 90 days.
|
||
- **Server page:** RAM and CPU charts for 24 h / 7 days / 90 days with **spikes marked** (peaks
|
||
above 85%), disk bars, the sites on it with each process's memory, recent jobs, the frpc blocks
|
||
it manages and the ports they use.
|
||
- **Alerts** (email to Joshua, through the existing alerts, once per incident and again when it
|
||
clears): agent not heard from for 5 minutes; disk over 85%; RAM over 90% for 5 minutes; a PM2
|
||
process restarting more than 5 times in 10 minutes; frpc not running; on the central server,
|
||
`proxy_bind_addr` missing or a managed tunnel not listening.
|
||
|
||
## 11. Reports and emails
|
||
|
||
All in the Hub (Jobs page, with the full report) and by email:
|
||
|
||
| Email | To | When |
|
||
|-------|----|------|
|
||
| Install report | Joshua | Every install, success or failure: each step with its time, ports, the central check, the log tail on failure, what was cleaned up |
|
||
| **FastPanel setup** | Joshua | After a successful install. What *he* does in FastPanel; the agent does none of it (below) |
|
||
| Welcome | Site admin | When the install is Live |
|
||
| Update planned / reminder / done / didn't go through | Site admin | §8 |
|
||
| Update report | Joshua | Every update (and batch summary) |
|
||
| Missed job | Joshua | A scheduled job wasn't picked up in time |
|
||
| Uninstall planned / export ready | Site admin | §9 |
|
||
| Uninstall report, purge in 3 days | Joshua | §9 |
|
||
| Alerts | Joshua | §10 |
|
||
|
||
**The FastPanel email** (an instruction to Joshua, nothing is set up automatically):
|
||
|
||
```
|
||
Set up in FastPanel: events.hope.co.za (Hope Family Church, Cross Events)
|
||
|
||
1. Add the site events.hope.co.za.
|
||
2. Proxy:
|
||
/api, /uploads, /health, /docs → http://127.0.0.1:20011 (backend)
|
||
everything else → http://127.0.0.1:20010 (frontend)
|
||
Pass the Host header and X-Forwarded-For / X-Forwarded-Proto: https.
|
||
Allow uploads up to 25 MB.
|
||
3. Turn on Let's Encrypt for the site.
|
||
4. DNS: events.hope.co.za → <central server IP> (checked: ✅ points there / ⚠ not yet)
|
||
5. Then click "Proxy done" in the Hub: <link>
|
||
|
||
Central check: both ports are open in frps and only on 127.0.0.1.
|
||
```
|
||
|
||
## 12. What changes in each repo
|
||
|
||
**`crosscode-platform`**
|
||
- `apps/agent` (new, `agent@1.0.0`) with its install script and systemd unit.
|
||
- `scripts/` (the `crosscode` script): `--json` reports on every command, `uninstall`, `export`,
|
||
`purge`, `restart`, the frpc block handling (lib/frpc.sh), the env template handling
|
||
(lib/env.sh), ports given on the command line instead of picked.
|
||
- `apps/events`: `env.template.json`, and the export (`crosscode export` calls a backend script
|
||
that writes `crosscode.church-export/1`).
|
||
- `packages/contracts` **1.2.0**: `crosscode.agent-job/1`, `crosscode.agent-report/1`,
|
||
`crosscode.agent-stats/1`, `crosscode.env-template/1`, `crosscode.church-export/1`.
|
||
|
||
**`events-hub` (Hub 2.1.0)**
|
||
- Models: `Server` (name, mode, public keys, encrypted config, last seen, versions), `ServerStat`
|
||
(minute and hourly), `Job` (type, args, status, schedule, who, report), `ScheduledUpdate` /
|
||
batch, `Port` (number, server, install, released at), `Export` (encrypted blob, expires).
|
||
`Install` gets `serverId`, `shortName`, `status` (installing, waiting-for-proxy, live,
|
||
uninstalling, removed).
|
||
- Agent API `/api/agent/v1/*` (join, poll, stats, report, export upload), signed by the agent.
|
||
- Pages: Servers, Server, New install, Jobs, Updates (schedule, batch), Install defaults, the
|
||
public export download page.
|
||
- Emails in §11; alerts in §10; the job-signing key in `.env`.
|
||
|
||
**`crosscode-site`:** nothing (this changes how Joshua works, not what churches buy).
|
||
|
||
## 13. Testing
|
||
|
||
- **Agent:** unit tests for job checking (bad signature, wrong server, expired, replayed,
|
||
unknown type, an argument with shell characters), and **a test that the running agent has no
|
||
listening socket** (`ss -ltnp` shows nothing for its process).
|
||
- **Script:** bats tests for the frpc blocks (append to a file with hand-written entries and
|
||
comments, remove one block, the other blocks and the hand-written lines unchanged byte for byte;
|
||
a failed `frpc verify` leaves the file untouched; `[common]` admin lines added only once), the
|
||
env template (new variable appended, existing lines untouched, required one missing → refused),
|
||
and the export (a demo church's export validates against `crosscode.church-export/1` and has no
|
||
internal ids).
|
||
- **On VMs** (multipass): a "central" VM with frps (with `proxy_bind_addr = 127.0.0.1`) and nginx
|
||
in place of FastPanel, two church VMs with frpc and a hand-written tunnel, the Hub running
|
||
locally:
|
||
1. Join all three. Stats show up.
|
||
2. New install on church VM 1 → report, FastPanel email, ports open on the central VM, the
|
||
hand-written tunnel still works. Add the nginx site by hand, Proxy done → Live.
|
||
3. A second church on VM 2 → different ports, unique tunnel names.
|
||
4. Schedule an update 4 h ahead (time moved forward in the test) → emails, banner, update,
|
||
report. Try 3 h → refused. Emergency with 15 min → needs 2FA.
|
||
5. A broken release → rolled back, "didn't go through" email, batch stopped.
|
||
6. Hub stopped during a scheduled update → missed, nothing runs, sites keep running.
|
||
7. Uninstall → export downloads and decrypts in the browser, everything removed, the other
|
||
church's tunnel and the hand-written one untouched. Purge after 30 days (clock moved).
|
||
8. A forged job (wrong key) and a replayed job → refused and reported.
|
||
|
||
## 14. Rolling it out
|
||
|
||
1. Contracts 1.2.0 → the `crosscode` script changes (with Events 2.2.0 out first) → agent 1.0.0
|
||
→ Hub 2.1.0, each released as usual (branch → merge → tag → push).
|
||
2. On the central server: add `proxy_bind_addr = 127.0.0.1` to `frps.ini` if it's missing (check
|
||
that each FastPanel site still answers), then join the central agent.
|
||
3. Join each church server; set its server config in the Hub. Existing installs are linked to
|
||
their server and their current ports recorded in the port list (read from `church.env` and
|
||
`frpc.ini`; nothing is changed).
|
||
4. First real use: the next new church. Existing churches get their next update through a
|
||
scheduled update.
|
||
5. Docs: CHANGELOGs, HANDOVER.md (1d done), `events-hub/deployment.md` (the job-signing key),
|
||
a server how-to in `apps/agent/README.md`.
|
||
|
||
## Decisions
|
||
|
||
Agreed with Joshua on 2026-11-12 (ROADMAP-v3 decisions 27–37):
|
||
|
||
1. The Hub can install, update and uninstall through an agent on each server, with **signed jobs
|
||
of fixed types**; the agent only runs the `crosscode` script; uninstall and emergency updates
|
||
need 2FA again.
|
||
2. **The agent has no URL or listening port**; it only polls the Hub.
|
||
3. **Scheduled updates at least 4 hours ahead, emergency updates at least 15 minutes**, with an
|
||
email and banner to the site admin; Joshua picks the time.
|
||
4. **Uninstall:** the church gets a **JSON export in our own format** (no internal data
|
||
structures), encrypted so the Hub can't read it, downloadable for 30 days; backups kept 30
|
||
days, then deleted.
|
||
5. All church servers are behind frp to **one central FastPanel server**. The Hub keeps the port
|
||
list; **frpc.ini is only appended to**; the `[common]` admin lines are added once.
|
||
6. **FastPanel and DNS are never automated**; Joshua gets an email with what to set up.
|
||
7. **Database and other server settings are set once per server in the Hub** and sent encrypted
|
||
with each job; `.env` files are built from a per-product template and only appended to on
|
||
updates; generated secrets stay on the server.
|
||
8. A read-only agent on the central server checks ports and `proxy_bind_addr`.
|
||
|
||
## Still open
|
||
|
||
Nothing.
|