Files
events-hub/deployment.md
joshuaandClaude Opus 5.5 5a90c713d8 Deploy script with rollback (1.1.2)
scripts/deploy.sh backs up the database (pg_dump), checks out the release,
runs npm ci, prisma migrate deploy and generate, builds the frontend,
restarts the PM2 apps and health-checks them. Any failure puts back the
previous code and build, and restores the dump if new migrations had run.
deployment.md §12 now uses it. tsconfig.tsbuildinfo leaves git so builds
don't leave the tree dirty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 23:55:46 +02:00

12 KiB

Deploying the Hub

How to run the Hub on the server at events-hub.crosscode.co.za, behind FASTPANEL's nginx and an frp tunnel.

Internet ──► frps (public VPS) ══ tunnel ══► frpc (this server, /etc/frp/frpc.ini)
                                                │
                                                ▼
                                    nginx (FASTPANEL, :80 / :443, HTTPS)
                                       ├── /api/*  ──► backend  (Express, 127.0.0.1:5200)
                                       └── /*      ──► frontend (Next.js, 127.0.0.1:3200)
                                                         │
                                               PostgreSQL (localhost:5432)

Only nginx is reachable from outside. Ports 5200 and 3200 must not be added to frpc.

Replace <siteuser> below with the system user FASTPANEL creates for the site.


1. Prerequisites (once per server)

  • Node 20+ (node -v), and PM2: sudo npm install -g pm2
  • PostgreSQL 14+
  • Git access to this repo

If Cross Events is already deployed on this server, you already have all of these.


2. DNS

Add an A record for events-hub.crosscode.co.za pointing at the public IP of the frps VPS, not this server's IP. Check it with dig +short events-hub.crosscode.co.za.


3. frp

Open /etc/frp/frpc.ini. How much you need to do depends on how it already forwards web traffic.

a) Plain TCP forwarding of 80 and 443 — you'll see something like:

[web-http]
type = tcp
local_ip = 127.0.0.1
local_port = 80
remote_port = 80

[web-https]
type = tcp
local_ip = 127.0.0.1
local_port = 443
remote_port = 443

Then you don't need to change anything: nginx picks the site by domain name. Go to step 4.

b) Per-domain http / https proxies (custom_domains = ..., with vhost_http_port / vhost_https_port set on frps) — add a pair for the Hub:

[events-hub-http]
type = http
local_ip = 127.0.0.1
local_port = 80
custom_domains = events-hub.crosscode.co.za

[events-hub-https]
type = https
local_ip = 127.0.0.1
local_port = 443
custom_domains = events-hub.crosscode.co.za

Use https (passthrough), not https2http: TLS is ended by nginx on this server with the FASTPANEL certificate.

Check the file and restart:

sudo frpc verify -c /etc/frp/frpc.ini
sudo systemctl restart frpc
sudo journalctl -u frpc -n 30 --no-pager     # look for "start proxy success"

Port 80 must reach nginx through the tunnel, or Let's Encrypt can't issue the certificate.


4. Database

Create a database and user. In FASTPANEL: Databases → Create, choose PostgreSQL, name events_hub. Or from the shell:

sudo -u postgres psql <<'SQL'
CREATE USER events_hub WITH PASSWORD 'a-long-random-password';
CREATE DATABASE events_hub OWNER events_hub;
SQL

Your connection string is then postgresql://events_hub:a-long-random-password@localhost:5432/events_hub. (URL-encode the password if it contains @ : / ? #.)


5. Site in FASTPANEL

  1. Sites → Create site, domain events-hub.crosscode.co.za. Note the site's system user (<siteuser>) and folder, e.g. /var/www/<siteuser>/data/www/events-hub.crosscode.co.za.
  2. SSL certificates → Let's Encrypt for the domain. Turn on the redirect from HTTP to HTTPS.
  3. Leave the nginx config for step 8; the apps need to be running first.

6. Get the code and configure it

Work as the site user so files end up with the right owner:

sudo -iu <siteuser>
cd /var/www/<siteuser>/data/www/events-hub.crosscode.co.za
git clone <repo-url> app
cd app

Backend

cd backend
cp .env.example .env
nano .env

Set at least:

Variable Value
DATABASE_URL the connection string from step 4
FRONTEND_URL https://events-hub.crosscode.co.za
JWT_SECRET openssl rand -base64 48
HUB_ENCRYPTION_KEY openssl rand -base64 48. Never change this once installs have registered
HUB_SIGNING_KEY_FILE ./keys/hub-signing-key.pem (the default)
TRUST_PROXY 1
SMTP_*, ALERT_EMAIL your mail settings, so alerts get sent
chmod 600 .env
npm ci
npx prisma migrate deploy && npx prisma generate
npm run generate-keys          # only on the very first deploy; it refuses to overwrite
npm run create-admin -- --email you@example.com --name "Your Name"

generate-keys prints the public key. Put it into Cross Events (backend/src/hub/publicKey.js) and release. Copy keys/hub-signing-key.pem somewhere safe and offline now (see step 11).

Frontend

NEXT_PUBLIC_API_URL is built into the frontend at build time, so set it before building:

cd ../frontend
echo "NEXT_PUBLIC_API_URL=https://events-hub.crosscode.co.za" > .env.local
npm ci
npm run build

Don't add /api to that URL: the frontend already calls paths like /api/auth/login.


7. Run with PM2

Still as <siteuser>, from the app folder:

pm2 start npm --name events-hub-backend  --cwd backend  -- start
pm2 start npm --name events-hub-frontend --cwd frontend -- start
pm2 save

The names are different from Cross Events' backend / frontend processes, so the two don't clash on the same server.

Start on boot (run once; it prints a sudo command to paste):

pm2 startup systemd

Check:

pm2 status
pm2 logs events-hub-backend --lines 20     # expect "[hub] Listening on 5200"
curl -s http://127.0.0.1:5200/api/health   # {"ok":true}
curl -sI http://127.0.0.1:3200 | head -1   # HTTP/1.1 200 OK

The backend refuses to start (and says why in the logs) if DATABASE_URL, JWT_SECRET, HUB_ENCRYPTION_KEY or the signing key is missing.

Run one backend process only (no -i max / cluster mode): the background alert jobs run in-process and would fire once per copy.


8. nginx (FASTPANEL)

In FASTPANEL, open the site → Configuration files (the nginx config of the site). Inside the server { ... } block that listens on 443, replace the default location / (the one serving static files / PHP) with:

location /api/ {
    proxy_pass         http://127.0.0.1:5200;
    proxy_http_version 1.1;
    proxy_set_header   Host              $host;
    proxy_set_header   X-Real-IP         $remote_addr;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_read_timeout 60s;
}

location / {
    proxy_pass         http://127.0.0.1:3200;
    proxy_http_version 1.1;
    proxy_set_header   Host              $host;
    proxy_set_header   X-Real-IP         $remote_addr;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
}

Leave the listen, server_name, ssl_certificate and /.well-known/acme-challenge/ parts that FASTPANEL wrote alone, so certificate renewal keeps working. If FASTPANEL also generated an Apache backend or PHP-FPM for this site, turn it off in the site settings. The Hub doesn't use it.

Save (FASTPANEL tests the config and reloads nginx). From the shell, you can check with:

sudo nginx -t && sudo systemctl reload nginx

9. Check it from outside

curl -s https://events-hub.crosscode.co.za/api/health     # {"ok":true}

Then open https://events-hub.crosscode.co.za, log in, and set up two-factor (required on first login). In Settings, check that the public key shown matches the one in Cross Events.

Point Cross Events installs at the Hub with HUB_URL=https://events-hub.crosscode.co.za in their backend/.env.


With frp in front, nginx sees every connection as coming from frpc (127.0.0.1). So the backend thinks all visitors share one IP. That has two effects:

  • The login rate limit (20 tries / 15 min) counts everyone together. Someone hammering the login page also locks you out.
  • Audit log entries all show 127.0.0.1.

(The heartbeat limit is 30/min per IP, and installs call every 15 minutes, so heartbeats are fine up to a few hundred installs, even sharing one IP.)

You can fix this with the PROXY protocol, which carries the real client IP through the tunnel. This applies to every site on this nginx: once a listen has proxy_protocol, all server blocks on that port must have it, and nothing can reach nginx on that port except through frp. Only do it if all your sites come in through frp.

  1. In each web proxy section of /etc/frp/frpc.ini (80 and 443), add:

    proxy_protocol_version = v2
    
  2. In nginx, add proxy_protocol to every listen on those ports. With FASTPANEL, that means every site's config, e.g. listen 443 ssl proxy_protocol;. Then add this once, in the http {} block (e.g. a file in /etc/nginx/conf.d/realip.conf):

    set_real_ip_from 127.0.0.1;
    real_ip_header   proxy_protocol;
    
  3. sudo nginx -t && sudo systemctl reload nginx && sudo systemctl restart frpc

After that, $remote_addr is the real visitor, and with TRUST_PROXY=1 the backend sees it too. If you'd rather not touch the other sites, skip this. The Hub works without it.


11. Backups

Two things must survive the server: the database and the signing key (plus .env, because HUB_ENCRYPTION_KEY can't be recreated).

  • Signing key and .env: copy backend/keys/hub-signing-key.pem and backend/.env to an offline place (password manager, encrypted USB) once. They don't change.

  • Database, daily: if FASTPANEL's backup feature covers PostgreSQL for this site, turn it on. Otherwise, as <siteuser>, crontab -e:

    30 2 * * * pg_dump "postgresql://events_hub:PASSWORD@localhost:5432/events_hub" -Fc -f "$HOME/backups/events_hub-$(date +\%a).dump"
    

    (mkdir -p ~/backups first). That keeps one dump per weekday. Copy them off the server regularly.

To restore a dump: pg_restore --clean --no-owner -d "<DATABASE_URL>" events_hub-Mon.dump


12. Updating

sudo -iu <siteuser>
cd /var/www/<siteuser>/data/www/events-hub.crosscode.co.za/app
bash scripts/deploy.sh            # the latest main
bash scripts/deploy.sh v1.2.0     # or a tag / commit

The script:

  1. Refuses to start if tracked files have local changes (it would overwrite them).
  2. Backs up the database with pg_dump to ~/backups/events-hub/ (the newest 10 are kept).
  3. Checks out the release (main as a detached origin/main).
  4. Backend: npm ci, prisma migrate deploy, prisma generate.
  5. Frontend: npm ci, npm run build.
  6. pm2 restart events-hub-backend events-hub-frontend, then waits up to 30 seconds each for /api/health on the backend and the frontend's home page to answer.

If any step fails, it goes back to the version that was running: the old code, dependencies and build, a PM2 restart if it had already restarted, and a restore of the database backup if the release brought new migrations. A restore drops anything written since the backup (usually a few heartbeats in those minutes). If the restore itself fails, the script prints the pg_restore command to run by hand.

Different backup folder or PM2 names? Set HUB_BACKUP_DIR, HUB_PM2_BACKEND, HUB_PM2_FRONTEND when running it. It needs pg_dump/pg_restore and curl on the server.

The first time only, the script isn't on the server yet, and the clone is on the main branch. Get it with:

git checkout -- frontend/tsconfig.tsbuildinfo   # a build file no longer kept in git
git pull
bash scripts/deploy.sh

From then on the checkout is detached, so use the script rather than git pull.

Installs keep working while the Hub is briefly down (licences are valid offline for 30 days).


Troubleshooting

Symptom Look at
Site doesn't load at all systemctl status frpc, journalctl -u frpc; DNS points at the frps VPS; frps allows ports 80/443
Let's Encrypt fails Port 80 isn't reaching nginx through frp (step 3)
502 Bad Gateway pm2 status, pm2 logs events-hub-backend / events-hub-frontend
Login page loads, but login fails with a network/CORS error FRONTEND_URL in backend/.env (exact https:// origin, no trailing slash); NEXT_PUBLIC_API_URL in frontend/.env.local, then rebuild the frontend
"Too many attempts" for everyone Shared IP through frp, see step 10
Installs fail signature checks nginx must pass the body through unchanged. Don't add body-rewriting modules to /api/
Backend exits on start pm2 logs events-hub-backend: it names the missing variable or key