Production deployment

This guide takes a fresh VPS and deploys Pia Server behind Caddy with automatic Let’s Encrypt TLS, from images pulled out of GitHub Container Registry.

Prerequisites: A server with Docker and Docker Compose installed, and a domain whose DNS you control. Throughout, <your-domain> is the host serving the API and <server-ip> is its public address.

1. DNS records

Point the domain at your server’s public IP.

Type	Name	Value
A	<your-domain>	<server-ip>

docker-compose.prod.yml also ships the marketing site and this documentation as their own services. Add an A record for each hostname you intend to serve, or drop the services you do not want from the compose file.

Caddy needs DNS to resolve before it can issue certificates. Verify propagation:

dig <your-domain> +short

It should return your server IP.

2. Install Docker on the server

ssh root@<server-ip>
curl -fsSL https://get.docker.com | sh
docker --version
docker compose version

3. Mount the persistent volume

If your provider offers block storage — a Hetzner Cloud Volume, for instance — attach it, then mount it on the server:

# Find the device
ls /dev/disk/by-id/

# Format (first use only — destroys data!)
mkfs.ext4 /dev/disk/by-id/scsi-0HC_Volume_<id>

# Mount
mkdir -p /mnt/hc-volume
mount /dev/disk/by-id/scsi-0HC_Volume_<id> /mnt/hc-volume

# Persist across reboots
echo '/dev/disk/by-id/scsi-0HC_Volume_<id> /mnt/hc-volume ext4 defaults 0 2' >> /etc/fstab

If your mount path differs from /mnt/hc-volume, update docker-compose.prod.yml after copying it to the server (step 6).

`postgres` does not use `/mnt/hc-volume`

One mount in docker-compose.prod.yml is not the generic path: postgres writes to a host-specific volume path rather than to /mnt/hc-volume. Where that exact path does not exist, Docker creates the directory on the boot disk rather than failing, and Postgres initializes an empty cluster inside it — the stack comes up green with no data on the volume you just mounted. Point that line at your own mount path before the first up.

4. Authenticate Docker with ghcr.io

The images are served from GitHub Container Registry, and the packages are private — the server needs credentials before it can pull. Request registry access from kontakt@pia-ai.de alongside your licence, then on the server:

echo "<your-token>" | docker login ghcr.io -u <your-username> --password-stdin

You should see Login Succeeded.

5. Production environment files

Configuration lives in two files. .env.prod is mounted into postgres and umami; .env.prod.server is mounted into pia-server and the short-lived pia-migrate. The division is deliberate: .env.prod carries POSTGRES_PASSWORD, the owner role’s password, and handing that to the application container would let anything running in the process read the owner credential straight out of its own environment — which is the whole of what the role split exists to prevent. pia-migrate is the one container that receives it, composed at the Compose layer rather than stored in either file, and it exits before pia-server starts.

The infrastructure side:

mkdir -p /opt/pia
cat > /opt/pia/.env.prod << 'EOF'
POSTGRES_PASSWORD=<db-owner-password>
TEMPORAL_DB_PASSWORD=<temporal-db-password>
VECTORDB_PASSWORD=<vectordb-password>
APP_SECRET=<random-64-hex-chars>
EOF

The application side:

cat > /opt/pia/.env.prod.server << 'EOF'
ASPNETCORE_ENVIRONMENT=Production
Database__Provider=postgresql
Database__ConnectionString=Host=postgres;Database=pia;Username=pia;Password=<db-owner-password>
Jwt__SecretKey=<random-string-at-least-32-chars>
Encryption__MasterKey=<random-64-hex-chars>
Knowledge__ConnectionString=Host=pia-vectordb;Database=pia_knowledge;Username=pia;Password=<vectordb-password>
EOF
chmod 600 /opt/pia/.env.prod /opt/pia/.env.prod.server

Generate secure values:

openssl rand -base64 32   # Jwt__SecretKey
openssl rand -hex 32      # Encryption__MasterKey and APP_SECRET (64 hex chars exactly, one each)
openssl rand -base64 24   # POSTGRES_PASSWORD, then again for TEMPORAL_DB_PASSWORD and VECTORDB_PASSWORD

Leave Knowledge__ConnectionString empty to run without the knowledge base entirely. Everything else the server reads — AI providers, embeddings and chunking, OAuth, admin emails — is optional and documented in .env.prod.server.example, with the infrastructure keys in .env.prod.example. Both example files ship with the server source rather than living on the host: a deploy copies only the four files that hold no secrets — docker-compose.prod.yml, Caddyfile, init-db.sh and temporal-dynamicconfig.yaml.

TEMPORAL_DB_PASSWORD is the one key in .env.prod with no fallback in the compose file. Unset, temporal-db-init creates the Temporal role with an empty password and the Temporal server never connects — while pia-server starts fine regardless, because it deliberately does not depend on Temporal.

Match the passwords across the two files

POSTGRES_PASSWORD in .env.prod is the pia role’s password, and Database__ConnectionString in .env.prod.server connects as pia — so the two must be identical. A mismatch means the server boots but can’t connect to Postgres. Once you adopt the database role split, that string carries pia_app’s password instead and the two deliberately differ.

VECTORDB_PASSWORD belongs to a different cluster and must instead match Knowledge__ConnectionString. Keep it different from POSTGRES_PASSWORD: unset, pia-vectordb falls back to POSTGRES_PASSWORD, which puts the main cluster’s owner password back inside the app container by another route. APP_SECRET is likewise optional but cheap now and expensive later — unset, umami derives its signing secret from DATABASE_URL, so moving it to its own database role logs out every session.

Back up both files

Store copies somewhere safe (a password manager works). .env.prod.server is gitignored exactly as .env.prod is, so neither exists anywhere but the host. Encryption__MasterKey is the critical one — if you lose it, encrypted user data is unrecoverable.

Existing hosts: split the file you already have

A host set up before this layout has one /opt/pia/.env.prod holding both sides. pia-server no longer reads it: docker compose up -d fails with env file /opt/pia/.env.prod.server not found, and pia-server does not start until that file exists.

1. Copy the application keys — everything except POSTGRES_PASSWORD, VECTORDB_PASSWORD, APP_SECRET, UMAMI_DB_USER and UMAMI_DB_PASSWORD — into a new /opt/pia/.env.prod.server, and chmod 600 it.

2. Deploy, which copies the current docker-compose.prod.yml onto the host. pia-server reads .env.prod.server from here on.

3. Delete those keys from .env.prod, leaving the infrastructure ones behind. postgres and umami read nothing else from that file, so nothing needs recreating.

Keep that order: trimming .env.prod before the new compose file lands points the still-running configuration at a file that no longer holds it.

Optional: the Mesh operator runtime

docker-compose.prod.yml already defines a temporal service, but the operator runtime stays off until you say otherwise. To enable it, add to .env.prod.server:

Operators__Enabled=true
Temporal__Address=temporal:7233
Temporal__Namespace=pia
Temporal__TaskQueue=pia-operators

Every other Operators__* key has a working default — see Configuration. All four keys above are restart-only.

Three deployment facts worth knowing before you flip it on:

• pia-server deliberately does not depends_on temporal. An unreachable Temporal degrades to a logged warning and an idle worker; coupling the two would turn a Temporal failure into a pia-server restart loop and a chat outage, for a feature that defaults off.
• Temporal has no published ports and no Caddy route. The default single-namespace dev-server setup has no authentication of its own. Keep it that way.
• Its state lives in Postgres, in the temporal and temporal_visibility databases that temporal-db-init creates and temporal-schema-init populates — two one-shots that gate the server between them. Losing that state costs in-flight assignments only; assignments and assignment_events in the pia database are the record of truth.

6. Deploy

Copy the four files that carry no secrets onto the host, then start the stack. Images come from the registry you authenticated against in step 4 — nothing is built on the server.

Edit Caddyfile before you copy it: its site blocks carry the hostnames of the deployment they were written for, and Caddy requests a certificate for whatever name it finds in a block. Replace them with your own, and delete the blocks for services you are not running.

# On your local machine, from the server source
scp docker-compose.prod.yml Caddyfile init-db.sh temporal-dynamicconfig.yaml root@<server-ip>:/opt/pia/

# On the server
ssh root@<server-ip>
cd /opt/pia
docker compose --env-file .env.prod -f docker-compose.prod.yml pull
docker compose --env-file .env.prod -f docker-compose.prod.yml up -d

Every later upgrade is those same two commands — see Upgrading. Re-copy the four files whenever a release changes them.

`--env-file .env.prod` is not optional

Seven values in docker-compose.prod.yml are Compose-level substitutions: the migration connection string, the three Temporal passwords, umami’s DATABASE_URL and pia-vectordb’s password. Compose reads those from --env-file, never from a service’s own env_file:, so without the flag it warns once and substitutes empty strings — pia-migrate then authenticates with no password, fails, and pia-server never starts behind its service_completed_successfully gate. Every pull and up needs the flag. ps, logs and exec do not, and neither does the run --rm pia-server used for pod provisioning — though passing it there too silences seven unset-variable warnings at the moment a token is printing to stdout.

Copy all four files, not just the compose file

Caddyfile, init-db.sh and temporal-dynamicconfig.yaml are bind-mounted as files. Where one is missing, Docker creates a directory at the mount point instead of failing: without temporal-dynamicconfig.yaml the Temporal server exits on unable to create dynamic config before it opens a port, and without init-db.sh umami’s database is never created — which copying the file in later does not fix, since /docker-entrypoint-initdb.d runs only against an empty data directory.

7. Database roles and row-level security

Optional, and inert until the last cutover step. It splits the single pia credential into five database roles and puts row-level security on assignments and assignment_events, enforced by which login role the connection used rather than by a session flag the application sets.

Treat it as a second layer on top of route scoping and the store seam, not as a database guarantee of tenant isolation. It stops a missing WHERE UserId = … on SELECT, UPDATE and INSERT against those two tables — with no identity set count(*) returns 0 with no error, a cross-user UPDATE reports UPDATE 0, and a cross-user INSERT fails with 42501. It does not stop a missing WHERE on DELETE: referential-integrity cascades are exempt from row-level security and users has 30 cascade children, so a role that sees zero rows in both protected tables still destroys every tenant’s assignment data with DELETE FROM users. It covers 2 of the 47 public tables — the runtime role keeps unrestricted access to users, user_roles, refresh_tokens and DataProtectionKeys — and it does nothing against host root or anyone who can docker exec. Its other real gain is the umami analytics container, which otherwise holds a superuser credential on the cluster that stores assignment plaintext.

FORCE ROW LEVEL SECURITY is on both tables. Its one benefit: ownership stops being an implicit bypass, so pg_policies is the complete answer to who can read these tables.

The five roles

Role	Attributes	Owns	Credential lives in
pia_root	SUPERUSER LOGIN, no password, ever	nothing	nowhere — reachable only as docker compose exec postgres psql -U pia_root
pia	LOGIN NOSUPERUSER NOBYPASSRLS NOREPLICATION CREATEDB CREATEROLE	database pia and all 47 public tables	nowhere persistent — composed into pia-migrate’s Database__MigrationConnectionString from POSTGRES_PASSWORD in .env.prod
pia_app	LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS	nothing	Database__ConnectionString
pia_maint	same as pia_app	nothing	Database__MaintenanceConnectionString
umami_app	same as pia_app, plus INHERIT	database umami and every object in it	umami’s DATABASE_URL

pia runs DDL and migrations — from the one-shot pia-migrate container only, which Compose runs to completion before pia-server starts and which then exits. That is why the ceiling above stops where it does: ownership is what row-level security cannot fence off, so the long-running process must not hold it. pia_app is the runtime role every request uses; the user’s identity reaches the database as a pia.user_id session setting written on every connection open. pia_maint is the bypass used by the admin roll-up and the retention sweep — not a session flag, a different login. pia_root exists only so that de-privileging pia is reversible, because pia is currently the sole superuser on the cluster.

One-time provisioning

Run by a human, once. pia_root is created without a password: a role with no SCRAM verifier cannot authenticate over TCP, and the container’s pg_hba.conf trusts local socket connections, so docker compose exec is the only way in.

cd /opt/pia
docker compose -f docker-compose.prod.yml exec postgres psql -U pia -d postgres -c \
  "CREATE ROLE pia_root LOGIN SUPERUSER;"

# Must print pia_root | t
docker compose -f docker-compose.prod.yml exec postgres psql -U pia_root -d postgres -c \
  "SELECT rolname, rolsuper FROM pg_roles WHERE rolname = 'pia_root';"

# Over TCP instead of the socket: must be refused with "password authentication failed"
docker compose -f docker-compose.prod.yml exec postgres \
  psql -h postgres -U pia_root -d postgres -c "SELECT 1"

Generate three passwords with openssl rand -base64 24 and create the roles as pia_root. Roles created by a superuser produce no pg_auth_members rows at all; roles created by pia would leave pia holding admin_option on them, which is exactly the membership the runtime role’s isolation depends on being absent.

-- docker compose -f docker-compose.prod.yml exec postgres psql -U pia_root -d postgres
SET createrole_self_grant = '';
CREATE ROLE pia_app   LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS NOINHERIT
  PASSWORD '<generated-password>';
CREATE ROLE pia_maint LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS NOINHERIT
  PASSWORD '<generated-password>';
CREATE ROLE umami_app LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS INHERIT
  PASSWORD '<generated-password>';

-- Must return (0 rows): neither granted to anyone, nor holding any membership themselves.
SELECT roleid::regrole, member::regrole, admin_option, set_option
  FROM pg_auth_members
 WHERE roleid::regrole::text IN ('pia_app', 'pia_maint', 'umami_app')
    OR member::regrole::text IN ('pia_app', 'pia_maint', 'umami_app');

Connection strings

Three keys in /opt/pia/.env.prod.server, alongside the other server settings from step 5.

Key	Role	Used for
Database__ConnectionString	pia_app	every request
Database__MaintenanceConnectionString	pia_maint	admin roll-up, retention sweep

Move the runtime string in the same edit

Leave Database__MaintenanceConnectionString unset and the server runs everything on Database__ConnectionString exactly as before, with the policies inert. Set it, and Database__ConnectionString must move to pia_app at the same time — a runtime role that still owns the tables is refused at startup with PIA-RLS-009, because ownership defeats the policies whatever they say.

`Database__MigrationConnectionString` does not belong in `.env.prod.server`

It is the owner credential, and the application process is denied it on purpose. docker-compose.prod.yml composes it from POSTGRES_PASSWORD and passes it to the one-shot pia-migrate container instead. Set it here and the server applies migrations in-process again — it logs a warning saying so at boot, and the ceiling above widens back to TRUNCATE, DROP POLICY and NO FORCE.

Giving umami its own role

umami must stop using the pia superuser credential before pia is de-privileged, or the container comes back up against a role that can no longer run its Prisma migrations.

Never run `REASSIGN OWNED BY pia TO umami_app`

REASSIGN OWNED is cross-database — databases are shared objects, so it reaches all of them regardless of which one you are connected to. On this host it would hand the pia, postgres and template databases to the analytics role. Use the guarded block below instead: it moves only the objects inside umami, and refuses to run anywhere else.

As pia_root, connected to umami:

-- docker compose -f docker-compose.prod.yml exec postgres psql -U pia_root -d umami
\set ON_ERROR_STOP on
BEGIN;
DO $do$
DECLARE r record; tgt text := 'umami_app'; n int := 0;
BEGIN
  IF current_database() <> 'umami' THEN
    RAISE EXCEPTION 'refusing to run outside database umami (current=%)', current_database();
  END IF;

  FOR r IN
    SELECT c.oid::regclass AS obj
      FROM pg_class c JOIN pg_namespace n2 ON n2.oid = c.relnamespace
     WHERE n2.nspname NOT IN ('pg_catalog','information_schema','pg_toast')
       AND c.relkind IN ('r','p','f','v','m','S')
       AND pg_get_userbyid(c.relowner) <> tgt
       -- Identity and serial sequences follow their table; altering one directly aborts the transaction.
       AND NOT (c.relkind = 'S' AND EXISTS (
                 SELECT 1 FROM pg_depend d
                  WHERE d.classid = 'pg_class'::regclass AND d.objid = c.oid
                    AND d.refclassid = 'pg_class'::regclass AND d.deptype IN ('a','i')))
     ORDER BY CASE c.relkind WHEN 'S' THEN 2 WHEN 'v' THEN 3 WHEN 'm' THEN 3 ELSE 1 END, c.oid
  LOOP EXECUTE format('ALTER TABLE %s OWNER TO %I', r.obj, tgt); n := n + 1; END LOOP;

  FOR r IN SELECT p.oid::regprocedure AS fn FROM pg_proc p JOIN pg_namespace n2 ON n2.oid = p.pronamespace
            WHERE n2.nspname NOT IN ('pg_catalog','information_schema') AND pg_get_userbyid(p.proowner) <> tgt
  LOOP EXECUTE format('ALTER ROUTINE %s OWNER TO %I', r.fn, tgt); n := n + 1; END LOOP;

  FOR r IN SELECT t.oid::regtype AS ty FROM pg_type t JOIN pg_namespace n2 ON n2.oid = t.typnamespace
            WHERE n2.nspname NOT IN ('pg_catalog','information_schema') AND t.typtype IN ('e','d')
              AND pg_get_userbyid(t.typowner) <> tgt
              AND NOT EXISTS (SELECT 1 FROM pg_class c WHERE c.reltype = t.oid)
  LOOP EXECUTE format('ALTER TYPE %s OWNER TO %I', r.ty, tgt); n := n + 1; END LOOP;

  RAISE NOTICE 'reassigned % object(s)', n;
END $do$;

-- Only if the schema is literally owned by pia: a pg_database_owner-owned public schema needs nothing.
DO $$ BEGIN
  IF (SELECT pg_get_userbyid(nspowner) FROM pg_namespace WHERE nspname = 'public')
       NOT IN ('pg_database_owner', 'umami_app') THEN
    ALTER SCHEMA public OWNER TO umami_app;
  END IF;
END $$;
COMMIT;

Re-running it is a no-op. If it aborts, nothing moved — the transaction is the point.

Then, as pia_root connected to postgres (ALTER DATABASE cannot run inside a transaction block):

ALTER DATABASE umami OWNER TO umami_app;
REVOKE CONNECT ON DATABASE umami FROM PUBLIC;
GRANT  CONNECT ON DATABASE umami TO   umami_app;

The credential move itself is two edits to /opt/pia/.env.prod, and their order is not negotiable. umami’s DATABASE_URL is composed in the compose file from UMAMI_DB_USER and UMAMI_DB_PASSWORD, which fall back to the pia credential while unset — but umami derives its session secret from DATABASE_URL when APP_SECRET is unset, so changing the username without a pinned secret logs out every umami session as an unexplained side effect.

cd /opt/pia
openssl rand -hex 32                       # if step 5 did not already pin APP_SECRET
# APP_SECRET=<value> in /opt/pia/.env.prod
docker compose --env-file .env.prod -f docker-compose.prod.yml up -d --force-recreate umami

# Only once umami is running on a pinned secret, in the same file:
# UMAMI_DB_USER=umami_app
# UMAMI_DB_PASSWORD=<umami_app's password>
docker compose --env-file .env.prod -f docker-compose.prod.yml up -d --force-recreate umami

Giving the vector store its own password

pia-vectordb is a separate cluster, but with VECTORDB_PASSWORD unset it authenticates with POSTGRES_PASSWORD — and the app holds that value in Knowledge__ConnectionString. So while the two are equal, the app process still knows the main cluster’s owner password and the role split buys nothing. This is a required cutover step, not optional hardening.

`VECTORDB_PASSWORD` alone does nothing on an existing host

The postgres image reads POSTGRES_PASSWORD only when initializing an empty data directory. Where the volume already exists, the role’s password has to be altered in place first.

cd /opt/pia
# This cluster's `pia` role is unrelated to the main cluster's role of the same name.
docker compose -f docker-compose.prod.yml exec pia-vectordb \
  psql -U pia -d pia_knowledge -c "ALTER ROLE pia PASSWORD '<vectordb-password>';"

Then put that value in both files — VECTORDB_PASSWORD in /opt/pia/.env.prod, so a rebuilt volume initializes with it, and the same password in Knowledge__ConnectionString in /opt/pia/.env.prod.server — and recreate the app:

docker compose --env-file .env.prod -f docker-compose.prod.yml up -d --force-recreate pia-server

Knowledge-base lookups fail between the ALTER ROLE and that recreate; nothing else reaches this cluster. Afterwards, 28P01 in docker compose logs pia-server means Knowledge__ConnectionString still carries the old password.

De-privileging pia

-- as pia_root
ALTER ROLE pia NOSUPERUSER NOBYPASSRLS NOREPLICATION;

All three attributes, in one statement. The initdb bootstrap role carries rolbypassrls and rolreplication explicitly, so removing SUPERUSER alone leaves pia bypassing every policy while pg_policies still looks exactly right. Confirm with the query below rather than assuming.

Cutover order

Every row before the connection-string change is inert: the roles exist and the policies are created, but the app still connects as the owner and is exempt from them.

Row 2 is the exception to “optional” — splitting the environment files is required on every host the moment the current docker-compose.prod.yml lands, whether or not you adopt anything else on this page.

#	Step	Rollback
1	Back up: pg_dumpall --roles-only, then the data dump. Record the running image digest (docker inspect --format '{{index .RepoDigests 0}}' ghcr.io/pia-ai-dev/pia-server:latest).	—
2	Split /opt/pia/.env.prod into .env.prod + .env.prod.server (step 5). Required before any deploy, independently of everything below.	Merge both files back into .env.prod and restore the previous docker-compose.prod.yml
3	Create and verify pia_root. Do not proceed unless both verification commands above behaved as stated.	DROP ROLE pia_root;
4	Provision pia_app, pia_maint, umami_app.	ALTER ROLE … NOLOGIN on all three — unusable immediately. Dropping them is separate cleanup: DROP OWNED BY must run inside database pia, or DROP ROLE fails on the dependent grants
5	Deploy the build carrying the migration, connection strings still unset. Policies land; the owner keeps its exemption.	Redeploy the recorded digest via a compose override
6	Pin APP_SECRET in .env.prod and recreate umami on it alone.	Remove the line and recreate (sessions log out again)
7	In .env.prod.server, move Database__ConnectionString to pia_app and add Database__MaintenanceConnectionString; docker compose up -d --force-recreate pia-server. First step that enforces anything.	Revert Database__ConnectionString to Username=pia, drop the maintenance key, recreate. Seconds, no SQL to undo
8	Soak — watch docker compose logs -f pia-server for 42501 and for at least one retention tick. Then transfer umami’s objects, set UMAMI_DB_USER / UMAMI_DB_PASSWORD in .env.prod and recreate umami.	Unset both keys and recreate; pia is still a superuser here, so it works regardless of ownership
9	Give pia-vectordb its own password: ALTER ROLE, then VECTORDB_PASSWORD in .env.prod and the matching Knowledge__ConnectionString in .env.prod.server, recreate pia-server.	ALTER ROLE pia PASSWORD back to the old value, revert both keys, recreate
10	ALTER ROLE pia NOSUPERUSER NOBYPASSRLS NOREPLICATION;	ALTER ROLE pia SUPERUSER BYPASSRLS REPLICATION; as pia_root — one statement, instant, no restart
11	Verify, then re-run pg_dumpall --roles-only into the backup set.	—

Verifying the split

-- docker compose -f docker-compose.prod.yml exec postgres psql -U pia_root -d pia
SELECT rolname, rolsuper, rolbypassrls, rolreplication, rolcanlogin FROM pg_roles
 WHERE rolname IN ('pia', 'pia_app', 'pia_maint', 'umami_app') ORDER BY 1;
-- once pia is de-privileged, every row: f | f | f | t
-- rolreplication is the discriminating column: an ALTER ROLE that dropped only two of the three
-- attributes leaves pia bypassing every policy, and nothing else here shows it

SELECT count(*) FROM pg_policies
 WHERE schemaname = 'public' AND tablename IN ('assignments', 'assignment_events');
-- 6: a user policy, a maintenance policy and an owner-exemption policy per table

Then one functional check, as pia:

SELECT count(*) FROM assignments;        -- non-zero
SELECT count(*) FROM assignment_events;  -- non-zero

A zero on either count means a missing owner-exemption policy

Under FORCE that is a silent deny-all for the owner, with no error. Fix it on the spot while pia_root is live — CREATE POLICY assignment_events_owner ON assignment_events FOR ALL TO pia USING (true) WITH CHECK (true); — and do not restart pia-server until it is fixed, because the next boot’s migration step would run against a deny-all owner.

Finally, in the app: a normal user still sees their own assignments, and the admin Mesh roll-up is not empty. The roll-up runs on pia_maint, so an empty one means the maintenance string is wrong.

Traps

• init-db.sh cannot provision roles. /docker-entrypoint-initdb.d fires only on an empty data directory, so on an existing host the script never runs. Role provisioning is the human step above.
• Production runs :latest, so code and config cannot be rolled back independently. An older image plus a pia_app connection string means empty reads and 500s on POST. That is what the recorded image digest in cutover row 1 is for.
• pg_dumpall --roles-only belongs in the backup routine, before the data dump. Roles are cluster-wide: restoring a database dump does not recreate them, and every GRANT … TO pia_app in that dump fails if the roles are absent. Restore roles first, application data second.
• Self-healed roles are not a breach, but they do break the zero-row assertion. If pia_app or pia_maint was ever created by a boot’s disaster-recovery self-heal rather than by pia_root, pia created them and holds admin_option on them. Drop and recreate as pia_root. Once they are superuser-created, rotating their passwords is a pia_root operation — pia gets permission denied to alter role.

8. Verify

On the server, the long-running containers should all be up — caddy, pia-server, pia-web, pia-docs, postgres, pia-vectordb, umami and temporal, minus any you dropped from the compose file.

Four more are one-shot jobs that have already finished: pia-migrate, temporal-db-init, temporal-schema-init and temporal-namespace. Exited (0) on each is the correct state, and they stay listed in docker ps -a — one container per job, replaced in place by the next deploy rather than piling up. Leave them there: their logs are the only record of what the migration and the schema init actually did. docker compose ps without -a lists just the running services.

docker compose -f /opt/pia/docker-compose.prod.yml ps
docker compose -f /opt/pia/docker-compose.prod.yml logs            # all logs
docker compose -f /opt/pia/docker-compose.prod.yml logs pia-server # one service

From anywhere:

curl https://<your-domain>/health   # should return 200 + JSON

Caddy obtains TLS certificates on first request — the very first hit may take a few seconds.

Troubleshooting

Caddy can’t get TLS certificates

• Verify DNS records resolve to the server IP.
• Open ports 80 and 443 in your provider’s firewall.
• Read docker compose logs caddy — ACME failures are noisy and self-explanatory.

pia-server won’t start

• env file /opt/pia/.env.prod.server not found means the environment files have not been split yet.
• Confirm Postgres is healthy: docker compose ps postgres.
• Re-check both files — POSTGRES_PASSWORD in .env.prod must equal the password in Database__ConnectionString in .env.prod.server, unless you have adopted the role split, in which case that string carries pia_app’s password and POSTGRES_PASSWORD stays pia’s.
• Read docker compose logs pia-server.

Can’t pull images from ghcr.io

• Re-authenticate: docker login ghcr.io (step 4).
• Confirm the token you were issued still carries read:packages.
• The packages are private; an anonymous docker pull fails with denied whatever the tag.

Firewall

Port	Protocol	Purpose
22	TCP	SSH
80	TCP	HTTP (Caddy redirect + ACME)
443	TCP	HTTPS