Skip to main content

Offboarding Deploy Runbook

Operational reference for the deployment / migration sequence the Offboarding module requires when shipping a new release.

Audience. SRE / on-call running a production deploy. Pairs with the per-feature operator runbook at offboarding/operator-runbook.

Migration dependency​

Spec 43 is migration-coupled. Every release must apply the matching migrations before the new API/worker containers start, otherwise the new code hits 42703 undefined_column errors (the routes are written defensively with SAVEPOINT fallbacks for some but not all columns).

MigrationSpec §Purpose
067§4Original offboarding tables (requests, tasks, asset returns, revocations)
201§10.4KT-template support
206§7A.4, §7A.5, §7D, §9.1Scheduling fields: offboarding_sla_days, exit_interview_mode, contract_end_cr_lead_days, plus per-row *_sent_at columns
207§10.2, §10.3KT template & template-item tables; KT-task §10.3 columns
208§12.7, §17.1alumni_rehire_invitations, perf_hub_attrition_signal_labels
209§9.7Contractor opt-ins (include_contractors_exit_interview, include_contractors_alumni)
211§9.3Departure-reason backfill (existing exit interviews → VOLUNTARY_PERSONAL)
212§16.1Alumni consent provenance (opt_in_source, opt_in_consented_at)
213§9.3, §17.1Tenant departure-reason code catalog table + 11-platform seed
214GA-hardeningtenant_configs.offboarding_enabled kill switch

A production deploy from a previous version that lands ≥1 of these migrations MUST run them before the new API/worker images boot. The API does not auto-migrate on startup.

Standard deploy sequence​

# 1. Sanity: are we at the expected migration baseline?
docker exec production-postgres-1 \
  psql -U frontline -d frontline -tAc \
  "SELECT name FROM pgmigrations ORDER BY id DESC LIMIT 5;"

# 2. Apply pending migrations against PROD. node-pg-migrate is
#    idempotent — it skips already-applied rows. The 'migrate' service
#    in docker-compose.yml is the standard runner; it executes once and
#    exits, then the api/worker containers start with the schema in place.
docker compose -f docker-compose.production.yml run --rm migrate

# 3. Verify the schema is up to spec 43 ≥ 214 (latest at time of writing).
docker exec production-postgres-1 \
  psql -U frontline -d frontline -tAc \
  "SELECT name FROM pgmigrations WHERE name LIKE '%offboarding%' ORDER BY id DESC LIMIT 5;"

# 4. Bring up api + worker against the new image.
docker compose -f docker-compose.production.yml up -d api worker

# 5. Smoke: hit operational-status with an HR admin token.
curl -s -H "Authorization: Bearer $TOK" \
  https://api.frontlinehq.io/api/v1/offboarding/operational-status \
  | jq '.data.tenant, .data.workers[].queue_name, .data.in_flight'

# Expected: tenant.offboarding_enabled = true; all 5 worker queue_names
# present; in_flight numbers reasonable for the tenant's headcount.

If step 5 returns a non-200, roll back the api/worker images (step 6 below) — do NOT roll back the migrations. The migrations are designed to be additive and backward-compatible with the prior code (all new columns have defaults, all new tables are referenced via defensive SAVEPOINT-guarded reads).

# 6. ROLLBACK (api/worker only). Migrations stay applied — they are
#    forward-compatible with the previous code.
docker compose -f docker-compose.production.yml down api worker
docker tag frontline-api:previous-known-good frontline-api:latest
docker tag frontline-worker:previous-known-good frontline-worker:latest
docker compose -f docker-compose.production.yml up -d api worker

Per-migration notes​

213 (departure-reason code catalog)​

Backfills the 11 platform codes per tenant via INSERT…SELECT CROSS JOIN tenants ON CONFLICT (tenant_id, code) DO NOTHING. Safe on a tenant that already has the row (no-op). If you provision a NEW tenant after this migration ran, the tenant-provisioner.ts step 22 (seedPlatformDepartureCodes) takes over — no further action.

214 (kill switch)​

ALTER TABLE tenant_configs ADD COLUMN offboarding_enabled BOOLEAN NOT NULL DEFAULT true. Adds the column on the existing row for every tenant with the default true, so previously-enabled tenants stay enabled. There is no separate backfill step.

Dependency-drift checks (a 2026-05-29 verification lesson)​

A browser verification of the HR-16 feature surfaced two image-drift issues that bit a dev stack but would equally bite a production deploy if the image were stale:

  • API image missing jszip — the compliance / SOC2-evidence path pulls it in transitively. A fresh npm ci includes it (it's in package.json); a docker image baked before that dependency was added does not. Always rebuild images from a clean state on the release that adds a new top-level dependency.
  • Web image missing cmdk — the command palette imports it. Same resolution as above.

Pre-deploy check (run on the runner image, BEFORE pushing):

# Build the new images locally first, then introspect their node_modules
docker build -t frontline-api:rc -f services/api/Dockerfile .
docker run --rm frontline-api:rc \
  sh -lc 'for d in jszip cmdk pino bullmq pg; do
    [ -d /app/node_modules/$d ] && echo "have $d" || echo "MISSING $d"
  done'

If anything reads MISSING, the Dockerfile / lockfile has drifted and the build is not safe to deploy.

Tenant kill switch — emergency use​

-- DISABLE offboarding for ONE tenant (operator-driven kill switch, mig 214).
UPDATE tenant_configs SET offboarding_enabled = false WHERE tenant_id = $1;

-- VERIFY (the next mutating call from that tenant should 503).
-- RE-ENABLE.
UPDATE tenant_configs SET offboarding_enabled = true  WHERE tenant_id = $1;

See operator-runbook for the behavioral contract (what changes when disabled; what doesn't).

  • Spec 43 Offboarding Module Spec — specs/43_Offboarding_Module_Spec.md
  • Operator runbook — operator-runbook
  • API reference (overview) — overview
  • Migration files — app/services/db/migrations/*.sql