Audit & Privacy Operator Runbook

Operational reference for SREs and on-call engineers responding to issues in the Compliance Dashboard (spec 36) — unified audit log, PII access log, failed-access log, retention engine, legal holds, DSAR workflow, right-to- erasure, and SOC 2 evidence package generation.

Audience. Anthropic-style SRE runbook — assumes shell access to the production server (serge@clear) and a working psql session against the production database. For end-user docs (operator-facing UI walkthroughs) see the compliance dashboard pages directly.

---

Quick reference — service map

Component	Container	Notes
API	production-api-1 / staging-api-1	Routes under /api/v1/compliance/* + the unauthenticated /api/v1/compliance/soc2/download/:token
Worker (BullMQ)	production-worker-1 / staging-worker-1	Notifications queue, email queue, SOC 2 evidence ZIP generation runs out-of-band of the API too (worker.ts only handles notifications + email — SOC 2 currently uses setImmediate from the API process)
Postgres	production-postgres-1	RLS-scoped compliance_* tables + audit_events
MinIO (S3)	production-minio-1 / staging-minio-1	Bucket frontline-documents, prefixes tenant/{tenant_id}/{soc2-packages,audit-log-export,dsar-packages}/...
Redis	production-redis-1	BullMQ queues + distributed locks for the scheduler

Compliance scheduler cadences

The compliance-scheduler.ts Fastify plugin fires the following periodic jobs. All are gated by a Redis distributed lock so multi-replica deployments don't multiplex.

Job	Cadence	Lock key shape
Retention enforcement (nightly purge/archive)	24h	lock:compliance-scheduler:retention-enforcement:{YYYY-MM-DD}
DSAR deadline alert scan	24h	lock:compliance-scheduler:dsar-deadline-alerts:{YYYY-MM-DD}
PII access volume alert scan	15min	lock:compliance-scheduler:pii-alert-scan:{15min-slot}
Cross-tenant probe alert scan	15min	lock:compliance-scheduler:cross-tenant-probe-scan:{15min-slot}
Audit log export recovery sweep	60s	lock:compliance-scheduler:audit-log-export-recovery:{1min-slot}
SOC 2 package recovery sweep	60s	lock:compliance-scheduler:soc2-package-recovery:{1min-slot}

Set DISABLE_SCHEDULED_JOBS=true in dev/test environments to suppress all of them. NEVER set this in production.

Key tables

Table	Purpose
audit_events	Append-only audit trail (every privileged or sensitive operation)
compliance_audit_log_export_jobs	Async export job lifecycle
compliance_soc2_evidence_packages	SOC 2 ZIP generation job lifecycle
compliance_soc2_download_tokens	One-shot download tokens (spec §11.4, Phase 36.6e)
compliance_retention_policies	Per-category retention overrides
compliance_retention_runs	Retention worker run log
compliance_dsar_requests	DSAR (PIPEDA + CCPA) lifecycle
compliance_erasure_records	Per-erasure operation audit (anonymization token + scope)
legal_holds	Legal hold scope + status
compliance_pii_access_alerts	PII volume threshold alert queue
notification_delivery_policies	Mandatory categories (compliance_alerts is one)

---

Compliance bootstrap — required env vars

These env vars must be set on the API + worker containers, stable across deploys, and never rotated without a coordinated session-invalidation window.

Auth secrets — rotating any of these invalidates active state

Variable	What rotates if you change it
JWT_SECRET	All active sessions invalidated (users get kicked back to login)
SSO_SECRET_KEY	Per-tenant encrypted IdP client secrets become unreadable; tenant admins must re-register every SSO provider
DEVICE_TRUST_SECRET	All "Remember this device" cookies invalidated
ERASURE_ANONYMIZATION_SALT	One-way door. Future erasure runs produce different anonymization tokens for the same (tenant_id, subject_id). Past anonymization tokens become non-deterministic; cross-tenant correlation protection downgrades for already-erased subjects. Rotate only with a documented incident response justification.

The CI pipeline preserves existing values from the server. To audit them on the server: docker exec production-api-1 env | grep -E '^(JWT|SSO|DEVICE_TRUST|ERASURE_)' | sed 's/=.*/=<redacted>/'.

URL config — must point to the production app domain

Variable	Production	Staging
APP_BASE_URL	https://app.frontlinehq.io	https://staging.frontlinehq.io
NEXT_PUBLIC_APP_URL	https://app.frontlinehq.io	https://staging.frontlinehq.io
CANDIDATE_PORTAL_URL	https://app.frontlinehq.io	https://staging.frontlinehq.io
STORAGE_PUBLIC_URL	https://storage.frontlinehq.io	https://staging.storage.frontlinehq.io

In NODE_ENV=production the URL resolver in notifications-core throws if none of APP_BASE_URL, APP_URL, or NEXT_PUBLIC_APP_URL is set — preventing the silent localhost-in-transactional-email failure mode from leaking compliance notification deep-links.

Storage / object access

Variable	Notes
STORAGE_ACCESS_KEY / STORAGE_SECRET_KEY	MinIO/S3 credentials. Read-only audit access acceptable; SOC 2 + DSAR workers need write access.
STORAGE_PUBLIC_ENDPOINT	The bucket's public hostname used for signed URLs. Must be reachable from the user's browser.

Notification / email

Variable	Notes
NOTIFICATIONS_FORCE_TRANSPORT	Staging hardcodes database (capture-only — no emails leave the building). Production must be unset or ses.
AWS_SES_REGION + IAM role	SES dispatcher in the worker. The compliance_alerts mandatory category routes through here for cross-tenant probe + PII alerts.

---

Smoke-test runbook for the compliance-officer role

Run this after every production deploy that touches compliance/*, migrations/16x_* or 17x_*. Time-box: ~10 minutes.

All steps assume a compliance_officer JWT (or tenant_admin — both have the permission set). Use the Belov staging tenant for non-prod verification.

MFA enrollment prerequisite for bulk exports. Audit log export, SOC 2 generate, retention runs, legal-hold release, and DSAR erasure execute are gated behind heightened MFA (spec 36 §14 — req.mfa === true). The session JWT only gets mfa: true if the user logged in via the TOTP challenge OR via a trusted-device cookie. Sessions where users.mfa_enabled = false will never have mfa: true — those users must enroll MFA at /settings/mfa first, then sign out + back in completing the authenticator-code prompt. The audit log page renders an actionable amber banner pointing to Settings → MFA when this gate fires.

1. Audit log smoke

1. Navigate to /compliance/audit-log. Verify rows render with structured columns (actor name + email, resource name, result pill).
2. Apply the Result = denied filter. URL should update to ?result=denied; only *.denied / *.access_denied rows show with red pills.
3. Use the actor picker (combobox) to filter by a specific employee — verify the actor_email is sent to the API (network tab) and the result list narrows.
4. Click Export → CSV → verify a 202 + job_id is returned, then poll completes, and the CSV downloads. The file's first line is the column header; subsequent lines are events.

Common issue: "Export job COMPLETED but no download URL was issued". Cause: storage_key was set but the MinIO object isn't there (staging-only issue from old seed data). Fix: re-run the export — fresh jobs upload correctly.

2. PII access log smoke

1. Navigate to /compliance/pii-access-log. Verify rows are limited to PII-classified events (national_id reveal, license_number viewed, change_request reviews, notification body views).
2. Click Generate report. Verify the three sections render: by_actor, by_data_type, by_period_month.
3. Click Download CSV on the report — verify a multi-section CSV downloads with a Content-Disposition: attachment; filename=… header.

3. Failed access log smoke

1. Navigate to /compliance/failed-access-log. Verify the summary cards show: failures by actor, failures by IP, cross_tenant_probe count.
2. Apply Severity = HIGH filter — verify only cross_tenant_probe + break_glass rows show.

4. Retention policy smoke

1. Navigate to /compliance/retention. Verify the 11 categories listed with their regulatory minimums.
2. Edit a tenant-customizable category (e.g., chat_transcripts) → change override days → save. Verify the change appears in the row.
3. Attempt to set an override BELOW the regulatory minimum → 422 with floor detail.
4. Run a dry-run via POST /retention-runs → verify the response surfaces records_considered per category without modifying any rows.

5. Legal hold smoke

1. Navigate to /compliance/legal-holds. Click New legal hold.
2. Scope: pick a specific employee or candidate via the picker. Save.
3. Verify the hold appears in the list with status active. Verify the audit log shows a compliance.legal_hold.create event.
4. Click Release → MFA prompt → confirm. Verify status flips to released and released_at is set.

6. DSAR workflow smoke

1. Navigate to /compliance/dsar. Submit a new DSAR for an EMPLOYEE subject with type=ACCESS and jurisdiction=PIPEDA.
2. Verify deadline_at = submitted_at + 30 days shows correctly on the detail page.
3. Click Generate package (DSAR is in IN_PROGRESS or after the worker collects data). Verify the worker run sets status to DATA_GATHERED.
4. Click Download package (24h link) → verify ZIP downloads + contains module-specific data (collector ran).
5. Confirm a compliance.dsar.package_downloaded event appears in the audit log.

7. SOC 2 evidence package smoke

1. Navigate to /compliance/soc2. Click New evidence package.
2. Select a 30-day period + all 4 categories → click Generate.
3. Verify the dialog shows Queued… → Generating… then closes; the new package appears in the list with status COMPLETED within ~15 seconds.
4. Click Download on the new row. Verify the ZIP downloads + opens. Inside: cover_sheet.json, cover_sheet.pdf, generation_audit_event.json, plus a CSV per category.
5. Critical: click Download a second time on the SAME row, BEFORE the URL is consumed. Verify a fresh URL is minted (different token_id in network tab). Then click the FIRST URL again — it should now 410 EXPIRED_OR_CONSUMED (one-shot tokens, spec §11.4).

8. Cross-tenant probe alert smoke (optional — requires multi-tenant setup)

1. From tenant A's compliance officer account, attempt to GET a SOC 2 package UUID known to belong to tenant B (you can pluck one from staging DB).
2. Repeat 3+ times within 60 minutes.
3. Within 15 minutes (next scheduler tick), the compliance officer on tenant A should receive an in-app notification + email titled "Cross-tenant access probe (single user)".

---

Common issues + fixes

"Download returns 404 / NoSuchKey"

The DB row's package_ref or storage_key points to a MinIO object that doesn't exist. Two causes:

• Old seed data — fixed via the one-time backfill in /tmp/backfill-compliance.py (staging only).
• Storage upload failed mid-job — the row's status should be FAILED with last_error populated. Re-run the generation.

"Download returns 410 / EXPIRED_OR_CONSUMED"

Expected if:

• The token has been used once already (SOC 2 download tokens are one-shot per spec §11.4)
• The token's expires_at is past (default 24h after mint)
• The expires_at on the underlying job/package row is past (longer-window check, applies to audit-log export + SOC 2)

Click Download again from the UI to mint a fresh token.

"Migration 173 / 175 / 176 violates constraint"

Migration 173 (spec 36 v2.13, Phase 36.6d): pre-async-generation SOC 2 rows are missing completed_at / expires_at / cover_sheet_json. The migration includes a backfill UPDATE that derives values from existing fields — if you hit this on a third-party fork or a partially-applied migration, cherry-pick the backfill block from the migration file.

Migration 175: requires migration 174's column additions. Apply in order.

Migration 176: idempotent via ON CONFLICT DO NOTHING. Safe to re-run.

"Compliance officer not receiving cross-tenant probe alerts"

Check, in order:

1. Is compliance_alerts in notification_delivery_policies for the tenant with mandatory=true?

   SELECT * FROM notification_delivery_policies
   WHERE tenant_id = '<tenant>' AND event_category = 'compliance_alerts';
   If missing, apply migration 176 or the tenant-provisioner seed.
2. Is the user in the user_role_bindings table with role compliance_officer or tenant_admin?
3. Is the auth.session.cross_tenant_probe action actually emitting? Check audit_events filtered on that action — if zero rows in the past day, no probes have occurred AND detectCrossTenantProbe isn't wired into the relevant route.
4. Is the scanner running? Check the API logs for compliance.cross-tenant-probe-scan: starting 15-min sweep — if absent, DISABLE_SCHEDULED_JOBS may be set.

"SOC 2 generation stuck at PENDING"

The recovery sweep runs every 60s. After 2 minutes max, a stale PENDING should be re-fired automatically. If it stays PENDING longer:

• Check the API container is actually running (docker ps | grep api)
• Check Redis is reachable (docker exec api redis-cli -h redis ping should return PONG)
• Check the lock:compliance-scheduler:soc2-package-recovery:* lock isn't held by a dead replica (rare — TTL is 45s; deadlocks self-clear in under a minute)

---

Deploy-time checklist

After every Compliance Dashboard deploy:

1. Migrations applied: SELECT name, run_on FROM pgmigrations ORDER BY run_on DESC LIMIT 5;
2. Mandatory categories seeded: SELECT COUNT(*) FROM notification_delivery_policies WHERE event_category = 'compliance_alerts' — should match active tenant count.
3. Scheduler running: docker logs production-api-1 2>&1 | grep -i "compliance.*scheduler\|compliance.*scan" | tail -20 — should show recent sweep starts.
4. Smoke-test runbook §1–§7 end-to-end (~10 min).

---

See also

• Spec 36 — Compliance & Audit Dashboard
• Spec 18 — Notifications (channel routing + mandatory categories)
• Spec 49 — Notification body retention (legal-hold projection)

---

Last verified against spec 36 v2.18 + spec 18 v1.12 (2026-05-18). The scheduler-cadence table, migration list, and env-var catalog drift quickly — when running a smoke test that fails on an absent migration or a changed cadence, treat this doc as stale and check the linked specs before debugging downstream.