Tier-2 Capacity Baseline — Runbook

Status

• Version: v1.0
• Last updated: 2026-05-06
• Owner: Engineering
• Audience: anyone with a working dev laptop and ~1 hour

Purpose

Run the Tier-2 capacity test against a locally-seeded 5K-agent tenant and capture defensible baseline numbers. This is the procedure for the first run; subsequent runs reuse the same harness with smaller deltas.

The test produces three numbers we need before signing any enterprise SLA:

1. p95 read-keyset latency at 200 VUs sustained 5 min
2. p95 write-path latency at 50 VUs sustained 5 min
3. The actual error/skip distribution (does the system stay green at this load, or do we need infra tuning first?)

Pre-flight

Check	Command	Expected
Docker running	docker ps	no error
k6 installed	k6 version	k6 v0.4x.x
jq installed	jq --version	jq-1.x
Node 24+	node --version	v24.x
Dev DB port free	lsof -i :5433	empty

If any check fails, fix it before continuing. The harness assumes everything above is working.

Step 1 — Stand up the local stack

cd app
npm run test:infra:up
# Wait ~30s for postgres + redis + minio to settle.

Health-check:

curl -s http://localhost:3100/api/v1/_health | jq .
# Expect: { "status": "ok", "db": "ok", "redis": "ok" }

Step 2 — Synthesize the fixture

npx tsx app/load-tests/tier2-synth.ts

Expected output (truncated):

Tier-2 synthesizer: 5000 agents, 5 LOBs, 7 days × 1 shifts (≈35,000 rows)
Tenant: 22000000-... (slug=tier2-load-test)
Admin:  tier2-admin@tier2.test / Tier2LoadTest123!
✓ tenant + client + site (Xms)
✓ admin user with tenant_admin + workforce_planner roles
✓ 5 LOBs
✓ 5000 agents (created=5000, existing=0) (X.Xs)
✓ schedule period 2026-MM-DD → 2026-MM-DD
✓ 35000 shift assignments (X.Xs)

Total elapsed: X.Xs

If you see "shifts already populated for this period — skipping": fixture exists from a prior run. That's fine; it's idempotent.

If it errors with permission denied or relation does not exist: migrations haven't applied. Run npm run db:migrate then retry.

Step 3 — Get a bearer token

TIER2_TOKEN=$(curl -s http://localhost:3100/api/v1/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"tier2-admin@tier2.test","password":"Tier2LoadTest123!","tenant_slug":"tier2-load-test"}' \
  | jq -r .access_token)
echo "${TIER2_TOKEN:0:20}..."   # sanity-check it's non-empty

If empty: the synth admin user didn't seed properly. Re-run step 2.

Step 4 — Read-path baseline

STAGING_BASE_URL=http://localhost:3100 \
STAGING_TOKEN=$TIER2_TOKEN \
k6 run app/load-tests/scheduling-keyset.js

Note this currently runs at the Tier-1 default of 10 VUs. For the Tier-2 baseline crank to 200 by editing the stages block in the script, or override via env if your version of the script supports it.

While it runs, capture index-hit telemetry from a second terminal:

docker exec -i frontline-postgres-1 psql -U frontline frontline_test <<'SQL'
SELECT relname, indexrelname, idx_scan, idx_tup_read
FROM pg_stat_user_indexes
WHERE relname IN ('workforce_swap_requests', 'workforce_shift_assignments', 'workforce_schedule_periods')
ORDER BY idx_scan DESC LIMIT 20;
SQL

Record the numbers in the Results section below.

Step 5 — Write-path baseline

STAGING_BASE_URL=http://localhost:3100 \
STAGING_TOKEN=$TIER2_TOKEN \
k6 run app/load-tests/tier2-write-path.js

While it runs, watch BullMQ queue depth from a second terminal:

docker exec -i frontline-redis-1 redis-cli LLEN bull:notification:wait
# Should stay bounded — anything > 10K means the queue worker isn't keeping up.

Step 6 — Capture results

Fill in this table from the k6 console output:

Metric	Threshold	Read keyset	Write path
p95 latency (ms)	<2000 read / <800 write	_____	_____
Error rate (%)	<1 read / <2 write	_____	_____
Per-endpoint success rate (%)	≥95-99	_____	_____
Queue depth max	<10,000	n/a	_____

Past-the-threshold rows should also note which threshold tripped first — that's the signal for what to tune.

Step 7 — Decision tree

Symptom	First action	Second action
Read p95 > 2s	Verify pg_stat_user_indexes.idx_scan > 0 for migration-139 indexes; if 0, query is bypassing the index	Increase Postgres shared_buffers (default 128MB is low)
Write p95 > 800ms	Check if it's specifically /exceptions (idempotency-key contention?)	Lower k6 VUs to find the knee, then scale Postgres max_connections
Error rate > 1%	Look at the worst endpoint's response body in the k6 output	If 409s dominate write-shift-assign: the draft period is saturating, expected late in run
Queue depth > 10K	Check worker logs for stuck handlers	Increase BullMQ concurrency per queue
Synth aborts on > 100K shifts	This is the resource guardrail	Set TIER2_FORCE=1 if the machine has the headroom, or reduce TIER2_AGENTS

Results template

Copy this into a follow-up commit when you've got numbers:

## Tier-2 baseline run YYYY-MM-DD

Hardware: <e.g. M2 Mac, 16GB RAM, postgres in docker>

### Read-path (scheduling-keyset.js, 200 VUs sustain 5min)
- p95: ___ ms (target \<2000)
- error rate: ___% (target \<1)
- Top index hit: <relname.indexrelname idx_scan=____>
- Verdict: PASS / FAIL / NEEDS-TUNING

### Write-path (tier2-write-path.js, 50 VUs sustain 5min)
- exception p95: ___ ms (target \<800)
- shift-assign p95: ___ ms (target \<800)
- exception success rate: ___% (target >99)
- shift-assign success rate: ___% (target >95)
- max queue depth: ___ (target \<10,000)
- Verdict: PASS / FAIL / NEEDS-TUNING

### Notes
<anything weird; what was the first threshold to trip; what we tuned to fix>

File the results in docs/operations/tier2-baseline-YYYY-MM-DD.md and link it from the next sales / SLA conversation.

Re-running

Subsequent runs only need steps 3-6 (the synthesizer is idempotent; tear down with npm run test:infra:down if you want a clean slate).

If you change the synth (tier2-synth.ts), tear down first to drop stale fixtures:

npm run test:infra:down
npm run test:infra:up
npx tsx app/load-tests/tier2-synth.ts

Known limitations

• The k6 scenarios don't currently exercise /swaps (FK-validation cost dominates without real shift IDs); add a SharedArray of real shift IDs from the synth output when this becomes the next gap.
• The harness runs everything against a single tenant. Multi-tenant noisy-neighbor effects need a separate Tier-3 harness.
• Postgres auto_explain is not enabled by default; if you suspect a specific slow query, enable it before re-running and grep the worker logs.

Change log

• v1.0 (2026-05-06): Initial runbook covering one full pass: synth → token → read → write → results → tuning decision tree.