Skip to main content

Performance Hub Operator Runbook

Operational reference for SREs and on-call engineers responding to issues in the Agent Performance Hub (spec 27) — the nightly composite-score snapshot pipeline, supervisor alerts, formal reviews, and improvement plans.

Audience. SRE / on-call. Assumes shell access to the production server (serge@clear) and a working psql session against the production database. For end-user docs (HR walkthroughs of the dashboard) see /settings/performance and /performance in-app.

Quick reference — service map​

ComponentContainerNotes
APIproduction-api-1 / staging-api-1Routes under /api/v1/perf-hub/*
Worker (BullMQ)production-worker-1 / staging-worker-1Runs perf-hub-snapshot + perf-hub-review-period-end + perf-hub-plan-check-in queues
Postgresproduction-postgres-1Tables: perf_hub_snapshots, perf_hub_configs, perf_hub_goals, perf_hub_reviews, perf_hub_improvement_plans, plus the shared worker_cron_completions ledger
Redisproduction-redis-1BullMQ queues + scheduled-job state

Cron-chain cadences​

The Performance Hub upstream + its CSAT downstream form a chain. Times are UTC. Each downstream requireUpstream() gate refuses to start until the upstream marker is fresh (see services/worker/src/lib/cron-chain.ts).

JobCadenceUpstream gate
perf-hub-snapshot0 2 * * * (02:00)— (chain head)
csat-score-snapshot30 2 * * * (02:30)perf-hub-snapshot
csat-alert-evaluation0 3 * * * (03:00)csat-score-snapshot
csat-coaching-trigger15 3 * * * (03:15)csat-score-snapshot
perf-hub-review-period-endhourlyperf-hub-snapshot
perf-hub-plan-check-inhourlyperf-hub-snapshot

The status panel​

HR admins (and anyone with perf_hub.config.manage) see a "Snapshot pipeline" panel at the top of /settings/performance — your first stop on a triage call.

It reads GET /api/v1/perf-hub/operational-status and renders:

  • Freshness pill — fresh ≤ 26h, stale ≤ 48h, critical > 48h, never_ran if no marker
  • Tenant coverage — snapshotted_yesterday ÷ active_agents
  • CSAT coverage — fraction of yesterday's rows with score_csat IS NOT NULL
  • Anomalies — tenant-scoped flags from the M3 detector (see below)
  • Downstream chain — per-queue freshness, if more than one queue has ever marked completion

If the panel itself fails to load, the API or DB is down — pivot to general service health (/health/ready).

Failure mode: snapshot freshness drops​

stale (>26h since last marker, ≤48h)​

A single missed run. Usually a runner restart, deploy window, or BullMQ stall.

  1. Check the worker is up: 
    ssh serge@clear "docker compose -p production ps worker"
  2. Look at the most recent run attempt in the worker logs: 
    ssh serge@clear "docker logs production-worker-1 --since 36h 2>&1 | grep -E 'gamification_points_award|perf_hub' | head -50"
  3. If the job never started, restart BullMQ scheduling by bouncing the worker: 
    ssh serge@clear "docker compose -p production restart worker"
  4. After the restart, manually trigger today's run rather than waiting until 02:00: 
    ssh serge@clear "docker compose -p production exec worker node --import tsx -e \"
      import { Queue } from 'bullmq';
      const q = new Queue('perf-hub-snapshot', { connection: { host: 'redis', port: 6379 } });
      await q.add('manual', {}, { jobId: 'manual-' + Date.now() });
      await q.close();
    \""

critical (>48h)​

Two or more consecutive missed runs. Page on-call. While waiting for acknowledgment, run the same diagnostic chain as stale — most critical states resolve to the same root cause (worker container dead) caught later.

never_ran​

A fresh tenant onboarding or a wiped DB. Confirm against worker_cron_completions:

SELECT queue_name, last_completed_at FROM worker_cron_completions ORDER BY queue_name;

If the row is genuinely missing, the worker hasn't completed a single production run yet. Trigger one manually (snippet above).

Failure mode: tenant coverage drop​

Coverage band guide​

coverage_pctToneInterpretation
≥ 95%GreenNormal. A few inactive employees getting missed is expected.
75–95%AmberWorth checking. Likely a recent batch of terminations not yet reflected in employees.status or a planner working off stale assignments.
< 75%RedInvestigate. Probably a regression in the active-employee enumeration query.

Diagnosis​

The snapshot worker enumerates active employees with this shape:

SELECT e.tenant_id, e.id, eph.client_lob_id
  FROM employees e
  JOIN employee_profile_history eph
    ON eph.employee_id = e.id AND eph.effective_to IS NULL
 WHERE e.status = 'active';

Compare the candidate count to the snapshot count for the same date:

WITH candidates AS (
  SELECT count(*) AS n FROM employees e
   JOIN employee_profile_history eph ON eph.employee_id = e.id AND eph.effective_to IS NULL
   WHERE e.tenant_id = '<TENANT>' AND e.status = 'active'
), snapshots AS (
  SELECT count(*) AS n FROM perf_hub_snapshots
   WHERE tenant_id = '<TENANT>' AND snapshot_date = CURRENT_DATE - 1
)
SELECT candidates.n AS candidates, snapshots.n AS snapshotted,
       round(100.0 * snapshots.n / NULLIF(candidates.n, 0), 2) AS pct
  FROM candidates, snapshots;

If candidates - snapshotted is large, look at the worker's errored counter in the last run metadata:

SELECT last_run_metadata FROM worker_cron_completions WHERE queue_name = 'perf-hub-snapshot';

A non-zero errored count means per-employee exceptions during the loop (each one logs at error level with the offending employee_id).

Failure mode: anomalies fired​

The M3 detector runs after the per-employee UPSERT loop and writes any tenant-level anomalies into worker_cron_completions.last_run_metadata.anomalies. The status panel surfaces them as an amber callout.

Three kinds:

row_count_drop​

Today's row count is > 30% below the prior snapshot date.

Likely causes:

  • A bulk-termination event yesterday that the snapshot picked up correctly — confirm by checking employees.status changes
  • Employee profile history rows mass-expired (effective_to filled in) — check employee_profile_history for unexpected effective_to timestamps
  • The active-employee enumeration query regressed (look at the worker's recent diffs)

Recovery: none required if the cause is legitimate (terminations). Otherwise, fix the source and the next nightly run clears the flag.

median_score_drop​

Today's median composite is > 15pts below yesterday's.

Likely causes:

  • An upstream rollup miscounted — most often the QA dimension (qa_evaluations.total_score_pct aggregation). Check the QA module's own status.
  • A planner published a "what-if" schedule mid-day that the attendance dimension misread — check workforce_shift_assignments for unexpected draft_assigned rows
  • Weight configuration changed and the new weights heavily penalize a previously-dominant dimension — check audit_events for recent perf_hub.config.updated

Recovery: verify the upstream is sound, re-run the snapshot. The idempotent UPSERT replaces yesterday's row.

null_composite_spike​

More than 25% of today's rows have NULL composite_score.

Likely causes:

  • Every dimension is NULL for many agents — usually means the source-data workers (QA, training, adherence) haven't run or are emitting empty rows
  • The tenant's weights are misconfigured such that every scored dimension carries weight 0 — the worker already logs composite_score null despite populated dimensions for this; check the worker logs

Recovery: fix the upstream, re-run. Same idempotency story.

Failure mode: CSAT coverage stuck at 0​

The status panel's CSAT card shows 0%. The score_csat column in perf_hub_snapshots is the join target — if it's all NULL, either CSAT itself isn't running or the CSAT chain gate (csat-score-snapshot's upstream requirement on perf-hub-snapshot) is tripping.

  1. Check the CSAT marker: 
    SELECT * FROM worker_cron_completions WHERE queue_name = 'csat-score-snapshot';
  2. If missing or stale, look at the CSAT worker log: 
    ssh serge@clear "docker logs production-worker-1 --since 36h 2>&1 | grep csat-score-snapshot | tail -50"
  3. If the chain gate is tripping, that means perf-hub-snapshot failed and CSAT correctly refused to start. Resolve the upstream first.
  4. If CSAT did run but coverage is still 0%, verify the snapshot table has agent-scope rows: 
    SELECT count(*) FROM csat_score_snapshots
     WHERE tenant_id = '<TENANT>'
       AND agent_employee_id IS NOT NULL
       AND insufficient_data = false
       AND computed_at >= now() - interval '24 hours';
  5. The PH snapshot reads the latest ROLLING_30D row per agent via scoreCsat() in perf-hub-snapshot.ts. If CSAT has the data but PH isn't picking it up, the cron ordering is broken — verify perf-hub-snapshot runs at 02:00 and csat-score-snapshot at 02:30 (PH reads yesterday's CSAT rows; if CSAT runs first on a given day, that day's PH still picks up the previous-day CSAT — that's by design).

Recovery: backfill missing days​

When the snapshot pipeline has been down for multiple days, use the backfill script. It walks any date range and invokes the production handler once per day, with alerts suppressed (so a 30-day backfill doesn't fire 30 supervisor notifications).

ssh serge@clear "docker compose -p production exec worker pnpm backfill:perf-hub --from 2026-05-01 --to 2026-05-10"

The handler's UPSERT is idempotent — re-running the backfill over already- populated dates is safe. The cron-chain marker is NOT written by backfill runs (they pass opts.targetDate, which skips markJobCompleted) so the freshness pill keeps reflecting the real most-recent live run.

Diagnostic queries cheat sheet​

Composite score distribution for a tenant on a given date​

SELECT percentile_cont(ARRAY[0.10, 0.25, 0.50, 0.75, 0.90])
         WITHIN GROUP (ORDER BY composite_score) AS percentiles,
       count(*) AS n,
       count(*) FILTER (WHERE composite_score IS NULL) AS null_n
  FROM perf_hub_snapshots
 WHERE tenant_id = '<TENANT>' AND snapshot_date = '<YYYY-MM-DD>';

Which agents are missing from yesterday's snapshot?​

WITH expected AS (
  SELECT e.id AS employee_id
    FROM employees e
   WHERE e.tenant_id = '<TENANT>' AND e.status = 'active'
), got AS (
  SELECT employee_id FROM perf_hub_snapshots
   WHERE tenant_id = '<TENANT>' AND snapshot_date = CURRENT_DATE - 1
)
SELECT e.employee_id FROM expected e
 LEFT JOIN got ON got.employee_id = e.employee_id
 WHERE got.employee_id IS NULL;

Show the chain status all at once​

SELECT queue_name,
       last_completed_at,
       round(extract(epoch from now() - last_completed_at) / 3600, 2) AS age_hours,
       last_run_metadata
  FROM worker_cron_completions
 ORDER BY queue_name;

Watch the most recent anomaly fan-out across tenants​

SELECT jsonb_array_elements(last_run_metadata -> 'anomalies') AS anomaly
  FROM worker_cron_completions
 WHERE queue_name = 'perf-hub-snapshot';

Escalation​

  • On-call rotation: #frontline-oncall Slack channel; primary rotates weekly.
  • Spec questions: Engineering owns the implementation. HR Operations owns the score weights and the improvement-threshold policy.
  • Data integrity (anomalies that persist > 2 runs): escalate to the Engineering tech lead — a persistent anomaly is a real regression, not a one-off blip.