Skip to main content

Gamification Operator Runbook

This runbook is paired with the PipelineStatusPanel on /settings/gamification (HR-admin view) and the GET /api/v1/gamification/operational-status endpoint behind it. It covers what each panel state means, what to check first, and how to recover.

Spec reference: 27A Gamification §4.1–§4.4 (see specs/27A_Gamification_Spec.md in the repo). Production-readiness loop is complete (B1 + H1 + H2 + H3 + M1 + M2 + M3 + M4 + L1 + L2 all shipped).

The gamification pipeline is a fan of four workers — they're mostly independent, unlike the Performance Hub which is a strict chain. The runbook is organized per-worker because the failure modes don't cascade.

Quick reference — service map​

ComponentWhereOwns
gamification-points-awardworker · handlers/gamification-points-award.tsWrites one row per qualifying source event (QA / training / coaching / attendance / CSAT) into gamification_point_events.
gamification-badge-evalworker · handlers/gamification-badge-eval.tsInserts into gamification_agent_badges for every employee meeting an auto-badge trigger. Gated on perf-hub-snapshot completing today (for the consecutive_composite_above evaluator).
gamification-leaderboard-refreshworker · handlers/gamification-leaderboard-refresh.tsUPSERTs render-ready rankings into gamification_leaderboard_cache per (tenant, scope, metric, window).
gamification-challenge-progressworker · handlers/gamification-challenge-progress.tsAppends daily progress entries to gamification_challenges.progress_json and transitions ACTIVE→PENDING_CONFIRMATION on end_date.
Settings UIweb · /settings/gamification (HR-admin)Renders the panel + engine config + active point-rule table.
Operational endpointapi · routes/gamification/operational-status.tsReads worker_cron_completions and the gamification tables for the panel.

Cron-chain cadences​

QueueCronCadenceFresh bandStale band
gamification-points-award0 3 * * *Nightly, 03:00 UTC (after perf-hub snapshot at 02:00)≤26h≤48h
gamification-badge-eval30 3 * * *Nightly, 03:30 UTC≤26h≤48h
gamification-leaderboard-refresh0 */4 * * *Every 4h≤5h≤8h
gamification-challenge-progress0 4 * * *Nightly, 04:00 UTC≤26h≤48h

The leaderboard refresh has tighter bands than the nightly jobs — see classifyAge dispatch in operational-status.ts.

The status panel​

The panel renders four blocks:

  1. Overall pill — worst of the four worker statuses. The badge color (green/amber/red/grey) tells you whether any worker is unhealthy without scanning the table.
  2. Coverage stats — active agents, engaged this month, active rules, active challenges. The juxtaposition is the diagnostic: rules>0 but engaged=0% means rules don't match any source data (a config issue); rules=0 means HR hasn't set the engine up yet.
  3. 14-day point-event volume sparkline — answers "is the engine still producing output?" Raw row counts, not coverage_pct. A sudden cliff is the strongest signal of a stuck cron — earlier than the worker freshness pill, which is age-based.
  4. Workers table — one row per worker with its freshness pill, last-run timestamp, and a per-worker metadata hint (awards inserted / cache rows refreshed / etc.).

Failure mode: a worker is stale or critical​

stale (>fresh-band, ≤stale-band)​

One missed run. Look at last_run_metadata first — the worker may have completed but produced zero output (e.g. zero matching source rows), which is a no-op, not a failure.

ssh serge@clear "docker compose -f /home/serge/frontline2/app/docker-compose.demo.yml logs worker --since 24h | grep gamification"

Common causes:

  • Worker container restarted during the cron window. BullMQ should pick up the missed job within ~30 min via the repeat schedule; the next nightly run reads fresh sources and the marker advances.
  • Database overrun. A slow INSERT...SELECT on a tenant with millions of qa_evaluations / training_completions can push run-time past the next cron window. Check idx_gamification_point_events_employee_history (and the indexes on the source tables — qa_evaluations.submitted_at, training_module_completions.completed_at).

If the next nightly run advances the marker, you're done — no further action.

critical (>stale-band)​

Two or more missed runs in a row. This is the escalation point.

  1. Confirm the worker container is up: 
    ssh serge@clear "docker compose ps worker"
  2. Tail the live logs for the gamification queues: 
    ssh serge@clear "docker compose logs worker -f | grep -E 'gamification-(points-award|badge-eval|leaderboard-refresh|challenge-progress)'"
  3. Check worker_cron_completions to confirm what the API sees: 
    SELECT queue_name, last_completed_at, last_run_metadata
      FROM worker_cron_completions
     WHERE queue_name LIKE 'gamification-%'
     ORDER BY last_completed_at DESC;
  4. If logs show the handler completed but worker_cron_completions wasn't updated, the markJobCompleted write failed — check pool connectivity. The helper logs cron_chain: markJobCompleted failed on its own.

never_ran​

The queue has no row in worker_cron_completions at all. Either:

  • Fresh tenant install that hasn't hit its first nightly run yet. Expected for the first 24h; the next 03:00 UTC tick will resolve it.
  • Queue registration regression. Check app/services/worker/src/worker.ts for the registerScheduledJob call for that queue. Without registration, BullMQ never schedules it.

Failure mode: badge-eval blocked by upstream gate​

The badge evaluator calls requireUpstream("perf-hub-snapshot", "gamification-badge-eval") at the top. When the gate trips you'll see in the worker log:

cron_chain: skipping downstream — upstream has not completed for today

The consecutive_composite_above evaluator reads perf_hub_snapshots; without a same-day snapshot the evaluator would idempotently inscribe badge awards against yesterday's data. The skip is intentional — the next nightly run after perf-hub fires catches up.

Investigate the upstream: the perf-hub runbook covers perf-hub-snapshot stale/critical states.

Failure mode: engaged_this_month is 0% but active_rules > 0​

This is the most common "broken but not throwing" failure mode. Workers are running, markers are fresh, but no agents are accruing points.

Diagnosis​

-- For a tenant, show how many rule evaluations each event_type SHOULD produce vs. did.
WITH rules AS (
  SELECT id, event_type, threshold_config_json
    FROM gamification_point_rules
   WHERE tenant_id = $1 AND active = true
),
events AS (
  SELECT event_type, COUNT(*) AS awards
    FROM gamification_point_events
   WHERE tenant_id = $1 AND awarded_at >= date_trunc('month', now())
   GROUP BY event_type
)
SELECT r.event_type, r.threshold_config_json, COALESCE(e.awards, 0) AS awards_this_month
  FROM rules r LEFT JOIN events e USING (event_type);

Look for:

  • Threshold mis-set. min_score_pct: 100 for QA — no agent ever scores exactly 100.
  • No matching source data. A coaching_acknowledged rule with zero coaching_session_acknowledgements rows means coaching hasn't been used yet on this tenant.
  • LOB scope filters everyone out. A rule scoped to a LOB with no active assigned employees. Cross-check via: 
    SELECT cal.name, COUNT(*) AS active_agents
      FROM client_account_lobs cal
      LEFT JOIN employee_profile_history eph
        ON eph.client_lob_id = cal.id
       AND eph.tenant_id = cal.tenant_id
       AND eph.effective_to IS NULL
      LEFT JOIN employees e
        ON e.id = eph.employee_id AND e.status = 'active'
     WHERE cal.tenant_id = $1
     GROUP BY cal.name;

Failure mode: leaderboard cache is stale despite refresh runs​

The refresh worker runs every 4h but only writes a cache row when a gamification_configs row exists for the tenant + LOB. Three checks:

-- 1. Does the tenant have any active configs at all?
SELECT client_lob_id, leaderboard_scope, leaderboard_metric, active
  FROM gamification_configs
 WHERE tenant_id = $1 AND active = true;

-- 2. For configs with scope=CLIENT, is the LOB linked to a client_account?
SELECT cal.client_account_id
  FROM gamification_configs gc
  JOIN client_account_lobs cal ON cal.id = gc.client_lob_id
 WHERE gc.tenant_id = $1 AND gc.leaderboard_scope = 'CLIENT';

-- 3. Are there any opted-out agents skewing the cohort? (Not a bug,
--    but a "where did they go" answer.)
SELECT COUNT(*) FROM gamification_agent_prefs
 WHERE tenant_id = $1 AND leaderboard_opt_out_at IS NOT NULL;

The refresh worker log.warn's tenant-id when its per-config refresh throws — search for that in the worker logs.

Failure mode: anomalies fired​

The status panel surfaces anomalies in an amber card immediately below the coverage stats. HR admins also receive an aggregated in-app notification (gamification.anomaly_detected, routes through the performance category) when anomalies fire — one notification per HR admin per tenant per run, deterministically deduped so a worker re-run is a no-op.

Two anomaly kinds ship today (the points-award worker detects both):

zero_awards_with_active_rules​

The tenant has ≥ 1 active rule but the points-award worker wrote zero gamification_point_events rows for today. This is the more actionable of the two — it tells the operator "rules ran, nothing matched" rather than "we used to be producing output, now we're not."

Common causes:

  • Threshold mis-set on a recently-edited rule. Someone bumped min_score_pct from 80 to 100 in /settings/gamification; no QA evaluation will hit exactly 100.
  • Source data outage. No QA evaluations submitted today (e.g., QA module misconfigured), no training completed, no coaching acknowledged.
  • LOB scope filters everyone out. A rule scoped to a LOB that has no active assigned employees this period.

Diagnosis: Use the "engaged_this_month is 0% but active_rules > 0" diagnostic block above.

points_event_drop​

Today's gamification_point_events row count fell more than 30% below the LATERAL-picked prior date (the most recent date in the last 14 days with non-zero awards). The LATERAL pick is important: a naive "yesterday vs today" would read every Monday as a 100% drop vs Sunday for tenants with no weekend QA traffic.

Common causes:

  • Source-data regression. QA submissions volume dropped because a scorecard was retired without notice. Training module assignments paused for a holiday.
  • One rule deactivated. Someone toggled active=false on the highest-volume rule (often coaching_acknowledged or training_completed_on_time).
  • Workforce volume shift. Half the agents are on a leave window; events follow proportionally.

Diagnosis: Compare per-event-type counts day-over-day to see which rule lost output:

SELECT event_type, awarded_at::date AS day, COUNT(*) AS awards
  FROM gamification_point_events
 WHERE tenant_id = $1
   AND awarded_at::date >= CURRENT_DATE - INTERVAL '7 days'
 GROUP BY event_type, day
 ORDER BY day DESC, awards DESC;

If one event_type cliffs to 0 on a specific day, look at the corresponding source table on that day.

Why anomalies don't abort the run​

The points-award worker writes idempotently — re-running on a clean source-data day overwrites yesterday's findings. Aborting on an anomaly would only deepen the data gap. Detection is observability only.

Recovery: extended-lookback backfill​

For most catch-up scenarios (a single missed nightly run, a few-day worker outage), the next normal nightly tick handles it — the worker's 14-day lookback covers the window automatically via its idempotent UNIQUE.

For longer scenarios, use pnpm backfill:gamification-points. Two real cases:

A tenant just enabled gamification mid-month​

The nightly worker only credits the last 14 days of source events. To extend back to the start of the month (or up to 90 days):

ssh serge@clear "cd /home/serge/frontline2/app && \
  DATABASE_URL=$DATABASE_URL pnpm backfill:gamification-points --lookback-days 60"

The script invokes the same handler the cron uses, with lookbackDays overridden. Idempotent — re-running on the same source window is a no-op.

After a longer outage​

pnpm backfill:gamification-points --lookback-days 30

By default the script suppresses anomaly notifications so a multi-day replay doesn't fan out one "anomaly" notification per day to HR. The anomalies are STILL detected and persisted to worker_cron_completions.last_run_metadata.anomalies — only the L2 in-app fan-out is muted. Pass --alerts if you actually want HR to see them:

pnpm backfill:gamification-points --lookback-days 30 --alerts

Safety cap​

The script caps --lookback-days at 90. Beyond that, ROLLING_90D-mode tenants would see events whose expiry_at is in the past — they'd land in the DB but never count toward an agent's balance.

One-off cron tick for the other workers​

For badge-eval, leaderboard-refresh, or challenge-progress (which have no backfill script — their work is current-state-only), trigger a one-off run by enqueueing a job directly:

ssh serge@clear "docker compose exec api node -e \"
  const { Queue } = require('bullmq');
  const q = new Queue('gamification-badge-eval', {
    connection: { host: 'redis', port: 6379 }
  });
  q.add('manual-recovery', {}).then(() => process.exit(0));
\""

Substitute the queue name for the worker you want to re-run. The handler is idempotent across all four workers (UNIQUE constraints on output tables), so re-running over the same day is safe.

Diagnostic queries cheat sheet​

Top 10 point-event-earning agents this month​

SELECT employee_id, COUNT(*) AS events, SUM(points_awarded) AS points
  FROM gamification_point_events
 WHERE tenant_id = $1
   AND awarded_at >= date_trunc('month', now())
 GROUP BY employee_id
 ORDER BY points DESC
 LIMIT 10;

Per-event-type award rate over last 14 days​

SELECT event_type, awarded_at::date AS day, COUNT(*) AS awards
  FROM gamification_point_events
 WHERE tenant_id = $1
   AND awarded_at >= CURRENT_DATE - INTERVAL '14 days'
 GROUP BY event_type, day
 ORDER BY day DESC, awards DESC;

Active challenges + days remaining​

SELECT id, name, metric, target_value, start_date, end_date,
       (end_date - CURRENT_DATE) AS days_remaining
  FROM gamification_challenges
 WHERE tenant_id = $1 AND status = 'ACTIVE'
 ORDER BY end_date;

Show the chain status all at once​

SELECT queue_name,
       last_completed_at,
       (now() - last_completed_at) AS age,
       last_run_metadata
  FROM worker_cron_completions
 WHERE queue_name LIKE 'gamification-%'
 ORDER BY queue_name;

Escalation​

If a worker is critical for >72h or the engaged_this_month coverage is anomalously low compared to history (the sparkline cliffs), page the on-call. The audit trail is in audit_events (search resource_type = 'gamification_*') and gamification_point_events.awarded_at — both are append-only.

For data-correctness incidents (wrong points awarded, double-credit, missing badges), do not delete rows. The UNIQUE constraints make the worker idempotent; deletions just re-create the wrong state on the next run. Open a spec amendment instead.