feat(telemetry): cutover runbook + the end-to-end gate that de-risks it (CG-14)

The account-level steps of the PostHog cutover are the maintainer's to run, so
this lands the runbook they follow and the check that has to pass first.

The runbook (telemetry-worker/README.md) walks the six steps in the order that
keeps them reversible: Workers Paid → migrate → deploy → watch 24h → verify the
first rollup and the dashboard → only then delete POSTHOG_KEY and cancel the
subscription. Step 3 records the outgoing version id because `wrangler rollback`
is the escape hatch for the whole verification window, and that window is
precisely why the PostHog key is deleted last rather than first.

The new gate (scripts/smoke-cutover.sh, `npm run smoke:cutover`) covers the one
seam nothing else did. Both workers declare the same D1 database_id, so pointing
them at a single --persist-to directory runs the real chain: a client batch →
the ingest worker → D1 → the nightly rollup → the dashboard API reading the
numbers back. Every other suite stops at one link — smoke-ingest at the events
table, smoke-rollup at hand-checked SQL, smoke-api at a hand-written fixture
that the cron never touched. That left the dimension names the rollup WRITES
versus the ones the dashboard READS agreeing by convention across two branches,
where a mismatch is silent: no error, no failed request, just a panel reading
zero forever. 61 assertions, all 13 dimensions, and three deliberate traps — a
ci machine that is active but not a production user, usage_rollup counts that
must be summed rather than tallied, and an uninstall's `targets` that must not
leak into the install-scoped breakdown.

Writing it caught that the activation funnel's denominator is first-seen
machines, not install events (deliberate — a reinstall must not re-enter the
funnel), so the suite now pins that distinction rather than assuming it.

Also rewords the last PostHog reference in dashboard code: a comment justifying
the 14-day retention curve by pointing at a dashboard step 6 deletes. The
reasoning now stands on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

telemetry-dashboard/src/api.ts

@@ -26,7 +26,11 @@ const DAY_MS = 86_400_000;
 const MAX_RANGE_DAYS = 366;
 const DEFAULT_RANGE_DAYS = 30;
 
-/** Retention curve length, matching the PostHog dashboard this replaces. */
+/**
+ * Retention curve length. Two weeks covers the day-1 and day-7 cliffs where nearly
+ * all churn happens, and matches the window the previous analytics dashboard drew,
+ * so the numbers stay comparable across the cutover.
+ */
 const RETENTION_DAYS = 14;
 /** Days a machine gets to run its first index before it counts as churned. */
 const DEFAULT_ACTIVATION_WINDOW = 7;

telemetry-worker/README.md

@@ -117,6 +117,93 @@ secret is `ADMIN_TOKEN`, which enables `POST /admin/rollup`; leave it unset and
 does not exist. Generate one with `openssl rand -hex 32`, and note that rotating it takes
 effect on the next request.
 
+## Cutover from PostHog (one-time)
+
+The replacement of PostHog by this worker's own D1 storage. It is a **hard cutover with no
+backfill** — PostHog history is disposable, and the new charts start from an empty database.
+**Clients are unaffected at every step:** they keep POSTing to `telemetry.getcodegraph.com`
+and every response shape is unchanged, so no client can tell which storage backend is live.
+
+The one-way door is step 6. Everything before it is reversible with `npx wrangler rollback`,
+which is why the PostHog key stays put until the new path has proven itself for a day.
+
+**Before you start:** `npm run smoke:cutover`. It runs the whole chain locally — a client
+batch through the ingest worker into D1, the nightly rollup over it, then the dashboard
+reading the numbers back — and is the only check that covers the seam between the two
+workers. They are separate deployments that agree on a list of dimension names by
+convention alone, and a mismatch there is silent: no error, no failed request, just a panel
+that reads zero forever.
+
+1. **Put the account on Workers Paid (~$5/mo).** Ingest already runs ~97k requests/day
+   against the free plan's 100k/day cap, so this is overdue independently of D1 — and the
+   included D1 quota (5 GB storage, 50M row writes/mo) comes with it. The volume arithmetic
+   is under [Storage](#storage-cloudflare-d1); at ~97k POSTs/day it fits, with the retention
+   window sized to the 10 GB per-database cap.
+
+2. **Bring the production database up to schema.** `codegraph-telemetry`
+   (`5ed36dfb-d2d7-4e35-9e63-a1b99d0b1ed3`) already exists on the account and is bound in
+   `wrangler.jsonc`; this only applies migrations, and is a no-op if it is already current.
+
+   ```bash
+   cd telemetry-worker
+   npm run db:migrate          # remote; bootstraps from empty
+   npm run db:migrations       # confirm 0001_init is listed as applied
+   ```
+
+3. **Deploy, and note the version you are leaving.** Print the deployment list first — the
+   id at the top is your rollback target for the next 24 hours.
+
+   ```bash
+   npx wrangler deployments list      # record the current version id
+   npm run deploy
+   npx wrangler secret put ADMIN_TOKEN   # if not already set; enables manual rollups
+   ```
+
+4. **Watch for 24 hours before trusting it.** The number that matters is the daily ingest
+   rate: it should track the ~95–97k/day PostHog was seeing. A materially lower number means
+   events are being dropped somewhere between the client and the table — a schema or binding
+   mistake, not a real change in usage.
+
+   ```bash
+   npm run db:sql "select count(*) as rows, max(received_at) as newest from events"
+   npm run db:sql "select day, count(*) from events group by day order by day desc limit 3"
+   ```
+
+   `max(received_at)` should be seconds old at any time of day. Watch Workers Logs
+   (`npx wrangler tail`) alongside it for a non-zero error rate — the D1 write is deliberately
+   fail-silent, so a broken write shows up as a log line and a flat row count, never as a
+   failing request.
+
+   **If anything looks wrong, stop here and `npx wrangler rollback [version-id]`.** PostHog is
+   still live and still holds the key, so rolling back restores the old behaviour completely.
+
+5. **Verify the nightly rollup and the dashboard.** After the first 00:30 UTC cron has run,
+   the completed day must be present in the rollup tables — the dashboard reads those, not raw
+   events, so an empty rollup is an empty dashboard even with ingest working perfectly.
+
+   ```bash
+   npm run db:sql "select day, machines, prod_machines from daily_machines order by day desc limit 3"
+   npm run db:sql "select day, event, count from daily_event_counts order by day desc limit 10"
+   ```
+
+   Then open the dashboard (`stats.getcodegraph.com`, see
+   [`../telemetry-dashboard/README.md`](../telemetry-dashboard/README.md)) and confirm the
+   panels render live numbers rather than empty states. If the cron did not fire, roll the day
+   up by hand with `POST /admin/rollup?day=…` above rather than waiting another 24 hours.
+
+6. **Only now, retire PostHog.** Past this point the previous worker version can still be
+   rolled back, but it will have no key to forward with — this is the step that makes the
+   cutover final.
+
+   ```bash
+   npx wrangler secret delete POSTHOG_KEY   # the last PostHog reference on the account
+   npx wrangler secret list                 # confirm ADMIN_TOKEN is the only secret left
+   ```
+
+   The `POSTHOG_HOST` var and every line of forwarding code are already gone from this repo —
+   `git grep -i posthog telemetry-worker/` returns nothing. Finish by cancelling the PostHog
+   subscription and deleting the project.
+
 ## Local dev & checks
 
 ```bash
@@ -126,6 +213,9 @@ npm run dev                  # http://localhost:8787 (local D1 in .wrangler/)
 npm run smoke                # end-to-end: boots `wrangler dev`, POSTs, asserts stored rows
 npm run smoke:rollup         # end-to-end: seeds synthetic days, rolls them up, purges,
                              # asserts every number against hand-computed values
+npm run smoke:cutover        # the whole chain: a client batch → D1 → rollup → the dashboard
+                             # API reads it back. Boots BOTH workers against one shared local
+                             # D1, so it is the only check that covers the seam between them.
 
 curl -i localhost:8787/v1/events -H 'content-type: application/json' -d '{
   "machine_id": "00000000-0000-4000-8000-000000000000",

telemetry-worker/package.json

@@ -9,6 +9,7 @@
     "check": "wrangler types && tsc --noEmit && wrangler deploy --dry-run",
     "smoke": "./scripts/smoke-ingest.sh",
     "smoke:rollup": "./scripts/smoke-rollup.sh",
+    "smoke:cutover": "./scripts/smoke-cutover.sh",
     "db:migrate:local": "wrangler d1 migrations apply codegraph-telemetry --local",
     "db:migrate": "wrangler d1 migrations apply codegraph-telemetry --remote",
     "db:migrations": "wrangler d1 migrations list codegraph-telemetry --remote",

telemetry-worker/scripts/smoke-cutover.sh

@@ -0,0 +1,269 @@
+#!/usr/bin/env bash
+# The cutover gate (CG-14): drives the WHOLE chain the way production will run it —
+# a client POSTs a batch, the ingest worker writes D1, the nightly rollup aggregates,
+# and the dashboard reads the numbers back out.
+#
+# Every other suite tests one link. smoke-ingest.sh stops at the `events` table,
+# smoke-rollup.sh hand-checks the rollup SQL, and smoke-api.sh reads a fixture that
+# was written by hand rather than by the cron. That leaves exactly the seam this
+# cutover turns on unverified: the dimension names the rollup WRITES versus the ones
+# the dashboard READS. Those two lists live in different workers on different
+# branches, and a mismatch is silent — no error, no failed request, just a panel that
+# renders zero forever. Catching that after cutover means a day of lost telemetry;
+# catching it here costs a minute.
+#
+# Both workers declare the same D1 `database_id`, so pointing them at one
+# `--persist-to` directory gives them literally the same local SQLite file. The state
+# is a fresh mktemp each run, so every expected number below is exact rather than a
+# lower bound.
+#
+#   npm run smoke:cutover
+#
+# Expected numbers are derived from THE_BATCH below and nothing else; see the table
+# in that comment block.
+set -uo pipefail
+
+cd "$(dirname "$0")/.."
+WORKER_DIR="$PWD"
+DASH_DIR="$(cd .. && pwd)/telemetry-dashboard"
+
+[ -d "$DASH_DIR" ] || { echo "cannot find telemetry-dashboard/ next to telemetry-worker/"; exit 1; }
+
+INGEST_PORT="${CUTOVER_INGEST_PORT:-8795}"
+DASH_PORT="${CUTOVER_DASH_PORT:-8796}"
+INGEST="http://127.0.0.1:$INGEST_PORT"
+DASH="http://127.0.0.1:$DASH_PORT"
+
+# Test-only credentials. The point is to exercise the wiring, not to keep a secret.
+ADMIN_TOKEN=cutover-admin-token
+DASH_PASSWORD=cutover-dashboard-password
+SESSION_SECRET=cutover-session-secret
+
+STATE="$(mktemp -d -t cg-cutover-state)"
+JAR="$(mktemp -t cg-cutover-jar)"
+ILOG=/tmp/cg-cutover-ingest.log
+DLOG=/tmp/cg-cutover-dash.log
+
+pass=0; fail=0
+ok()  { pass=$((pass + 1)); printf '  ok   %s\n' "$1"; }
+bad() { fail=$((fail + 1)); printf '  FAIL %s — expected %s, got %s\n' "$1" "$2" "$3"; }
+is()  { [ "$2" = "$3" ] && ok "$1" || bad "$1" "$2" "$3"; }
+
+DEV_PID=""
+stop_dev() {
+  [ -n "$DEV_PID" ] || return 0
+  kill "$DEV_PID" 2>/dev/null
+  wait "$DEV_PID" 2>/dev/null
+  DEV_PID=""
+}
+cleanup() { stop_dev; rm -rf "$STATE" "$JAR"; }
+trap cleanup EXIT
+
+# Boot a worker in <dir> on <port> against the SHARED state, wait for <readyurl>.
+boot() { # boot <dir> <port> <readyurl> <log> [extra wrangler args...]
+  local dir="$1" port="$2" ready="$3" log="$4"; shift 4
+  ( cd "$dir" && exec npx wrangler dev --port "$port" --ip 127.0.0.1 \
+      --persist-to "$STATE" "$@" ) >"$log" 2>&1 &
+  DEV_PID=$!
+  for _ in $(seq 1 90); do
+    curl -sf -o /dev/null "$ready" && return 0
+    kill -0 "$DEV_PID" 2>/dev/null || break
+    sleep 1
+  done
+  echo "worker in $dir never came up on :$port — log follows"; cat "$log"; exit 1
+}
+
+# Resolve a dotted path through a JSON document. Numeric segments index arrays.
+jget() {
+  node -e '
+    let v = JSON.parse(process.argv[1]);
+    for (const k of process.argv[2].split(".")) v = v?.[k];
+    console.log(v === undefined ? "<missing>" : typeof v === "object" && v !== null ? JSON.stringify(v) : String(v));
+  ' "$1" "$2"
+}
+
+day_ago() { node -e 'console.log(new Date(Date.now()-process.argv[1]*864e5).toISOString().slice(0,10))' "$1"; }
+
+# Inside the ingest clamp window (30 days) and outside the cron's 3-day lookback.
+DAY="$(day_ago 5)"
+RANGE="from=$DAY&to=$DAY"
+
+# ---------------------------------------------------------------------------
+# THE_BATCH — three machines, one day. Everything asserted below follows from here.
+#
+#   machine  os      arch   node  version  ci     events
+#   m1       darwin  arm64  22    1.5.0    false  install(local/fresh, [claude,cursor])
+#                                                 index([typescript,python], 100-1k, 10-60s)
+#                                                 usage_rollup(codegraph_explore x12, Claude Code)
+#   m2       linux   x64    20    1.5.0    false  install(global/upgrade, [codex])
+#                                                 index([typescript], 1k-10k, 1-5m)
+#                                                 usage_rollup(codegraph_explore x8, Codex CLI)
+#   m3       linux   arm64  22    1.4.1    TRUE   index([go], <100, <10s)
+#                                                 uninstall([claude])
+#
+# The three deliberate traps:
+#   * m3 is ci=true, so it counts as active but NOT as a production user.
+#   * tool_calls must SUM the `count` prop (12 + 8 = 20), not count the 2 rows.
+#   * m3's uninstall carries targets=[claude], so a `target` breakdown that forgets
+#     to scope by event would report claude twice.
+# ---------------------------------------------------------------------------
+M1=11111111-1111-4111-8111-111111111111
+M2=22222222-2222-4222-8222-222222222222
+M3=33333333-3333-4333-8333-333333333333
+
+post_batch() { # post_batch <json>
+  curl -s -o /dev/null -w '%{http_code}' -X POST "$INGEST/v1/events" \
+    -H 'content-type: application/json' --data-binary "$1"
+}
+
+batch() { # batch <machine> <os> <arch> <node> <version> <ci> <events-json>
+  node -e '
+    const [m, os, arch, node_major, v, ci, events, day] = process.argv.slice(1);
+    process.stdout.write(JSON.stringify({
+      machine_id: m, codegraph_version: v, os, arch,
+      node_major: Number(node_major), ci: ci === "true", schema_version: 1,
+      events: JSON.parse(events).map((e) => ({ ...e, ts: `${day}T12:00:00Z` })),
+    }));
+  ' "$@" "$DAY"
+}
+
+# ---------------------------------------------------------------------------
+echo "cutover chain: client → ingest worker → D1 → rollup → dashboard"
+echo
+echo "migrating the shared local D1 state"
+( cd "$WORKER_DIR" && npx wrangler d1 migrations apply codegraph-telemetry \
+    --local --persist-to "$STATE" ) >/tmp/cg-cutover-migrate.log 2>&1 ||
+  { echo "migration failed:"; cat /tmp/cg-cutover-migrate.log; exit 1; }
+
+echo "booting the ingest worker on :$INGEST_PORT"
+boot "$WORKER_DIR" "$INGEST_PORT" "$INGEST/" "$ILOG" --var "ADMIN_TOKEN:$ADMIN_TOKEN"
+
+echo
+echo "ingest accepts the batch"
+is "m1 batch → 204" 204 "$(post_batch "$(batch "$M1" darwin arm64 22 1.5.0 false '[
+  {"event":"install","props":{"scope":"local","kind":"fresh","targets":["claude","cursor"]}},
+  {"event":"index","props":{"languages":["typescript","python"],"file_count_bucket":"100-1k","duration_bucket":"10-60s"}},
+  {"event":"usage_rollup","props":{"kind":"mcp_tool","name":"codegraph_explore","count":12,"client_name":"Claude Code"}}
+]')")"
+is "m2 batch → 204" 204 "$(post_batch "$(batch "$M2" linux x64 20 1.5.0 false '[
+  {"event":"install","props":{"scope":"global","kind":"upgrade","targets":["codex"]}},
+  {"event":"index","props":{"languages":["typescript"],"file_count_bucket":"1k-10k","duration_bucket":"1-5m"}},
+  {"event":"usage_rollup","props":{"kind":"mcp_tool","name":"codegraph_explore","count":8,"client_name":"Codex CLI"}}
+]')")"
+is "m3 (ci) batch → 204" 204 "$(post_batch "$(batch "$M3" linux arm64 22 1.4.1 true '[
+  {"event":"index","props":{"languages":["go"],"file_count_bucket":"<100","duration_bucket":"<10s"}},
+  {"event":"uninstall","props":{"targets":["claude"]}}
+]')")"
+
+sleep 2   # let the ctx.waitUntil writes drain before rolling up
+
+echo
+echo "the nightly rollup aggregates the day"
+ROLL=$(curl -s -X POST -H "x-admin-token: $ADMIN_TOKEN" "$INGEST/admin/rollup?day=$DAY")
+is "POST /admin/rollup → ok" true "$(jget "$ROLL" ok)"
+is "rollup wrote rows" true "$(node -e 'process.stdout.write(String((JSON.parse(process.argv[1]).rows ?? 0) > 0))' "$ROLL")"
+
+stop_dev   # free the D1 lock before the dashboard opens the same file
+
+echo
+echo "booting the dashboard on :$DASH_PORT against the same D1"
+( cd "$DASH_DIR" && npm run --silent vendor ) >/dev/null 2>&1
+boot "$DASH_DIR" "$DASH_PORT" "$DASH/robots.txt" "$DLOG" \
+  --var "ADMIN_PASSWORD:$DASH_PASSWORD" --var "SESSION_SECRET:$SESSION_SECRET"
+
+curl -s -o /dev/null -c "$JAR" -X POST "$DASH/login" --data-urlencode "password=$DASH_PASSWORD"
+api() { curl -s -b "$JAR" "$DASH/api/$1"; }
+is "dashboard session established" 200 "$(curl -s -o /dev/null -w '%{http_code}' -b "$JAR" "$DASH/api/health")"
+
+# --- the big numbers -------------------------------------------------------
+echo
+echo "summary panel reads back what was ingested"
+S=$(api "summary?$RANGE")
+is "production users (ci machine excluded)" 2 "$(jget "$S" production_users)"
+is "active machines"                        3 "$(jget "$S" active_machines)"
+is "new machines"                           3 "$(jget "$S" new_machines)"
+is "installs"                               2 "$(jget "$S" installs)"
+is "uninstalls"                             1 "$(jget "$S" uninstalls)"
+is "indexing runs"                          3 "$(jget "$S" index_runs)"
+is "tool calls SUM the count prop (12+8)"  20 "$(jget "$S" tool_calls)"
+
+# --- every dimension the dashboard offers ----------------------------------
+# This is the actual point of the suite: each of these is a distinct string that
+# must match between rollup.ts and api.ts's DIMS registry. An empty `labels` means
+# the dashboard is asking for a dimension the cron never writes.
+echo
+echo "every breakdown dimension resolves against the cron's output"
+bd() { # bd <desc> <query> <expected-labels-json> <expected-data-json>
+  local body; body=$(api "breakdown?$RANGE&$2")
+  is "$1 — labels" "$3" "$(jget "$body" labels)"
+  is "$1 — data"   "$4" "$(jget "$body" datasets.0.data)"
+}
+bd "os"                "dim=os"                '["linux","darwin"]'          '[2,1]'
+bd "arch"              "dim=arch"              '["arm64","x64"]'             '[2,1]'
+bd "version"           "dim=codegraph_version" '["1.5.0","1.4.1"]'           '[2,1]'
+bd "node major"        "dim=node_major"        '["22","20"]'                 '[2,1]'
+bd "language"          "dim=language"          '["typescript","go","python"]' '[2,1,1]'
+bd "files in project"  "dim=file_count_bucket" '["<100","100-1k","1k-10k","10k+"]' '[1,1,1,0]'
+bd "run length"        "dim=duration_bucket"   '["<10s","10-60s","1-5m","5m+"]'    '[1,1,1,0]'
+bd "install scope"     "dim=scope"             '["global","local"]'          '[1,1]'
+bd "install kind"      "dim=kind"              '["fresh","upgrade"]'         '[1,1]'
+bd "tool name"         "dim=name"              '["codegraph_explore"]'       '[20]'
+bd "agent"             "dim=client_name"       '["Claude Code","Codex CLI"]' '[12,8]'
+
+# The trap: `target` defaults to event=install, so the uninstall's own claude target
+# must NOT be folded in — and must still be reachable by asking for it explicitly.
+bd "agent target (install-scoped)" "dim=target" '["claude","codex","cursor"]' '[1,1,1]'
+bd "agent target (uninstall)" "dim=target&event=uninstall" '["claude"]' '[1]'
+
+# --- the remaining panels --------------------------------------------------
+echo
+echo "the timeseries and funnel panels see the day"
+# Every entry in api.ts's SERIES registry — each one reads a different rollup table,
+# so this is the second half of the write-vs-read seam the breakdowns cover above.
+ts() { # ts <desc> <metric> <series-0> [series-1]
+  local body; body=$(api "timeseries?$RANGE&metric=$2")
+  is "$1 — day"      "[\"$DAY\"]" "$(jget "$body" labels)"
+  is "$1 — series"   "$3"         "$(jget "$body" datasets.0.data)"
+  [ $# -ge 4 ] && is "$1 — second series" "$4" "$(jget "$body" datasets.1.data)"
+}
+ts "installs and uninstalls" installs_uninstalls '[2]' '[1]'
+ts "new installs"            new_installs        '[3]'
+ts "production users"        production_users    '[2]'
+ts "indexing activity"       indexing_activity   '[3]' '[3]'
+ts "tool calls (sums the prop)" tool_calls       '[20]' '[2]'
+
+MET=$(api "meta")
+is "meta anchors on the rolled-up day" "$DAY" "$(jget "$MET" latest_day)"
+is "meta reports the rollup ran"       "$DAY" "$(jget "$MET" latest_rollup_day)"
+
+# The funnel is the one panel that reads RAW events rather than a rollup, so it is
+# also the one the retention purge can blind — worth pinning that it works today.
+#
+# Its denominator is FIRST-SEEN MACHINES, not `install` events (api.ts: "a machine
+# that reinstalls does not re-enter the funnel"). m3 is the discriminator: it never
+# sent an install event, but it is new and it indexed, so it belongs in both legs.
+# Reading 2 here would mean the funnel had quietly become an install-event ratio.
+ACT=$(api "activation?$RANGE&window=1")
+is "funnel counts new machines, not install events" 3 "$(jget "$ACT" installs)"
+is "all three indexed within the window"            3 "$(jget "$ACT" activated)"
+is "nobody dropped out"                             0 "$(jget "$ACT" dropped)"
+is "raw-event floor is reported to the caller" "$DAY" "$(jget "$ACT" raw_events_from)"
+
+is "retention endpoint answers" 200 \
+   "$(curl -s -o /dev/null -w '%{http_code}' -b "$JAR" "$DASH/api/retention?$RANGE")"
+
+# --- the guarantee the cutover is selling ----------------------------------
+echo
+echo "the no-third-party guarantee still holds"
+is "ingest worker makes no outbound fetch" 0 \
+   "$(grep -E 'fetch\(' "$WORKER_DIR"/src/*.ts | grep -vc 'async fetch(request' || true)"
+is "no PostHog reference in ingest worker code or config" 0 \
+   "$(grep -ril posthog "$WORKER_DIR"/src "$WORKER_DIR"/wrangler.jsonc 2>/dev/null | wc -l | tr -d ' ')"
+
+echo
+if [ "$fail" -eq 0 ]; then
+  echo "$pass passed, 0 failed — the chain is whole; safe to cut over"
+else
+  echo "$pass passed, $fail failed"
+fi
+[ "$fail" -eq 0 ]