codegraph/scripts/agent-eval/allocation-fixtures.json

{
  "$comment": [
    "Regression fixtures for GitHub issue #1500 / epic CG-1 — relevance-proportional",
    "explore budget allocation. Run them with `node scripts/agent-eval/probe-allocation.mjs`",
    "against a built dist/.",
    "",
    "STATUS: CG-10 (relevance scoring) closed the RANKING half of both fixtures — nothing",
    "incidental reaches the envelope any more. What still fails is the BYTE SPLIT among the",
    "files that correctly ranked in, because the render loop spends by file size: a small",
    "weakly-relevant file ships whole while the strongly-relevant one is clipped at",
    "maxCharsPerFile. That is CG-12's gate. Each fixture's `baseline` records the pre-CG-10",
    "numbers; `afterCG10` records where it stands now.",
    "",
    "`groups` partitions the files explore rendered into `answer` (what the query is",
    "actually about) and `incidental` (what wins the envelope today on name collisions).",
    "Assertions are on the DELIVERED envelope unless suffixed `Allocated`; delivered is",
    "what the agent got, allocated is what the render loop chose before the hard ceiling.",
    "Shares are fractions of the whole response, meta-text included, so they never sum to 1."
  ],
  "fixtures": [
    {
      "id": "payroll-go",
      "title": "#1500 — generated Go CRUD beside a hand-written payroll workflow",
      "kind": "fixture",
      "path": "__tests__/fixtures/payroll-go",
      "query": "how does payroll cycle create and calculate payslips?",
      "rationale": [
        "The reporter's repo shape: a Go service whose generated FKIT CRUD layer sits",
        "beside the hand-written use-case that does the real work. The query deliberately",
        "does NOT name runPayrollCycleAll / BuildPayslip / Upsert — an architecture",
        "question phrased the way a newcomer would phrase it. The generated layer",
        "name-collides on every query term (CreatePayslip, PayrollCycleCreateRequest,",
        "CalculatePayrollCycleTotals, a second BuildPayslip, a second Upsert), so a",
        "scorer that rewards incidental name matches surfaces the CRUD path.",
        "Half the generated files carry ORDINARY names and are only detectable by their",
        "`// Code generated ... DO NOT EDIT.` header (CG-5), which is what makes this",
        "the #1500 case rather than a .pb.go case."
      ],
      "groups": {
        "answer": [
          "internal/usecase/**",
          "internal/store/**",
          "internal/transport/**",
          "internal/domain/**",
          "cmd/**"
        ],
        "incidental": ["internal/gen/**"]
      },
      "assert": {
        "answerShareAtLeast": 0.55,
        "incidentalShareAtMost": 0.25,
        "topFileGroup": "answer",
        "mustDeliverBytes": [
          "internal/usecase/payroll/cycle.go",
          "internal/usecase/payroll/payslip_builder.go"
        ],
        "$mustContainComment": "Needles are chosen to match the HAND-WRITTEN chain only — a bare `BuildPayslip`/`Upsert` also matches the generated collisions, which is the whole point of the fixture.",
        "mustContain": [
          "runPayrollCycleAll",
          "func (s *Service) BuildPayslip",
          "s.store.Upsert(ctx, slip)"
        ]
      },
      "baseline": {
        "measuredOn": "2026-08-03",
        "note": "19 files → very-tiny tier (13,000 budget); 23,020 chars allocated against it, cut to 16,011 by the 19,500 hard ceiling.",
        "delivered": {
          "internal/gen/fkit/payroll/payslip.go": 0.307,
          "internal/gen/fkit/payroll/payroll_cycle.go": 0.266,
          "internal/domain/payroll/payslip.go": 0.256,
          "internal/usecase/payroll/cycle.go": 0.0
        },
        "verdict": "The workflow file is allocated the single largest slice (30.6%) and delivers ZERO — the hard ceiling drops its whole section. Generated CRUD takes 57.4% of what the agent actually receives; runPayrollCycleAll, BuildPayslip and the real Upsert never reach the response."
      },
      "afterCG10": {
        "measuredOn": "2026-08-04",
        "delivered": {
          "internal/usecase/payroll/cycle.go": 0.389,
          "internal/gen/fkit/payroll/payroll_cycle.go": 0.235,
          "internal/domain/payroll/payslip.go": 0.226,
          "internal/gen/fkit/payroll/payslip.go": 0.0
        },
        "verdict": "PASSES answerShareAtLeast (61.5%), incidentalShareAtMost (23.5%), topFileGroup, and the cycle.go + runPayrollCycleAll + real-Upsert needles. The generated files now rank #3/#4 instead of #1/#2 — kind weighting plus a 0.3x generated penalty on BOTH the score and the graph mass, which is the key the comparator sorts on. STILL FAILING for CG-12: payslip_builder.go ranks #6 against the tier's maxFiles of 4, so `func (s *Service) BuildPayslip` never renders."
      }
    },
    {
      "id": "self-query",
      "title": "This repo — incidental `explore`/`BUDGET` matches in the agent-eval scripts",
      "kind": "self",
      "path": ".",
      "query": "how does explore allocate its output budget across files",
      "rationale": [
        "The same failure mode with no generated code in sight. `scripts/agent-eval/*.mjs`",
        "mention `explore` and `BUDGET` incidentally — they are eval harnesses, not the",
        "allocator — and they are small enough to ship WHOLE, while src/mcp/tools.ts (which",
        "carries getExploreOutputBudget and the render loop, and scores 4x higher on every",
        "signal) is large enough to be clipped at maxCharsPerFile. Allocation follows file",
        "size, not relevance.",
        "",
        "Unlike payroll-go this fixture reads THIS repo's live index, so its exact numbers",
        "move as the repo changes (indexed file count crossing 500 flips the budget tier).",
        "The assertions are therefore relative — answer-vs-incidental, not fixed percentages."
      ],
      "groups": {
        "answer": ["src/mcp/**"],
        "incidental": ["scripts/**"]
      },
      "assert": {
        "answerShareAtLeast": 0.5,
        "incidentalShareAtMost": 0.25,
        "topFileGroup": "answer",
        "mustDeliverBytes": ["src/mcp/tools.ts"]
      },
      "baseline": {
        "measuredOn": "2026-08-03",
        "note": "493 files → small tier (18,000 budget); 27,518 chars allocated against it, cut to 19,749 by the 25,000 hard ceiling.",
        "delivered": {
          "scripts/agent-eval/offload-eval-hook.mjs": 0.25,
          "scripts/agent-eval/offload-eval-metrics.mjs": 0.236,
          "scripts/agent-eval/parse-session.mjs": 0.232,
          "src/mcp/tools.ts": 0.185,
          "scripts/agent-eval/offload-eval-cost.mjs": 0.0
        },
        "verdict": "The script corpus takes 71.8% of the delivered envelope (79.4% of what was allocated) against tools.ts's 18.5%, despite tools.ts scoring 46 vs 10, carrying 2.3x the graph mass and 3x the distinct term hits."
      },
      "afterCG10": {
        "measuredOn": "2026-08-04",
        "delivered": {
          "src/resolution/memory-budget.ts": 0.512,
          "src/mcp/tools.ts": 0.329
        },
        "verdict": "PASSES incidentalShareAtMost: every scripts/agent-eval/*.mjs file is gone (0.0%), which is CG-10's acceptance bar — their sole match was an unused file-scope `explore`/`BUDGET` constant, worth 0.08x weight, and the relative floor (8.2) then cut them. tools.ts ranks #1. STILL FAILING for CG-12: memory-budget.ts scores 18 to tools.ts's 41 yet takes 51.2% of the envelope to tools.ts's 32.9%, purely because it is small enough to ship whole while tools.ts is clipped at maxCharsPerFile. Allocation still follows file size, not relevance."
      }
    }
  ]
}