# Generated-file detection — path convention plus content banner CG-5, issue #1500. Companion to `explore-budget-allocation.md` (CG-4's instrument) and a prerequisite for the scoring overhaul (CG-10). ## The problem `isGeneratedFile` was path-only. It matches the `..` convention — `.pb.go`, `_grpc.pb.go`, `.g.dart`, `_pb2.py` — which is where most codegen output lives, and which was enough for the cosmos-sdk audit that motivated it. It is not enough for Go. **Go's convention is a content marker, not a filename one:** ```go // Code generated by . DO NOT EDIT. ``` codified by `go generate`, honored by gofmt, golangci-lint and GitHub linguist, and emitted by protoc-gen-go, mockgen, sqlc, ent, wire, stringer — and by in-house generators. #1500 is exactly this: a Go monorepo with generated FKIT CRUD in ordinarily-named files (`payroll.go`) sitting beside hand-written workflow use-cases. Nothing in the path gives it away, so every generated-file down-rank in the codebase was a no-op on it. ### How large the gap is Measured on a shallow clone of `kubernetes/client-go` (2,453 Go files): | signal | files flagged | |---|---| | ground truth (grep the canonical banner in the first 60 lines) | 2,001 | | path convention (`isGeneratedFile`) | **0** | | content banner (`hasGeneratedHeader`) | **2,001** — 0 false positives, 0 misses | 82% of that repository is generated code with ordinary filenames, and the path check saw none of it. This is not a long-tail case. ## Design **Decide at index time, store on the file record, read from the DB.** Explore must never read file headers per request. - `isGeneratedFile(path)` — unchanged. Path-only, pure, synchronous, free to call in a sort comparator. Kept for callers with no database in hand. - `hasGeneratedHeader(content)` — the content signal (below). - `detectGeneratedFile(path, content)` — the union, which is what the indexer persists. - `files.generated INTEGER NOT NULL DEFAULT 0` (schema v9) + a **partial** index `idx_files_generated ON files(path) WHERE generated = 1`, so lookups cost the generated minority, not the repo. - `QueryBuilder.generatedPredicateFor(paths)` / `CodeGraph.generatedFilePredicate(paths)` — one bounded probe up front, `O(1)` per comparison after, unioned with the path check. ### Why a bounded lookup and not a cached set Every consumer already holds a short candidate list — a ranked file group, an FTS result page, a `LIMIT 20` aggregate. Intersecting that list against the partial index needs no whole-repo set to materialize and, more importantly, **no cache to invalidate**: a ranking call can never serve a verdict the last sync already replaced. The alternative (a lazily materialized `Set` of all generated paths) has to be invalidated on every file write and goes stale on the read-only pool workers, in exchange for saving a sub-millisecond query. ### Precision over recall A false positive silently demotes hand-written code in every ranking path, so the marker table is precision-first and the scan is fenced three ways: 1. **Header window only** — first 8,192 chars / 60 lines. Generous enough for build tags plus an Apache-2.0 preamble above the banner; tight enough that a generator's own source, which holds the banner as a *string constant in its body*, is not flagged. 2. **Comment lines only** — the marker must sit on a line with a comment leader (`//`, `#`, `--`, ``, `"""`, `'''`, `=begin`, `<# #>`), tracked with a small state machine over the window. Generators always emit banners as comments; requiring it rules out identifiers and string literals that merely contain the words. 3. **Tight markers** — `automatically generated` alone is prose ("the table is automatically generated at runtime"); `automatically generated **by**` is a banner. `DO NOT EDIT` alone is a style directive; paired with a generation claim it is a banner. The module deliberately keeps its own quoted banner literals **below** the header window so it does not classify itself; `generated-detection.test.ts` pins that, so moving the pattern table upward fails a test rather than silently demoting this file. ### Migration: no backfill, by necessity v9 is DDL only. The flag derives from file **content**, which the migration cannot see — `files` stores a hash, not bytes. Migrated rows stay 0 until a re-index, and because every reader unions the flag with the path check, an un-backfilled database keeps exactly the pre-#1500 behavior instead of regressing. `sync` heals it file-by-file as files change. This is why the CHANGELOG entry says a re-index is required. ## Cost The acceptance bar was "no measurable index-time cost regression." A single unanchored `/generat/i` test over the header rejects ~every hand-written file before any line splitting happens. `String.prototype.slice` on a long string yields a V8 sliced view, not a copy, so the fast path allocates nothing. - **Microbenchmark** (`detectGeneratedFile` over a whole corpus, 5 passes): 4.6 µs/file on client-go (2,453 files, 14.2 MB, 82% generated — the worst case, where the gate passes and the full line scan runs), 7.3 µs/file on this repo's `src`. - **End-to-end** `codegraph init` on client-go, n=3 alternating arms (current build vs. the same build with the content scan stubbed out): | arm | runs (s) | median | |---|---|---| | with content detection | 5.66, 5.73, 5.89 | **5.73** | | path-only baseline | 5.52, 5.76, 5.88 | **5.76** | The arms cross over between runs — the difference is inside run-to-run noise. ## What this task does NOT change Generated status remains a **stable tiebreak at equal score**, exactly where it was (`src/mcp/tools.ts` file sort, `findSymbolMatches`, `findAllSymbols`, search formatting, `getDominantFile`/`getTopRouteFile`/`getRoutingManifest`, the context formatter). A generated file with a higher raw score still outranks a hand-written one. Turning generated status into a **strong negative signal** is CG-10, which this task unblocks by making the signal correct and available. Verified end-to-end on a two-file Go package where a generated `payroll.go` and a hand-written `workflow.go` both define `ProcessPayroll`: with the flag set the hand-written file ranks first; clearing the flag in the same index (i.e. pre-#1500 behavior) puts the generated file first.