# Generated-file detection — path convention plus content banner

CG-5, issue #1500. Companion to `explore-budget-allocation.md` (CG-4's instrument) and a
prerequisite for the scoring overhaul (CG-10).

## The problem

`isGeneratedFile` was path-only. It matches the `<basename>.<tool>.<ext>` convention —
`.pb.go`, `_grpc.pb.go`, `.g.dart`, `_pb2.py` — which is where most codegen output lives,
and which was enough for the cosmos-sdk audit that motivated it.

It is not enough for Go. **Go's convention is a content marker, not a filename one:**

```go
// Code generated by <tool>. DO NOT EDIT.
```

codified by `go generate`, honored by gofmt, golangci-lint and GitHub linguist, and emitted
by protoc-gen-go, mockgen, sqlc, ent, wire, stringer — and by in-house generators. #1500 is
exactly this: a Go monorepo with generated FKIT CRUD in ordinarily-named files
(`payroll.go`) sitting beside hand-written workflow use-cases. Nothing in the path gives it
away, so every generated-file down-rank in the codebase was a no-op on it.

### How large the gap is

Measured on a shallow clone of `kubernetes/client-go` (2,453 Go files):

| signal | files flagged |
|---|---|
| ground truth (grep the canonical banner in the first 60 lines) | 2,001 |
| path convention (`isGeneratedFile`) | **0** |
| content banner (`hasGeneratedHeader`) | **2,001** — 0 false positives, 0 misses |

82% of that repository is generated code with ordinary filenames, and the path check saw
none of it. This is not a long-tail case.

## Design

**Decide at index time, store on the file record, read from the DB.** Explore must never
read file headers per request.

- `isGeneratedFile(path)` — unchanged. Path-only, pure, synchronous, free to call in a sort
  comparator. Kept for callers with no database in hand.
- `hasGeneratedHeader(content)` — the content signal (below).
- `detectGeneratedFile(path, content)` — the union, which is what the indexer persists.
- `files.generated INTEGER NOT NULL DEFAULT 0` (schema v9) + a **partial** index
  `idx_files_generated ON files(path) WHERE generated = 1`, so lookups cost the generated
  minority, not the repo.
- `QueryBuilder.generatedPredicateFor(paths)` / `CodeGraph.generatedFilePredicate(paths)` —
  one bounded probe up front, `O(1)` per comparison after, unioned with the path check.

### Why a bounded lookup and not a cached set

Every consumer already holds a short candidate list — a ranked file group, an FTS result
page, a `LIMIT 20` aggregate. Intersecting that list against the partial index needs no
whole-repo set to materialize and, more importantly, **no cache to invalidate**: a ranking
call can never serve a verdict the last sync already replaced. The alternative (a lazily
materialized `Set` of all generated paths) has to be invalidated on every file write and
goes stale on the read-only pool workers, in exchange for saving a sub-millisecond query.

### Precision over recall

A false positive silently demotes hand-written code in every ranking path, so the marker
table is precision-first and the scan is fenced three ways:

1. **Header window only** — first 8,192 chars / 60 lines. Generous enough for build tags
   plus an Apache-2.0 preamble above the banner; tight enough that a generator's own source,
   which holds the banner as a *string constant in its body*, is not flagged.
2. **Comment lines only** — the marker must sit on a line with a comment leader (`//`, `#`,
   `--`, `<!--`, `%`, `;`, `'`, …) or inside an open block comment (`/* */`, `<!-- -->`,
   `"""`, `'''`, `=begin`, `<# #>`), tracked with a small state machine over the window.
   Generators always emit banners as comments; requiring it rules out identifiers and
   string literals that merely contain the words.
3. **Tight markers** — `automatically generated` alone is prose ("the table is automatically
   generated at runtime"); `automatically generated **by**` is a banner. `DO NOT EDIT` alone
   is a style directive; paired with a generation claim it is a banner.

The module deliberately keeps its own quoted banner literals **below** the header window so
it does not classify itself; `generated-detection.test.ts` pins that, so moving the pattern
table upward fails a test rather than silently demoting this file.

### Migration: no backfill, by necessity

v9 is DDL only. The flag derives from file **content**, which the migration cannot see —
`files` stores a hash, not bytes. Migrated rows stay 0 until a re-index, and because every
reader unions the flag with the path check, an un-backfilled database keeps exactly the
pre-#1500 behavior instead of regressing. `sync` heals it file-by-file as files change.
This is why the CHANGELOG entry says a re-index is required.

## Cost

The acceptance bar was "no measurable index-time cost regression."

A single unanchored `/generat/i` test over the header rejects ~every hand-written file
before any line splitting happens. `String.prototype.slice` on a long string yields a V8
sliced view, not a copy, so the fast path allocates nothing.

- **Microbenchmark** (`detectGeneratedFile` over a whole corpus, 5 passes):
  4.6 µs/file on client-go (2,453 files, 14.2 MB, 82% generated — the worst case, where the
  gate passes and the full line scan runs), 7.3 µs/file on this repo's `src`.
- **End-to-end** `codegraph init` on client-go, n=3 alternating arms
  (current build vs. the same build with the content scan stubbed out):

  | arm | runs (s) | median |
  |---|---|---|
  | with content detection | 5.66, 5.73, 5.89 | **5.73** |
  | path-only baseline | 5.52, 5.76, 5.88 | **5.76** |

  The arms cross over between runs — the difference is inside run-to-run noise.

## What this task does NOT change

Generated status remains a **stable tiebreak at equal score**, exactly where it was
(`src/mcp/tools.ts` file sort, `findSymbolMatches`, `findAllSymbols`, search formatting,
`getDominantFile`/`getTopRouteFile`/`getRoutingManifest`, the context formatter). A
generated file with a higher raw score still outranks a hand-written one. Turning generated
status into a **strong negative signal** is CG-10, which this task unblocks by making the
signal correct and available.

Verified end-to-end on a two-file Go package where a generated `payroll.go` and a
hand-written `workflow.go` both define `ProcessPayroll`: with the flag set the hand-written
file ranks first; clearing the flag in the same index (i.e. pre-#1500 behavior) puts the
generated file first.
