|
|
@@ -425,6 +425,215 @@ const SCORE_FLOOR_MAX = 10;
|
|
|
*/
|
|
|
const SCORE_FLOOR_KEEP_MIN = 3;
|
|
|
|
|
|
+// ── Score-proportional byte allocation (CG-12 / #1500) ─────────────────────
|
|
|
+//
|
|
|
+// The score floor above decides WHICH files reach the response. This decides how
|
|
|
+// the byte envelope is SPLIT among them — and until this existed, it wasn't
|
|
|
+// really decided at all: every admitted file was capped at the same
|
|
|
+// `maxCharsPerFile`, and the whole-file rule handed anything under
|
|
|
+// `maxCharsPerFile * 3` its entire contents. So allocation followed FILE SIZE,
|
|
|
+// not relevance. On this repo's own "how does explore allocate its output budget
|
|
|
+// across files", `src/mcp/tools.ts` (score 41, 4x the graph mass, 3x the distinct
|
|
|
+// term hits — it literally holds the allocator) was clipped at 3,800 while a
|
|
|
+// score-18 file shipped whole at 5,672 and took 51% of the envelope, purely for
|
|
|
+// being small. On the #1500 Go fixture, two generated CRUD files shipped whole at
|
|
|
+// ~4.5K each and consumed the tier's 4 file slots, so `BuildPayslip` — the
|
|
|
+// hand-written half of "create and calculate payslips" — never appeared at all.
|
|
|
+//
|
|
|
+// The replacement: reserve each file a share of the envelope proportional to what
|
|
|
+// it is worth, up front, before anything renders. Three consequences:
|
|
|
+//
|
|
|
+// 1. A reservation is a GUARANTEE, not a race. The old loop spent the envelope
|
|
|
+// first-come-first-served in rank order, so the top two files could exhaust
|
|
|
+// it and every later file hit a `budget-90pct` skip regardless of merit.
|
|
|
+// 2. A file below the cliff gets ZERO source — its path, symbols and line
|
|
|
+// numbers only. It costs ~100 chars instead of ~4,500, and (crucially) it
|
|
|
+// does not consume a `maxFiles` slot, so the slot goes to a file that earns
|
|
|
+// its bytes. This is the concentration lever.
|
|
|
+// 3. The per-file cap stops being the primary guard. It survives only as
|
|
|
+// `ALLOC_MAX_SHARE`, a safety valve against a single god-file — which the
|
|
|
+// proportional split already bounds, since a file's share can't exceed its
|
|
|
+// weight share.
|
|
|
+const EXPLORE_ALLOCATION = {
|
|
|
+ /**
|
|
|
+ * A file whose weight is under this fraction of the top file's gets no source.
|
|
|
+ *
|
|
|
+ * Calibrated between the two shapes the fixtures pin: the #1500 generated CRUD
|
|
|
+ * lands at 10–11% of the top weight (penalised twice — once into the score by
|
|
|
+ * `rankPenalty`, once again here) and must cliff; a genuinely peripheral but
|
|
|
+ * hand-written flow file — `payslip_builder.go`, the direct callee of the
|
|
|
+ * workflow entry — lands at 25% and must NOT. Everything in between is a
|
|
|
+ * judgement call the agent can undo for ~0 cost, because a cliffed file is
|
|
|
+ * still NAMED in the response and one follow-up explore fetches it.
|
|
|
+ */
|
|
|
+ CLIFF_FRACTION: 0.15,
|
|
|
+ /**
|
|
|
+ * Ceiling on the cliff, in the same units as `SCORE_FLOOR_MAX` — and for the
|
|
|
+ * same reason. A file whose weight clears a full-strength direct match is never
|
|
|
+ * incidental, so no amount of concentration elsewhere may zero it: one
|
|
|
+ * overwhelming top file (a 99-scoring god-file among score-10 peers) otherwise
|
|
|
+ * puts the cliff at 14.9 and silences every peer the score floor had just
|
|
|
+ * deliberately admitted. The cliff is a RELATIVE prune of weak evidence, not a
|
|
|
+ * second admission gate — the score floor already owns admission.
|
|
|
+ */
|
|
|
+ CLIFF_MAX: SCORE_FLOOR_MAX,
|
|
|
+ /**
|
|
|
+ * Floor on a useful reservation — every admitted file gets this much before
|
|
|
+ * the proportional split divides the rest. Under it a slice can't hold one
|
|
|
+ * complete method, and a fragment is strictly worse than a pointer: it forces
|
|
|
+ * the Read this tool exists to prevent.
|
|
|
+ *
|
|
|
+ * It is a FLOOR, not a second cliff. Cliffing the starved file instead
|
|
|
+ * cascades: removing the smallest raises everyone else's share by so little
|
|
|
+ * that the next-smallest starves too, and a query with two dominant files ate
|
|
|
+ * six legitimately-ranked peers one at a time. Concentration is the relative
|
|
|
+ * cliff's job; this only keeps a served file's slice usable.
|
|
|
+ */
|
|
|
+ MIN_CHARS: 700,
|
|
|
+ /**
|
|
|
+ * Safety valve, as a fraction of the envelope. Not the primary guard any more —
|
|
|
+ * the proportional split is — so this only has to stop a pathological
|
|
|
+ * single-file response.
|
|
|
+ */
|
|
|
+ MAX_SHARE: 0.7,
|
|
|
+ /**
|
|
|
+ * Markdown overhead charged per rendered file (header + fences + blank lines),
|
|
|
+ * matching the render loop's own `+ 200` accounting. Held out of the pool
|
|
|
+ * before the split so the reservations plus their overhead fit the envelope —
|
|
|
+ * without this the last file's reservation is always the one that doesn't fit.
|
|
|
+ */
|
|
|
+ FILE_OVERHEAD: 200,
|
|
|
+ /**
|
|
|
+ * Flow-spine files are weighted up and are exempt from the cliff. Clipping the
|
|
|
+ * spine causes the Read fallback (it IS the answer to a flow question);
|
|
|
+ * clipping a peripheral file does not. This makes the existing advisory spine
|
|
|
+ * handling — `hasSpine`, `SPINE_CEILING` — strict at the allocation layer.
|
|
|
+ */
|
|
|
+ SPINE_WEIGHT_BOOST: 2,
|
|
|
+ /**
|
|
|
+ * Slack allowed on the whole-file rule: a file a little over its reservation
|
|
|
+ * still ships WHOLE rather than as clusters, because slicing off that last
|
|
|
+ * sliver saves ~1% of the envelope and costs a Read — the trade the whole-file
|
|
|
+ * rule exists to refuse. Proportional (with an absolute ceiling) because a
|
|
|
+ * "sliver" is relative: a flat 800 is 15% of a 5K reservation but 31% of a 2.5K
|
|
|
+ * one, and at the small end that overshoot is exactly what the file below then
|
|
|
+ * loses.
|
|
|
+ */
|
|
|
+ WHOLE_FILE_GRACE_FRACTION: 0.15,
|
|
|
+ WHOLE_FILE_GRACE_MAX: 800,
|
|
|
+} as const;
|
|
|
+
|
|
|
+/** One candidate file's allocation inputs, in final rank order. */
|
|
|
+export interface ExploreAllocationCandidate {
|
|
|
+ path: string;
|
|
|
+ /** Post-`rankPenalty` relevance score from the ranking pass. */
|
|
|
+ score: number;
|
|
|
+ /**
|
|
|
+ * How much this file's BYTES are worth, independent of how well it matched.
|
|
|
+ * Ranking answers "is this file about the query"; allocation answers "will
|
|
|
+ * these bytes teach the agent anything". Generated CRUD can legitimately rank
|
|
|
+ * (it name-collides on every domain word) while its bytes stay mechanical
|
|
|
+ * boilerplate the agent gains nothing from reading — so `rankPenalty` is
|
|
|
+ * applied a SECOND time here. That is what finally sinks the #1500 generated
|
|
|
+ * layer below the cliff: it survived CG-10's single penalty because the sort's
|
|
|
+ * leading keys (entry-point, graph mass) are structural, and a big densely
|
|
|
+ * self-referential generated file scores well on both.
|
|
|
+ */
|
|
|
+ worth: number;
|
|
|
+ /** Carries a symbol on the rendered flow spine. */
|
|
|
+ spine: boolean;
|
|
|
+}
|
|
|
+
|
|
|
+export interface ExploreAllocation {
|
|
|
+ /** path → chars of source it may render. Only holds admitted files. */
|
|
|
+ allowances: Map<string, number>;
|
|
|
+ /** Files the cliff zeroed, in rank order — pointers, not bytes. */
|
|
|
+ cliffed: string[];
|
|
|
+ /** The weight threshold the cliff fired at (0 when nothing was cliffed). */
|
|
|
+ cliffAt: number;
|
|
|
+ /** Chars actually split among the admitted files. */
|
|
|
+ pool: number;
|
|
|
+}
|
|
|
+
|
|
|
+/**
|
|
|
+ * Split `budget.maxOutputChars` across ranked candidates in proportion to
|
|
|
+ * relevance, with a hard relative cliff.
|
|
|
+ *
|
|
|
+ * `candidates` must arrive in FINAL RANK ORDER — `maxFiles` is applied to the
|
|
|
+ * survivors of the cliff, in that order, so cliffing genuinely hands a slot to
|
|
|
+ * the next file down rather than leaving it unused.
|
|
|
+ *
|
|
|
+ * Tier invariant (`getExploreOutputBudget`): a larger tier must never allow less
|
|
|
+ * per file than a smaller one. It holds here by construction — every bound is a
|
|
|
+ * fraction of `maxOutputChars` or of `maxCharsPerFile`, both monotonic across
|
|
|
+ * tiers — except `MIN_CHARS`, which is an absolute floor and so identical at
|
|
|
+ * every tier.
|
|
|
+ */
|
|
|
+export function allocateExploreBudget(
|
|
|
+ candidates: readonly ExploreAllocationCandidate[],
|
|
|
+ budget: ExploreOutputBudget,
|
|
|
+ maxFiles: number,
|
|
|
+): ExploreAllocation {
|
|
|
+ const A = EXPLORE_ALLOCATION;
|
|
|
+ const empty: ExploreAllocation = { allowances: new Map(), cliffed: [], cliffAt: 0, pool: 0 };
|
|
|
+ if (candidates.length === 0) return empty;
|
|
|
+
|
|
|
+ const weightOf = (c: ExploreAllocationCandidate) =>
|
|
|
+ Math.max(0, c.score) * Math.max(0, Math.min(1, c.worth)) * (c.spine ? A.SPINE_WEIGHT_BOOST : 1);
|
|
|
+
|
|
|
+ const weights = new Map(candidates.map((c) => [c.path, weightOf(c)]));
|
|
|
+ const topWeight = Math.max(...weights.values());
|
|
|
+ if (!(topWeight > 0)) return empty;
|
|
|
+
|
|
|
+ // Cliff over the WHOLE candidate list, before `maxFiles` — otherwise the file
|
|
|
+ // cap fills with cliff-bound files and the slot they free is never handed on.
|
|
|
+ const cliffAt = Math.min(topWeight * A.CLIFF_FRACTION, A.CLIFF_MAX);
|
|
|
+ const cliffed: string[] = [];
|
|
|
+ let admitted: ExploreAllocationCandidate[] = [];
|
|
|
+ for (const c of candidates) {
|
|
|
+ if (!c.spine && (weights.get(c.path) ?? 0) < cliffAt) cliffed.push(c.path);
|
|
|
+ else admitted.push(c);
|
|
|
+ }
|
|
|
+ // Never cliff every candidate: an empty response costs a whole round-trip.
|
|
|
+ if (admitted.length === 0) {
|
|
|
+ admitted = [candidates[0]!];
|
|
|
+ cliffed.splice(cliffed.indexOf(candidates[0]!.path), 1);
|
|
|
+ }
|
|
|
+ for (const c of admitted.slice(maxFiles)) cliffed.push(c.path);
|
|
|
+ admitted = admitted.slice(0, maxFiles);
|
|
|
+
|
|
|
+ // Serve fewer files well rather than many badly: the envelope has to afford
|
|
|
+ // MIN_CHARS for everything admitted. When it can't, cliff the lowest-weight
|
|
|
+ // files (never a spine file, never the last one) in one deterministic trim —
|
|
|
+ // not one at a time, which is how the old starvation rule snowballed.
|
|
|
+ const affordable = Math.max(1, Math.floor(budget.maxOutputChars / (A.MIN_CHARS + A.FILE_OVERHEAD)));
|
|
|
+ if (admitted.length > affordable) {
|
|
|
+ const byWeight = [...admitted].sort((a, b) => (weights.get(b.path) ?? 0) - (weights.get(a.path) ?? 0));
|
|
|
+ const keep = new Set(byWeight.slice(0, affordable).map((c) => c.path));
|
|
|
+ for (const c of admitted) if (c.spine) keep.add(c.path);
|
|
|
+ for (const c of admitted) if (!keep.has(c.path)) cliffed.push(c.path);
|
|
|
+ admitted = admitted.filter((c) => keep.has(c.path));
|
|
|
+ }
|
|
|
+
|
|
|
+ const allowances = new Map<string, number>();
|
|
|
+ const pool = Math.max(0, budget.maxOutputChars - A.FILE_OVERHEAD * admitted.length);
|
|
|
+ const total = admitted.reduce((s, c) => s + (weights.get(c.path) ?? 0), 0);
|
|
|
+ if (total <= 0 || admitted.length === 0) return { allowances, cliffed, cliffAt, pool };
|
|
|
+ // Everyone gets MIN_CHARS; the REMAINDER is what splits by weight. The floor
|
|
|
+ // is what keeps a diffuse survey question returning a useful spread, and the
|
|
|
+ // remainder is what concentrates a precise one — the top file's slice grows
|
|
|
+ // with its weight share, uncapped by any flat per-file limit.
|
|
|
+ const ceiling = Math.round(budget.maxOutputChars * A.MAX_SHARE);
|
|
|
+ const floors = Math.min(pool, A.MIN_CHARS * admitted.length);
|
|
|
+ const remainder = Math.max(0, pool - floors);
|
|
|
+ for (const c of admitted) {
|
|
|
+ const share = Math.floor(floors / admitted.length)
|
|
|
+ + Math.round((remainder * (weights.get(c.path) ?? 0)) / total);
|
|
|
+ allowances.set(c.path, Math.min(share, ceiling));
|
|
|
+ }
|
|
|
+ return { allowances, cliffed, cliffAt, pool };
|
|
|
+}
|
|
|
+
|
|
|
/**
|
|
|
* Whether `codegraph_explore` should prefix source lines with their line
|
|
|
* numbers (cat -n style: `<num>\t<code>`).
|
|
|
@@ -3497,6 +3706,26 @@ export class ToolHandler {
|
|
|
});
|
|
|
}
|
|
|
|
|
|
+ // Score-proportional byte allocation (CG-12). Every file's share of the
|
|
|
+ // envelope is reserved HERE, before a single byte renders, so the render loop
|
|
|
+ // spends a reservation instead of racing for whatever the files above it left.
|
|
|
+ const allocation = allocateExploreBudget(
|
|
|
+ sortedFiles.map(([fp, group]) => ({
|
|
|
+ path: fp,
|
|
|
+ score: group.score,
|
|
|
+ worth: rankPenalty(fp),
|
|
|
+ spine: group.nodes.some((n) => flow.pathNodeIds.has(n.id)),
|
|
|
+ })),
|
|
|
+ budget,
|
|
|
+ maxFiles,
|
|
|
+ );
|
|
|
+ diag?.setAllocation(allocation.allowances, allocation.cliffed, allocation.cliffAt, allocation.pool);
|
|
|
+ // Cliffed files ship as pointers — path, symbols, line numbers — so the agent
|
|
|
+ // can name one in a follow-up explore. Rendered below with the other
|
|
|
+ // not-shown files, and force-enabled even on tiers that suppress that list:
|
|
|
+ // a file we deliberately withheld source for must still be nameable.
|
|
|
+ const cliffedFiles = new Set(allocation.cliffed);
|
|
|
+
|
|
|
// Polymorphic-sibling detector for adaptive sizing. A class that implements/
|
|
|
// extends a supertype shared by >= MIN_SIBLINGS classes is one of many
|
|
|
// INTERCHANGEABLE implementations (OkHttp's 14 `: Interceptor` classes —
|
|
|
@@ -3557,6 +3786,13 @@ export class ToolHandler {
|
|
|
lines.push('> The code below is the **verbatim, current on-disk source** of these files — re-read from disk on this call and line-numbered, byte-for-byte identical to what the Read tool returns. It is NOT a summary, outline, or stale cache. Treat each block as a Read you have already performed: do not Read a file shown here.');
|
|
|
lines.push('');
|
|
|
|
|
|
+ // Absolute stop for the render loop. Reservations already fit the envelope, so
|
|
|
+ // this only catches their bounded overshoot (the whole-file grace, an oversize
|
|
|
+ // first cluster) — and catches it HERE, where a file can be skipped cleanly and
|
|
|
+ // a later one still render, instead of at the final truncation, which lops off
|
|
|
+ // whichever section happened to land last. Kept in sync with `hardCeiling`
|
|
|
+ // below; the margin covers the drift epilogue and the trailing notes.
|
|
|
+ const renderCeiling = Math.min(Math.round(budget.maxOutputChars * 1.5), 25000) - 600;
|
|
|
let totalChars = lines.join('\n').length;
|
|
|
let filesIncluded = 0;
|
|
|
// Paths we actually render source for below. Drives the curated header count
|
|
|
@@ -3577,19 +3813,22 @@ export class ToolHandler {
|
|
|
if (diag) for (const [fp] of sortedFiles) diag.recordSkip(fp, 'max-files');
|
|
|
break;
|
|
|
}
|
|
|
- // A file DEFINES a named/spine symbol (the answer) vs merely references the
|
|
|
- // flow. Past 90% budget, stop pulling INCIDENTAL files — but keep scanning
|
|
|
- // for necessary ones, which render even past the cap (bounded by maxFiles).
|
|
|
- // Without this `continue` (was an unconditional `break`), the loop stopped
|
|
|
- // after the build + validators-exec files and never reached the ranked-in
|
|
|
- // validate-logic file (Alamofire's Validation.swift).
|
|
|
- const fileNecessary = group.nodes.some(n =>
|
|
|
- entryNodeIds.has(n.id) || flow.pathNodeIds.has(n.id) || flow.uniqueNamedNodeIds.has(n.id));
|
|
|
- if (!fileNecessary && totalChars > budget.maxOutputChars * 0.9) {
|
|
|
- diag?.recordSkip(filePath, 'budget-90pct');
|
|
|
+ // Below the relevance cliff: no source, no `maxFiles` slot. It is still
|
|
|
+ // named — with its matched symbols and their line numbers — in the
|
|
|
+ // not-shown list, so one follow-up explore fetches it in full.
|
|
|
+ if (cliffedFiles.has(filePath)) {
|
|
|
+ diag?.recordSkip(filePath, 'cliff');
|
|
|
+ continue;
|
|
|
+ }
|
|
|
+ // This file's reserved share of the envelope. Every render path below is
|
|
|
+ // bounded by it instead of by the flat per-file cap, which is what stops
|
|
|
+ // allocation from following file size: a small weakly-relevant file no
|
|
|
+ // longer ships whole while the strongly-relevant one is clipped.
|
|
|
+ const allowance = allocation.allowances.get(filePath);
|
|
|
+ if (allowance === undefined) {
|
|
|
+ diag?.recordSkip(filePath, 'max-files');
|
|
|
continue;
|
|
|
}
|
|
|
-
|
|
|
const absPath = validatePathWithinRoot(projectRoot, filePath);
|
|
|
if (!absPath || !existsSync(absPath)) {
|
|
|
diag?.recordSkip(filePath, 'unreadable');
|
|
|
@@ -3648,7 +3887,7 @@ export class ToolHandler {
|
|
|
.filter(n => CALLABLE_BODY.has(n.kind) && (flow.pathNodeIds.has(n.id) || flow.uniqueNamedNodeIds.has(n.id)))
|
|
|
.reduce((s, n) => s + fileLines.slice(n.startLine - 1, n.endLine).join('\n').length, 0);
|
|
|
const onSpineGodFile = hasSpineNode
|
|
|
- && namedBodyChars > budget.maxCharsPerFile
|
|
|
+ && namedBodyChars > allowance
|
|
|
&& group.nodes.some(n => CALLABLE_BODY.has(n.kind) && flow.uniqueNamedNodeIds.has(n.id) && !flow.pathNodeIds.has(n.id));
|
|
|
if (!fileStale && adaptiveExploreEnabled() && flow.pathNodeIds.size > 0
|
|
|
&& (onSpineGodFile || (!hasSpineNode && isPolymorphicSibling(group.nodes) && !spared))) {
|
|
|
@@ -3666,14 +3905,14 @@ export class ToolHandler {
|
|
|
: flow.pathNodeIds.has(n.id) ? 0
|
|
|
: flow.uniqueNamedNodeIds.has(n.id) ? 1
|
|
|
: (fileDefinesSuper && flow.namedNodeIds.has(n.id)) ? 2 : 99;
|
|
|
- // One ~250-line WINDOW per file. syms are taken by priority (spine first,
|
|
|
- // then uniquely-named, then family-base), and the cap applies to ALL of
|
|
|
- // them — including the spine — so a big-spine god-file (tokio's worker.rs:
|
|
|
- // run→run_task→next_task→steal_work) can't eat the whole response and
|
|
|
- // starve the co-flow file (harness.rs's poll). The native agent windows
|
|
|
- // such a file too (~190 lines at a time), so this mimics, not truncates.
|
|
|
- // Always emit ≥1 (never an empty section).
|
|
|
- const bodyCap = budget.maxCharsPerFile * 1.5;
|
|
|
+ // One WINDOW per file, sized by this file's RESERVATION. syms are taken by
|
|
|
+ // priority (spine first, then uniquely-named, then family-base), and the cap
|
|
|
+ // applies to ALL of them — including the spine — so a big-spine god-file
|
|
|
+ // (tokio's worker.rs: run→run_task→next_task→steal_work) can't eat the whole
|
|
|
+ // response and starve the co-flow file (harness.rs's poll). The native agent
|
|
|
+ // windows such a file too (~190 lines at a time), so this mimics, not
|
|
|
+ // truncates. Always emit ≥1 (never an empty section).
|
|
|
+ const bodyCap = allowance;
|
|
|
const bodyIds = new Set<string>();
|
|
|
let bodyChars = 0;
|
|
|
for (const n of syms.filter(n => prio(n) < 99 && n.endLine >= n.startLine).sort((a, b) => prio(a) - prio(b))) {
|
|
|
@@ -3746,16 +3985,19 @@ export class ToolHandler {
|
|
|
// the ceiling and falls through to sectioning/clustering below — full method
|
|
|
// bodies + signatures — so we never dump (or overflow on) a whole god-file.
|
|
|
const isCentralFile = centralFiles.has(filePath);
|
|
|
- // Central files get a slightly larger whole-file window than peripheral ones,
|
|
|
- // but a TIGHT one (~1.5× the per-file cap): the native read of a central file
|
|
|
- // is a ~150–250 line orientation window, NOT the whole file. A flat "whole
|
|
|
- // central file" both overflowed the inline cap AND starved the co-flow files
|
|
|
- // (worker.rs ate the budget, dropping harness.rs's poll). A larger central
|
|
|
- // file falls through to per-method windowing/clustering below.
|
|
|
+ // A file ships whole when it fits its RESERVATION (plus a small grace — see
|
|
|
+ // WHOLE_FILE_GRACE). This is the site of the #1500 allocation bug: the
|
|
|
+ // peripheral bound used to be a flat `maxCharsPerFile * 3`, so ANY file under
|
|
|
+ // ~11K shipped its entire contents regardless of relevance, while a
|
|
|
+ // high-scoring file too big for that window was clipped to `maxCharsPerFile`
|
|
|
+ // — a 3x swing decided by file size alone. Tying both bounds to the
|
|
|
+ // reservation removes the swing without touching the rule's purpose (a small
|
|
|
+ // file sliced is a lossy subset the agent just Reads in full anyway).
|
|
|
const WHOLE_FILE_MAX_LINES = isCentralFile ? 280 : 220;
|
|
|
- const WHOLE_FILE_MAX_CHARS = isCentralFile
|
|
|
- ? Math.min(Math.max(0, budget.maxOutputChars - totalChars - 200), Math.round(budget.maxCharsPerFile * 1.5))
|
|
|
- : budget.maxCharsPerFile * 3;
|
|
|
+ const WHOLE_FILE_MAX_CHARS = allowance + Math.min(
|
|
|
+ EXPLORE_ALLOCATION.WHOLE_FILE_GRACE_MAX,
|
|
|
+ Math.round(allowance * EXPLORE_ALLOCATION.WHOLE_FILE_GRACE_FRACTION),
|
|
|
+ );
|
|
|
if (fileLines.length <= WHOLE_FILE_MAX_LINES && fileContent.length <= WHOLE_FILE_MAX_CHARS) {
|
|
|
const body = fileContent.replace(/\n+$/, '');
|
|
|
let wholeSection = exploreLineNumbersEnabled() ? numberSourceLines(body, 1) : body;
|
|
|
@@ -3772,10 +4014,9 @@ export class ToolHandler {
|
|
|
const staleSuffix = fileStale ? ' · ⚠ changed since last index sync — source below is current; the symbol list may be outdated' : '';
|
|
|
const wholeHeader = fileSectionHeader(filePath, (omitted > 0 ? `${headerNames.join(', ')}, +${omitted} more` : headerNames.join(', ')) + staleSuffix);
|
|
|
|
|
|
- if (!fileNecessary && totalChars + wholeSection.length + 200 > budget.maxOutputChars) {
|
|
|
- // Don't slice a whole file mid-method: an incidental file that doesn't
|
|
|
- // fit is skipped; a necessary one (below) renders in full. Half a file
|
|
|
- // forces the Read this is meant to prevent.
|
|
|
+ if (totalChars + wholeSection.length + 200 > renderCeiling) {
|
|
|
+ // Don't slice a whole file mid-method — a file that doesn't fit is
|
|
|
+ // skipped whole. Half a file forces the Read this is meant to prevent.
|
|
|
anyFileTrimmed = true;
|
|
|
diag?.recordSkip(filePath, 'budget-whole-file');
|
|
|
continue;
|
|
|
@@ -3883,8 +4124,16 @@ export class ToolHandler {
|
|
|
}
|
|
|
|
|
|
const gapThreshold = budget.gapThreshold;
|
|
|
- const clusters: Array<{ start: number; end: number; symbols: string[]; score: number; maxImportance: number; hasSpine: boolean; spineCallLine?: number }> = [];
|
|
|
- let current = {
|
|
|
+ type ExploreRange = typeof ranges[number];
|
|
|
+ type ExploreCluster = {
|
|
|
+ start: number; end: number; symbols: string[]; score: number;
|
|
|
+ maxImportance: number; hasSpine: boolean; spineCallLine?: number;
|
|
|
+ /** The whole symbol ranges this cluster merged — the unit an oversize
|
|
|
+ * cluster is shrunk by, so shrinking never cuts through a body. */
|
|
|
+ members: ExploreRange[];
|
|
|
+ };
|
|
|
+ const clusters: ExploreCluster[] = [];
|
|
|
+ let current: ExploreCluster = {
|
|
|
start: ranges[0]!.start,
|
|
|
end: ranges[0]!.end,
|
|
|
symbols: [`${ranges[0]!.name}(${ranges[0]!.kind})`],
|
|
|
@@ -3892,6 +4141,7 @@ export class ToolHandler {
|
|
|
maxImportance: ranges[0]!.importance,
|
|
|
hasSpine: ranges[0]!.spine,
|
|
|
spineCallLine: ranges[0]!.spineCallLine,
|
|
|
+ members: [ranges[0]!],
|
|
|
};
|
|
|
|
|
|
for (let i = 1; i < ranges.length; i++) {
|
|
|
@@ -3903,6 +4153,7 @@ export class ToolHandler {
|
|
|
current.maxImportance = Math.max(current.maxImportance, r.importance);
|
|
|
current.hasSpine = current.hasSpine || r.spine;
|
|
|
current.spineCallLine = current.spineCallLine ?? r.spineCallLine;
|
|
|
+ current.members.push(r);
|
|
|
} else {
|
|
|
clusters.push(current);
|
|
|
current = {
|
|
|
@@ -3913,6 +4164,7 @@ export class ToolHandler {
|
|
|
maxImportance: r.importance,
|
|
|
hasSpine: r.spine,
|
|
|
spineCallLine: r.spineCallLine,
|
|
|
+ members: [r],
|
|
|
};
|
|
|
}
|
|
|
}
|
|
|
@@ -3962,6 +4214,50 @@ export class ToolHandler {
|
|
|
return withLineNumbers ? numberSourceLines(slice, startIdx + 1) : slice;
|
|
|
};
|
|
|
|
|
|
+ /**
|
|
|
+ * Shrink an oversize cluster to the highest-importance symbols inside it
|
|
|
+ * that fit `cap`, rendered in source order with gap markers (CG-12).
|
|
|
+ *
|
|
|
+ * A cluster is a MERGE of whole symbol ranges, and on a densely-packed file
|
|
|
+ * every symbol merges into one blob spanning the file — cycle.go's 209-line
|
|
|
+ * `Service` is one cluster covering `RunCycle`, `runPayrollCycleAll` and
|
|
|
+ * seven incidental accessors. The old rule took the top-ranked cluster whole
|
|
|
+ * however big it was, so a single-cluster file simply ignored its budget:
|
|
|
+ * it took ~40% more than it was allotted, and the file below it was then
|
|
|
+ * dropped for lack of room (that is how `BuildPayslip` — the "calculate"
|
|
|
+ * half of the #1500 query — went missing entirely). Shrinking by MEMBER
|
|
|
+ * keeps every rule that matters: only whole symbol ranges are emitted, so a
|
|
|
+ * body is never cut, and the members are chosen by the same importance the
|
|
|
+ * cluster ranking uses. Returns null when nothing needed shrinking.
|
|
|
+ */
|
|
|
+ const shrinkCluster = (c: ExploreCluster, cap: number): string | null => {
|
|
|
+ if (c.members.length < 2) return null;
|
|
|
+ const byImportance = [...c.members].sort((a, b) =>
|
|
|
+ b.importance - a.importance || (a.end - a.start) - (b.end - b.start) || a.start - b.start);
|
|
|
+ const sizeOf = (r: ExploreRange) => fileLines.slice(r.start - 1, r.end).join('\n').length;
|
|
|
+ const keep: ExploreRange[] = [];
|
|
|
+ let kept = 0;
|
|
|
+ for (const r of byImportance) {
|
|
|
+ const sz = sizeOf(r) + GAP_MARKER.length;
|
|
|
+ // Always keep the most important range, even if it alone is oversize —
|
|
|
+ // an empty section sends the agent to Read, which costs far more.
|
|
|
+ if (keep.length > 0 && kept + sz > cap) continue;
|
|
|
+ keep.push(r);
|
|
|
+ kept += sz;
|
|
|
+ }
|
|
|
+ if (keep.length === c.members.length) return null;
|
|
|
+ // Re-merge the kept ranges in source order so adjacent survivors read as
|
|
|
+ // one block rather than a stutter of one-symbol fragments.
|
|
|
+ keep.sort((a, b) => a.start - b.start);
|
|
|
+ const merged: Array<{ start: number; end: number }> = [];
|
|
|
+ for (const r of keep) {
|
|
|
+ const last = merged[merged.length - 1];
|
|
|
+ if (last && r.start <= last.end + gapThreshold) last.end = Math.max(last.end, r.end);
|
|
|
+ else merged.push({ start: r.start, end: r.end });
|
|
|
+ }
|
|
|
+ return merged.map((m) => buildSection(m)).join(GAP_MARKER);
|
|
|
+ };
|
|
|
+
|
|
|
// Rank clusters for inclusion under the per-file cap. Entry-point
|
|
|
// clusters come first: a cluster containing a query entry point
|
|
|
// (importance 10) must outrank a dense block of mere declarations,
|
|
|
@@ -3988,28 +4284,41 @@ export class ToolHandler {
|
|
|
return a.span - b.span;
|
|
|
});
|
|
|
|
|
|
- // Per-file budget is the SMALLER of the per-file cap and what's left of the
|
|
|
- // total output cap — so selection (which ranks by importance) keeps the
|
|
|
+ // Per-file budget is this file's RESERVATION, bounded by what's left before
|
|
|
+ // the hard ceiling — so selection (which ranks by importance) keeps the
|
|
|
// high-importance clusters and drops peripheral ones, instead of the
|
|
|
// downstream source-order trim slicing off whatever comes last in the file.
|
|
|
// That source-order slice is what cut Django's `_fetch_all` (L2237, importance
|
|
|
// 9 — agent-named) when query.py was the last of four big files to be emitted.
|
|
|
- const fileBudget = Math.min(budget.maxCharsPerFile, Math.max(0, budget.maxOutputChars - totalChars - 200));
|
|
|
- // Spine ceiling: a flow-path cluster may exceed the per-file cap (the call
|
|
|
- // path is the answer), but bounded — at most ~2.5× the per-file cap and never
|
|
|
- // past what's left of the total output cap — so a pathological long in-file
|
|
|
+ // It used to be `min(maxCharsPerFile, remaining)`: a flat cap that clipped the
|
|
|
+ // top-scoring file at the same 3,800 as the weakest one, while the whole-file
|
|
|
+ // branch above handed a small file 3x that. The reservation is the whole point
|
|
|
+ // of CG-12 — bytes follow relevance, not file size.
|
|
|
+ const headroom = Math.max(0, renderCeiling - totalChars - 200);
|
|
|
+ const fileBudget = Math.min(allowance, headroom);
|
|
|
+ // Spine ceiling: a flow-path cluster may exceed the reservation (the call path
|
|
|
+ // IS the answer and clipping it forces the Read), but bounded — 1.5x the
|
|
|
+ // reservation and never past the ceiling — so a pathological long in-file
|
|
|
// spine can't run away or starve co-flow files entirely.
|
|
|
- const SPINE_CEILING = Math.min(budget.maxCharsPerFile * 2.5, Math.max(0, budget.maxOutputChars - totalChars - 200));
|
|
|
+ const SPINE_CEILING = Math.min(Math.round(allowance * 1.5), headroom);
|
|
|
const chosenIndices = new Set<number>();
|
|
|
+ // Shrunk renders for oversize clusters, by cluster index (CG-12). Computed
|
|
|
+ // during selection and reused at emission so the two never disagree.
|
|
|
+ const shrunkSections = new Map<number, string>();
|
|
|
let projectedChars = 0;
|
|
|
for (const rc of rankedClusters) {
|
|
|
const sectionLen = buildSection(rc.c).length + (chosenIndices.size > 0 ? GAP_MARKER.length : 0);
|
|
|
- // Always take the top-ranked cluster, even if oversize, so we don't
|
|
|
- // return an empty file section (agent would then re-Read the file,
|
|
|
- // negating the savings).
|
|
|
+ // The top-ranked cluster is always taken — an empty file section sends the
|
|
|
+ // agent to Read, negating the savings. But "always taken" is not "taken at
|
|
|
+ // any size": when it overruns the reservation it is SHRUNK to the
|
|
|
+ // highest-importance whole symbol ranges inside it, so a single-cluster
|
|
|
+ // god-file spends its allotment instead of the whole response's.
|
|
|
if (chosenIndices.size === 0) {
|
|
|
+ const cap = rc.c.hasSpine ? SPINE_CEILING : fileBudget;
|
|
|
+ const shrunk = sectionLen > cap ? shrinkCluster(rc.c, cap) : null;
|
|
|
+ if (shrunk !== null) shrunkSections.set(rc.idx, shrunk);
|
|
|
chosenIndices.add(rc.idx);
|
|
|
- projectedChars += sectionLen;
|
|
|
+ projectedChars += shrunk !== null ? shrunk.length : sectionLen;
|
|
|
continue;
|
|
|
}
|
|
|
// A spine cluster (the rendered call path) is the flow answer — include it
|
|
|
@@ -4028,18 +4337,19 @@ export class ToolHandler {
|
|
|
for (let i = 0; i < clusters.length; i++) {
|
|
|
if (!chosenIndices.has(i)) continue;
|
|
|
const cluster = clusters[i]!;
|
|
|
- const section = buildSection(cluster);
|
|
|
+ const section = shrunkSections.get(i) ?? buildSection(cluster);
|
|
|
if (fileSection.length > 0) fileSection += GAP_MARKER;
|
|
|
fileSection += section;
|
|
|
allSymbols.push(...cluster.symbols);
|
|
|
}
|
|
|
|
|
|
- // A chosen cluster is a COMPLETE method-range — we never cut through a body.
|
|
|
- // An oversize single cluster (a long monolithic function) renders in FULL:
|
|
|
- // half a method is useless (the agent just Reads the rest for the other half),
|
|
|
- // which is the very fallback explore exists to prevent. A pathological file is
|
|
|
- // bounded by the per-file cluster SELECTION above + the total hard ceiling.
|
|
|
- if (chosenIndices.size < clusters.length) {
|
|
|
+ // A chosen cluster is a COMPLETE method-range — we never cut through a body,
|
|
|
+ // and a shrunk cluster drops WHOLE members for the same reason. An oversize
|
|
|
+ // single MEMBER (one long monolithic function) still renders in full: half a
|
|
|
+ // method is useless (the agent just Reads the rest for the other half), which
|
|
|
+ // is the very fallback explore exists to prevent. A pathological file is
|
|
|
+ // bounded by the cluster SELECTION above + the total hard ceiling.
|
|
|
+ if (chosenIndices.size < clusters.length || shrunkSections.size > 0) {
|
|
|
anyFileTrimmed = true;
|
|
|
}
|
|
|
|
|
|
@@ -4062,20 +4372,16 @@ export class ToolHandler {
|
|
|
: headerSymbols.join(', ');
|
|
|
const fileHeader = fileSectionHeader(filePath, headerSuffix);
|
|
|
|
|
|
- // The total cap bounds INCIDENTAL files only. A file that DEFINES a symbol
|
|
|
- // the agent named (or that's on the flow spine) renders even when the
|
|
|
- // nominal total is used up — it's the answer, and the set is bounded by
|
|
|
- // maxFiles AND by true-spine/named-seeding having already trimmed each file
|
|
|
- // to its necessary content. A file that merely REFERENCES the flow
|
|
|
- // (Combine.swift name-drops request/task) is incidental → still capped, so
|
|
|
- // freed budget never leaks into noise. This is the last god-file layer:
|
|
|
- // build (Session, true-spined) + validators-exec (Request) + validate
|
|
|
- // (DataRequest/Validation) all render, instead of the cap dropping whichever
|
|
|
- // phase the file order happened to put last.
|
|
|
- if (!fileNecessary && totalChars + fileSection.length + 200 > budget.maxOutputChars) {
|
|
|
- // Incidental file that doesn't fit: SKIP it whole — never slice mid-method.
|
|
|
- // Keep scanning for necessary files (which bypass this cap and render in
|
|
|
- // full, bounded by the hard ceiling).
|
|
|
+ // Last stop before the hard ceiling. The reservation already bounded cluster
|
|
|
+ // selection above, so reaching this means the bounded overshoot (an oversize
|
|
|
+ // first cluster, taken whole rather than sliced mid-method) ran the response
|
|
|
+ // out of room. Skip the file whole and keep scanning — never slice mid-method.
|
|
|
+ // This used to compare against `maxOutputChars` and exempt "necessary" files,
|
|
|
+ // which is how arrival order decided the answer: whichever files ranked first
|
|
|
+ // spent the envelope, and everything after them was dropped on a cap they had
|
|
|
+ // no say in. Reservations replace that exemption — a file that earned bytes
|
|
|
+ // was already given them.
|
|
|
+ if (totalChars + fileSection.length + 200 > renderCeiling) {
|
|
|
anyFileTrimmed = true;
|
|
|
diag?.recordSkip(filePath, 'budget-clusters');
|
|
|
continue;
|
|
|
@@ -4118,9 +4424,16 @@ export class ToolHandler {
|
|
|
|
|
|
// Add remaining files as references (from both relevant and peripheral files).
|
|
|
// Small projects (per budget) skip this — the relevant story already fits
|
|
|
- // in the source section, and a trailing pointer list is pure overhead.
|
|
|
- if (budget.includeAdditionalFiles) {
|
|
|
- const remainingRelevant = sortedFiles.slice(filesIncluded);
|
|
|
+ // in the source section, and a trailing pointer list is pure overhead. But a
|
|
|
+ // CLIFFED file is source we deliberately withheld, so the list is forced on
|
|
|
+ // whenever there is one: withholding a file's bytes is only cheap if the agent
|
|
|
+ // can still name it in a follow-up call (CG-12).
|
|
|
+ if (budget.includeAdditionalFiles || cliffedFiles.size > 0) {
|
|
|
+ // Everything ranked that didn't render, in rank order — cliffed files first,
|
|
|
+ // since they outrank whatever the file cap cut. (Indexing by `filesIncluded`
|
|
|
+ // would be wrong now that cliffed files are skipped without consuming a slot.)
|
|
|
+ const rendered = new Set(renderedFilePaths);
|
|
|
+ const remainingRelevant = sortedFiles.filter(([fp]) => !rendered.has(fp));
|
|
|
// Ranked files are already covered by `remainingRelevant`; the rest of the
|
|
|
// gather (below the floor) becomes the pointer list. The Set guards the
|
|
|
// one overlap case — a file the SCORE_FLOOR_KEEP_MIN fallback pulled in
|
|
|
@@ -4133,8 +4446,17 @@ export class ToolHandler {
|
|
|
if (remainingFiles.length > 0) {
|
|
|
lines.push('**Not shown above — explore these names for their source**');
|
|
|
lines.push('');
|
|
|
+ // A pointer only has to make the file NAMEABLE in a follow-up call, so cap
|
|
|
+ // the symbols per line: an un-capped list ran to ~1.9K on the #1500 fixture
|
|
|
+ // (12 generated CRUD symbols on one line), meta-text bought at the price of
|
|
|
+ // the source bytes this section exists to point away from.
|
|
|
+ const POINTER_SYMBOLS = 6;
|
|
|
for (const [filePath, group] of remainingFiles.slice(0, 10)) {
|
|
|
- const symbols = group.nodes.map(n => `${n.name}:${n.startLine}`).join(', ');
|
|
|
+ const named = group.nodes.filter(n => n.kind !== 'import' && n.kind !== 'export');
|
|
|
+ const shown = (named.length > 0 ? named : group.nodes).slice(0, POINTER_SYMBOLS);
|
|
|
+ const more = (named.length > 0 ? named : group.nodes).length - shown.length;
|
|
|
+ const symbols = shown.map(n => `${n.name}:${n.startLine}`).join(', ')
|
|
|
+ + (more > 0 ? `, +${more} more` : '');
|
|
|
lines.push(`- ${filePath}: ${symbols}`);
|
|
|
}
|
|
|
if (remainingFiles.length > 10) {
|