Parcourir la source

fix(resolution): sweep orphaned unresolved refs so an interrupted index heals on sync (#1187) (#1191)

An indexing run killed mid-"Resolving refs" (crash, Ctrl-C, the #1122
watchdog kill) left the refs it never reached parked in unresolved_refs.
The git-scoped sync fast path only re-resolves changed files' refs, so
those files' call edges were missing permanently — a too-small blast
radius clustering by package/module (the #1187 field report: 3 of 10
caller files for a Spring @Resource-injected method) — until a full
re-index.

- sync() now sweeps leftover unresolved refs with the batched resolver
  after its scoped pass, including on no-change syncs, so a bare
  `codegraph sync` recovers a wedged index (and heals pre-fix indexes
  on the first post-upgrade sync)
- the scoped pass deletes unresolvable rows too (parity with the
  batched path), making "rows at rest" a sound orphan signal
- drop the batched loop's early break that abandoned all later batches
  when one batch was all-unresolvable (its rows WERE consumed — that
  early stop could orphan the rest of the table at init)
- surface the state: `codegraph status` warns, `status --json` gains
  index.pendingRefs, and MCP codegraph_status tells agents the blast
  radius is incomplete until the next sync

Verified end-to-end on a 2,414-file synthetic Spring repo: SIGKILL
mid-resolution reproduces the reporter's exact 3-of-10-callers state;
a bare sync now heals it to 10/10 with the edge count converging to
the clean-init total; a healthy-index sync stays a no-op.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Colby Mchenry il y a 2 mois
Parent
commit
4c15f84aa4
6 fichiers modifiés avec 323 ajouts et 7 suppressions
  1. 1 0
      CHANGELOG.md
  2. 231 0
      __tests__/orphaned-refs-sweep.test.ts
  3. 10 0
      src/bin/codegraph.ts
  4. 44 2
      src/index.ts
  5. 13 0
      src/mcp/tools.ts
  6. 24 5
      src/resolution/index.ts

+ 1 - 0
CHANGELOG.md

@@ -35,6 +35,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 ### Fixes
 
 - `codegraph init` and `codegraph index` no longer get killed by the safety watchdog at the "Resolving refs" step on large method-name-heavy codebases (big Java/enterprise monorepos were the main victims, especially on slower machines). Resolution used to come up for air only every 500 references, so a dense stretch of expensive ones could starve the watchdog long enough for it to assume the process was stuck and kill a perfectly healthy index. Resolution now checkpoints after every reference, and two of the expensive steps got much cheaper: repeated method lookups on the same type are now cached, and source files are no longer re-split line-by-line for every call being resolved — indexing such repos is several times faster as a result. Generated or minified single-line files are also skipped during receiver-type inference instead of being scanned per call. Thanks @UchihaYong and @wangmeng-95 for the reports. (#1122)
+- An index left incomplete by an interrupted run now heals itself on the next sync instead of silently staying wrong forever. If indexing died partway through resolving references (a crash, Ctrl-C, or the watchdog kill fixed above), the affected files still looked indexed but their caller/impact edges were missing — a too-small blast radius clustering by package or module, e.g. a Spring `@Resource`-injected method reporting 3 of its 10 real caller files — and because incremental syncs only re-resolve files that changed, the damage was permanent until a full re-index. Any sync (a watched file change, or a bare `codegraph sync`) now detects the leftover references and finishes resolving them, `codegraph status` warns when an index is in that state instead of passing it off as healthy, and a rare early-stop that could abandon resolution on repos whose first files reference only external libraries is fixed too. Thanks @KnifeOfLife for the report and the package-correlation observation that pinned it down. (#1187)
 - The automatic context hook for Claude Code now fires for structural questions asked in nearly thirty languages — French, Spanish, Portuguese, German, Italian, Dutch, Polish, Czech, Romanian, Hungarian, Greek, Swedish, Danish, Norwegian, Finnish, Russian, Ukrainian, Turkish, Indonesian, Vietnamese, Thai, Hindi, Arabic, Farsi, Hebrew, Japanese, Korean, and both simplified and traditional Chinese — instead of just English and simplified Chinese. Previously a natural question like "comment marche la state machine des commandes ?" injected nothing unless it happened to contain a code-shaped symbol name, making the hook look broken for non-English teams. English questions phrased with derived word forms ("explain the architecture…", "what are the dependencies…") now fire too, and prompts in any other language still fire when they name a symbol from the index. Thanks @anthonyle-roy-lgtm for the report. (#1126)
 - Lua and Luau method calls with capitalized names (`obj:Method()` — the standard Roblox convention) now link to the right method. Because Lua's method-call syntax looks identical to a Luau type annotation, a capitalized call like `lg:Log()` was misread as declaring the variable's type, so whenever two or more classes shared a method name (`Init`, `Update`, `Destroy`, …) the call was silently dropped from callers, impact/blast-radius, and flow traces. Lowercase method names were unaffected. Thanks @inth3shadows for the precise root-cause analysis and repro. (#1124)
 - Removed dead code left behind by the discontinued managed-reasoning feature. Its `codegraph login` flow was unplugged before ever shipping in a release, but the unused module still shipped inside the platform bundles, and a security review flagged its Windows browser-open step (it routed the login URL through `cmd`, which would have been unsafe had the flow ever been wired back up). The leftover module and its tests are now fully deleted. Thanks @inth3shadows for the report. (#1114)

+ 231 - 0
__tests__/orphaned-refs-sweep.test.ts

@@ -0,0 +1,231 @@
+/**
+ * Orphaned unresolved-refs sweep (#1187)
+ *
+ * A resolution pass that dies mid-run (watchdog SIGKILL, Ctrl-C, crash)
+ * leaves the refs it never reached in unresolved_refs. The git-scoped sync
+ * fast path only ever reads the changed files' rows, so those orphans — and
+ * the call edges they represent — used to be missing permanently until a
+ * full re-index. Field report: a Spring monorepo where blast radius showed
+ * 3 of 10 caller files for a method behind @Resource field injection.
+ *
+ * These tests pin the healing behavior: a completed pass consumes every row
+ * it processes (resolved or not), and sync sweeps any leftovers even when
+ * no files changed.
+ */
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import * as fs from 'fs';
+import * as path from 'path';
+import * as os from 'os';
+import CodeGraph from '../src/index';
+
+describe('Orphaned refs sweep (#1187)', () => {
+  let testDir: string;
+  let cg: CodeGraph;
+
+  beforeEach(() => {
+    testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-orphan-sweep-'));
+  });
+
+  afterEach(() => {
+    if (cg) {
+      cg.destroy();
+    }
+    if (fs.existsSync(testDir)) {
+      fs.rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+
+  /** Distinct files with a `calls` edge into the node. */
+  function callerFiles(target: { id: string }): string[] {
+    return [...new Set(cg.getCallers(target.id).map((c) => c.node.filePath))].sort();
+  }
+
+  /**
+   * Leave `relPath` in the exact on-disk state a resolution pass killed
+   * mid-run leaves behind: content re-extracted (nodes + refs re-inserted,
+   * old edges cascade-deleted, content hash stamped current) but resolution
+   * never run. The content tweak is needed because re-extraction of
+   * byte-identical content is a no-op; the hash stamp means a later sync
+   * sees NO changed files.
+   */
+  async function interruptAfterExtraction(relPath: string): Promise<void> {
+    fs.appendFileSync(path.join(testDir, relPath), '\n// interrupted-run edit\n');
+    await cg.indexFiles([relPath]);
+  }
+
+  function findMethod(name: string) {
+    const hit = cg
+      .searchNodes(name)
+      .find((r) => (r.node.kind === 'method' || r.node.kind === 'function') && r.node.name === name);
+    expect(hit, `expected an indexed definition of ${name}`).toBeDefined();
+    return hit!.node;
+  }
+
+  describe('sync() heals an interrupted resolution run', () => {
+    beforeEach(async () => {
+      // The #1187 shape: a concrete @Component class called through Spring
+      // @Resource field injection from another package.
+      const supportDir = path.join(testDir, 'src', 'support');
+      const notifyDir = path.join(testDir, 'src', 'notify');
+      fs.mkdirSync(supportDir, { recursive: true });
+      fs.mkdirSync(notifyDir, { recursive: true });
+
+      fs.writeFileSync(
+        path.join(supportDir, 'MemberDescriptionSupport.java'),
+        [
+          'package com.demo.support;',
+          '',
+          'public class MemberDescriptionSupport {',
+          '    public String getSuperVipName() {',
+          '        return "SVIP";',
+          '    }',
+          '}',
+          '',
+        ].join('\n')
+      );
+
+      fs.writeFileSync(
+        path.join(notifyDir, 'NotifyBuilder.java'),
+        [
+          'package com.demo.notify;',
+          '',
+          'import com.demo.support.MemberDescriptionSupport;',
+          '',
+          'public class NotifyBuilder {',
+          '    private MemberDescriptionSupport memberDescriptionSupport;',
+          '',
+          '    public String buildParams() {',
+          '        return memberDescriptionSupport.getSuperVipName();',
+          '    }',
+          '}',
+          '',
+        ].join('\n')
+      );
+
+      cg = CodeGraph.initSync(testDir);
+      await cg.indexAll();
+    });
+
+    it('resolves leftover refs on a sync with NO file changes', async () => {
+      const target = findMethod('getSuperVipName');
+
+      // Healthy baseline: the caller edge exists, no refs pending.
+      expect(callerFiles(target)).toContain('src/notify/NotifyBuilder.java');
+      expect(cg.getPendingReferenceCount()).toBe(0);
+
+      // Simulate the interrupted run: re-extract the caller (cascade-deleting
+      // its old nodes and edges, re-inserting its refs) and stop before
+      // resolution — exactly the state a killed "Resolving refs" phase
+      // leaves behind.
+      await interruptAfterExtraction('src/notify/NotifyBuilder.java');
+      expect(cg.getPendingReferenceCount()).toBeGreaterThan(0);
+      expect(callerFiles(target)).not.toContain('src/notify/NotifyBuilder.java');
+
+      // The file on disk is unchanged, so this sync re-extracts nothing —
+      // pre-fix it returned without touching resolution and the edge stayed
+      // missing forever.
+      const result = await cg.sync();
+      expect(result.filesAdded).toBe(0);
+      expect(result.filesModified).toBe(0);
+
+      expect(cg.getPendingReferenceCount()).toBe(0);
+      expect(callerFiles(target)).toContain('src/notify/NotifyBuilder.java');
+    });
+
+    it('is idempotent: a second no-change sync stays clean', async () => {
+      await interruptAfterExtraction('src/notify/NotifyBuilder.java');
+      await cg.sync();
+      const target = findMethod('getSuperVipName');
+      const healed = callerFiles(target);
+
+      const again = await cg.sync();
+      expect(again.filesAdded + again.filesModified + again.filesRemoved).toBe(0);
+      expect(cg.getPendingReferenceCount()).toBe(0);
+      expect(callerFiles(target)).toEqual(healed);
+    });
+  });
+
+  describe('completed passes consume every processed row', () => {
+    it('resolveReferences() deletes unresolvable rows (parity with the batched path)', async () => {
+      const srcDir = path.join(testDir, 'src');
+      fs.mkdirSync(srcDir, { recursive: true });
+      fs.writeFileSync(
+        path.join(srcDir, 'app.ts'),
+        [
+          'export function helper() { return 1; }',
+          'export function main() {',
+          '  helper();',
+          '  totallyUndefinedCall();', // resolves to nothing anywhere
+          '}',
+          '',
+        ].join('\n')
+      );
+
+      cg = CodeGraph.initSync(testDir);
+      await cg.indexAll();
+      expect(cg.getPendingReferenceCount()).toBe(0);
+
+      // Re-extract without resolving: both the resolvable helper() ref and
+      // the unresolvable one are back in the table.
+      await interruptAfterExtraction('src/app.ts');
+      expect(cg.getPendingReferenceCount()).toBeGreaterThan(0);
+
+      // The non-batched full pass (which also backs the git-scoped sync
+      // path) must consume BOTH: pre-fix it deleted only resolved rows, so
+      // unresolvable ones parked forever and defeated the orphan sweep's
+      // "non-empty table means interrupted run" invariant.
+      cg.resolveReferences();
+      expect(cg.getPendingReferenceCount()).toBe(0);
+    });
+
+    it('batched resolution does not stop at an all-unresolvable batch', async () => {
+      const srcDir = path.join(testDir, 'src');
+      fs.mkdirSync(srcDir, { recursive: true });
+      // File A: only unresolvable refs. Extracted first, so its rows sort
+      // first and fill the whole first batch.
+      fs.writeFileSync(
+        path.join(srcDir, 'a.ts'),
+        [
+          'export function a() {',
+          '  ghostOne();',
+          '  ghostTwo();',
+          '  ghostThree();',
+          '}',
+          '',
+        ].join('\n')
+      );
+      // File B: a resolvable ref whose rows sort after A's.
+      fs.writeFileSync(
+        path.join(srcDir, 'b.ts'),
+        [
+          "import { target } from './c';",
+          'export function b() { target(); }',
+          '',
+        ].join('\n')
+      );
+      fs.writeFileSync(
+        path.join(srcDir, 'c.ts'),
+        'export function target() { return 2; }\n'
+      );
+
+      cg = CodeGraph.initSync(testDir);
+      await cg.indexAll();
+
+      // Re-queue A's refs then B's, in that order.
+      await interruptAfterExtraction('src/a.ts');
+      await interruptAfterExtraction('src/b.ts');
+      expect(cg.getPendingReferenceCount()).toBeGreaterThan(0);
+
+      // Batch size 2 puts only A's unresolvable refs in the first batch.
+      // The old early break ended the whole run there, leaving B's ref an
+      // orphan even though the batch's rows WERE consumed (progress).
+      const resolver = (cg as unknown as { resolver: { resolveAndPersistBatched(p?: unknown, b?: number): Promise<unknown> } }).resolver;
+      await resolver.resolveAndPersistBatched(undefined, 2);
+
+      expect(cg.getPendingReferenceCount()).toBe(0);
+      const target = findMethod('target');
+      expect(callerFiles(target)).toContain('src/b.ts');
+    });
+  });
+});

+ 10 - 0
src/bin/codegraph.ts

@@ -807,6 +807,9 @@ program
       const buildInfo = cg.getIndexBuildInfo();
       const reindexRecommended = cg.isIndexStale();
       const indexState = cg.getIndexState();
+      // Zero on a healthy index; non-zero at rest means a resolution pass was
+      // interrupted, so some files' call edges are missing (#1187).
+      const pendingRefs = cg.getPendingReferenceCount();
 
       // JSON output mode
       if (options.json) {
@@ -842,6 +845,10 @@ program
             // (a run was killed mid-index — the index is truncated) |
             // 'failed' | null (predates the marker).
             state: indexState,
+            // References awaiting resolution. Non-zero at rest means an
+            // interrupted resolution pass left edges missing; the next
+            // sync sweeps them (#1187).
+            pendingRefs,
           },
         }));
         cg.destroy();
@@ -862,6 +869,9 @@ program
       } else if (indexState === 'failed') {
         warn('The last index run failed — results may be incomplete. Re-run "codegraph index".');
       }
+      if (pendingRefs > 0) {
+        warn(`${formatNumber(pendingRefs)} references from an interrupted run are awaiting resolution — some callers/impact edges are missing. Run "codegraph sync" to resolve them.`);
+      }
       console.log();
 
       // Index stats

+ 44 - 2
src/index.ts

@@ -608,7 +608,8 @@ export class CodeGraph {
         }
 
         // Resolve references if files were updated
-        if (result.filesAdded > 0 || result.filesModified > 0) {
+        const filesChanged = result.filesAdded > 0 || result.filesModified > 0;
+        if (filesChanged) {
           if (result.changedFilePaths) {
             // Scope resolution to changed files (git fast path — bounded set)
             const unresolvedRefs = this.queries.getUnresolvedReferencesByFiles(result.changedFilePaths);
@@ -644,7 +645,38 @@ export class CodeGraph {
               });
             });
           }
+        }
+
+        // Orphan sweep (#1187). A resolution pass that dies mid-run — the #850
+        // daemon liveness watchdog's SIGKILL (#1122), Ctrl-C, a crash — leaves
+        // the refs it never reached in unresolved_refs, and the git-scoped fast
+        // path above never revisits them (it reads only the changed files'
+        // rows). Those files' call edges were then missing PERMANENTLY, with
+        // nothing to see except a too-small blast radius, until a full
+        // re-index. A completed pass deletes every row it processed (resolved
+        // or not), so any row still present now is such an orphan — or a row
+        // parked by an older engine whose scoped pass kept unresolvable refs.
+        // Grind them down with the batched resolver; this also makes a bare
+        // `codegraph sync` the recovery command for a wedged index. On a
+        // healthy index this is one COUNT query.
+        const orphanCount = this.queries.getUnresolvedReferencesCount();
+        if (orphanCount > 0) {
+          options.onProgress?.({
+            phase: 'resolving',
+            current: 0,
+            total: orphanCount,
+          });
+
+          await this.resolveReferencesBatched((current, total) => {
+            options.onProgress?.({
+              phase: 'resolving',
+              current,
+              total,
+            });
+          });
+        }
 
+        if (filesChanged || orphanCount > 0) {
           // Second pass: chained calls whose method lives on a supertype the
           // receiver conforms to (protocol-extension / inherited). Needs the
           // implements/extends edges built above (#750).
@@ -655,7 +687,7 @@ export class CodeGraph {
         }
 
         // Refresh planner stats + checkpoint the WAL after bulk writes.
-        if (result.filesAdded > 0 || result.filesModified > 0 || result.filesRemoved > 0) {
+        if (filesChanged || result.filesRemoved > 0 || orphanCount > 0) {
           this.db.runMaintenance();
         }
 
@@ -873,6 +905,16 @@ export class CodeGraph {
     return this.resolver.resolveAndPersistBatched(onProgress);
   }
 
+  /**
+   * References extracted but not yet resolved into edges. Zero on a healthy
+   * index — a completed resolution pass consumes every row. Non-zero at rest
+   * means a pass was interrupted mid-run (killed indexer, crash — #1187), so
+   * some files' call edges are missing; the next `sync` sweeps them.
+   */
+  getPendingReferenceCount(): number {
+    return this.queries.getUnresolvedReferencesCount();
+  }
+
   /**
    * Get detected frameworks in the project
    */

+ 13 - 0
src/mcp/tools.ts

@@ -4038,6 +4038,19 @@ export class ToolHandler {
       );
     }
 
+    // Non-zero at rest means a resolution pass was interrupted mid-run, so
+    // some files' call/impact edges are missing until the next sync sweeps
+    // the leftovers (#1187). Surface it — an agent trusting an incomplete
+    // blast radius is worse than one that knows to re-sync.
+    const pendingRefs = cg.getPendingReferenceCount();
+    if (pendingRefs > 0) {
+      lines.push(
+        `**Pending resolution:** ⚠ ${pendingRefs} references from an interrupted ` +
+        `index run — some caller/impact edges are missing until the next sync ` +
+        `(any file change triggers it, or run \`codegraph sync\`)`
+      );
+    }
+
     lines.push('', '**Nodes by Kind:**');
 
     for (const [kind, count] of Object.entries(stats.nodesByKind)) {

+ 24 - 5
src/resolution/index.ts

@@ -968,6 +968,23 @@ export class ReferenceResolver {
       );
     }
 
+    // Delete unresolvable refs too — parity with resolveAndPersistBatched.
+    // Keeping them bought nothing: a ref is only ever retried when its file
+    // is re-extracted, which cascade-deletes and re-inserts its rows anyway.
+    // And it broke the #1187 orphan sweep's invariant — after a COMPLETED
+    // pass the table must hold nothing that pass processed, so that any row
+    // still present belongs to an interrupted run and the sweep can key off
+    // a bare row count.
+    if (result.unresolved.length > 0) {
+      this.queries.deleteSpecificResolvedReferences(
+        result.unresolved.map((r) => ({
+          fromNodeId: r.fromNodeId,
+          referenceName: r.referenceName,
+          referenceKind: r.referenceKind,
+        }))
+      );
+    }
+
     return result;
   }
 
@@ -1152,11 +1169,13 @@ export class ReferenceResolver {
       // Yield so progress UI can render between batches
       await new Promise(resolve => setImmediate(resolve));
 
-      // If nothing was resolved or removed in this batch, we'd loop forever
-      // on the same rows. Break to avoid infinite loop.
-      if (result.resolved.length === 0 && result.unresolved.length === batch.length) {
-        break;
-      }
+      // NOTE: there used to be an extra early break here when a batch resolved
+      // nothing (`result.unresolved.length === batch.length`). That was wrong:
+      // an all-unresolvable batch still DELETES its rows (progress), yet the
+      // break abandoned every batch after it in the same run — on a repo whose
+      // first 5000 refs are all external/stdlib calls, resolution stopped at
+      // batch one and left the rest of the table as permanent orphans (#1187).
+      // The count-based guard below catches the true no-progress case.
 
       // Non-progress guard (defense-in-depth). Because we re-read from offset 0
       // each pass, the unresolved_refs table MUST shrink every iteration — both