feat(ui): classify code from the engine's own tree-sitter parse, retiring Shiki (CG-57)

The viewer ran a second highlighter over source the engine had already parsed
with a real grammar: Shiki, plus 56 pruned TextMate grammars shipped in
dist/textmate/. The classification now comes off that tree instead, so a file is
read by exactly the grammar that decided what its symbols are.

The swap is complete rather than flagged: @shikijs/core, @shikijs/engine-javascript
and @shikijs/langs are off the dependency list, scripts/prune-grammars.mjs and
`npm run build:textmate` are deleted, and check-ui-build.mjs asserts the
tree-sitter grammars in dist/extraction/wasm instead of dist/textmate.

The wire contract is unchanged — `[classId, text]` pairs with the class names
alongside — so the viewer's decoder and code blocks did not have to be rewritten.
Two classes are added to the six: `type` (a named type reference, painted at
plain ink) and `def` (the name a definition declares, weight 600), the latter
taken from the extractors' own definition tables so it cannot drift from what
indexing calls a definition.

Three differences are not cosmetic:

* Interpolations (`${…}`, `#{…}`, `$"{…}"`, f-strings) are classified as code,
  not as string. The call-site overlay refuses to claim a token classed string,
  so calls written inside interpolated strings now link.
* Built-in type words are emitted whole and classed `type` in every language.
  The grammars disagree about whether `string` is a type_identifier or an
  anonymous token inside a predefined_type, and TextMate scoped them
  inconsistently too.
* 3 000 lines of TypeScript cost 24-41 ms instead of ~700 ms.

Given up deliberately: Liquid, Razor, YAML, Twig, XML and .properties render
plain. .svelte/.vue/.astro are classified through their <script> blocks, the same
delegation the SFC extractors do. Pulling html/css/vue out of tree-sitter-wasms
would cover them, but those ABI-13 builds are the known cause of shared-WASM-heap
corruption for every other language in the same process.

Measured parity, per-language before/after screenshots and the reproduction
recipe: docs/design/cg57-highlighting-parity.md.

CHANGELOG.md

@@ -56,6 +56,12 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
   Any row that names a symbol can start a **flow**: press `Flow ›`, then type a second symbol or press `→ here` on another row, and you get the path between them — so "how does `POST /v1/payroll/cycles/{cycleID}/run` reach the database" is two clicks. Typing into the search box now finds entry points too, under their own heading below the symbol matches, so a URL comes back with its handler attached instead of on its own.
 
+- **Syntax colouring in `codegraph ui` now comes from CodeGraph's own reading of your code.** The viewer used to run a second syntax highlighter over source CodeGraph had already parsed, with its own separate set of grammars. It doesn't any more: the colouring is taken straight from the parse that built your graph, so a file is coloured by exactly the grammar that decided what its symbols are. Three things you will notice — the name a definition declares now stands out on the line that declares it, wherever it appears; calls written inside a string (`${user.name()}`, `#{...}`, `$"{...}"`) are read as code and are now clickable links like every other call site; and built-in type words such as `string`, `int` and `void` look the same in every language instead of one way in Go and another in TypeScript. A big file paints far faster, most visibly in TypeScript, which was by a wide margin the slowest before.
+
+  Two formats change for the worse and it is worth saying so: Liquid, Razor, YAML, Twig, XML and `.properties` files are shown without colouring now, and in `.svelte`, `.vue` and `.astro` files the `<script>` block is coloured but the surrounding markup is not. Nothing about navigation changes there — call sites in those files still link, exactly as before.
+
+  This also takes about 3 MB of grammar files and two dependencies out of the install.
+
 ### Fixes
 
 - Fixed a long-running `codegraph ui` session serving a symbol that a sync had already deleted. The viewer keeps one connection to your index open, and its in-memory lookup didn't notice when another process — your agent's sync, or `codegraph sync` — rewrote the file underneath it, so a symbol screen could keep showing a body with no callers while search correctly reported it had moved. Because a symbol's identity includes the line it starts on, this happened after almost any edit above it.

CLAUDE.md

@@ -11,7 +11,7 @@ Distributed as `@colbymchenry/codegraph` on npm; same binary serves as installer
 ## Build, Test, Run
 
 ```bash
-npm run build           # tsc + copy schema.sql and *.wasm + prune TextMate grammars + build the viewer into dist/; chmods dist/bin/codegraph.js
+npm run build           # tsc + copy schema.sql and *.wasm + build the viewer into dist/; chmods dist/bin/codegraph.js
 npm run dev             # tsc --watch
 npm run clean           # rm -rf dist
 
@@ -29,10 +29,12 @@ npx vitest run __tests__/extraction.test.ts -t "TypeScript"
 
 `copy-assets` (called from `build`) copies `src/db/schema.sql` and all `src/extraction/wasm/*.wasm` files into `dist/`. **Any new SQL or grammar wasm must be copied or it won't ship.**
 
-Two other build steps write into `dist/` and are subject to the same rule:
-`build:textmate` (`scripts/prune-grammars.mjs`) writes the viewer's syntax grammars to `dist/textmate/`, and
-`build:ui` builds the browser viewer into `dist/viewer/` (never `dist/ui/` — that's the terminal ui).
-`scripts/check-ui-build.mjs` asserts both trees after every build and inside every release archive.
+One other build step writes into `dist/` and is subject to the same rule: `build:ui` builds the
+browser viewer into `dist/viewer/` (never `dist/ui/` — that's the terminal ui).
+`scripts/check-ui-build.mjs` asserts both `dist/viewer/` and the copied grammars in
+`dist/extraction/wasm/` after every build and inside every release archive — the viewer's syntax
+highlighting reads a file with the same grammar the engine indexed it with, so a missing wasm is an
+unhighlighted screen as well as an extraction gap.
 
 Node engines: `>=20.0.0 <25.0.0`. There is a hard exit on Node 25.x and below 20 (see `src/bin/node-version-check.ts`).
 

__tests__/ui-highlight.test.ts

@@ -1,50 +1,45 @@
 /**
- * The viewer's server-side syntax classification (CG-43).
+ * The viewer's server-side syntax classification (CG-43, rebuilt on the
+ * engine's own tree-sitter parse in CG-57).
  *
  * Two things are worth pinning here and they are not the colours. The first is
  * that a call-site link lands on the callee's own name — the accent underline
  * is the only colour in the code block, and putting it on the receiver or on a
  * word inside a comment is worse than not drawing it. The second is that
- * highlighting never becomes a way for a source request to fail: a missing
- * grammar, an oversized slice, a language nobody wrote a grammar for all have
- * to answer with the source and an honest `engine: 'plain'`.
+ * highlighting never becomes a way for a source request to fail: a language
+ * with no grammar, an oversized slice, a minified line all have to answer with
+ * the source and an honest `engine: 'plain'`.
  *
  * The end-to-end shape is deliberate: the server's tokens are fed straight
  * through the viewer's own `decodeLine` and `assignRefs`, because the seam
  * between "how a grammar chose to cut a line" and "which token the overlay
  * claims" is exactly where this breaks.
+ *
+ * These run against the real grammars, which live in `src/extraction/wasm/`
+ * and `tree-sitter-wasms` — the same ones indexing uses — so unlike the Shiki
+ * era there is nothing to build first and nothing to skip.
  */
 
-import { describe, it, expect, beforeAll, vi } from 'vitest';
+import { describe, it, expect, beforeAll } from 'vitest';
 import * as fs from 'fs';
-import * as os from 'os';
 import * as path from 'path';
 import {
   clearHighlightCache,
   grammarFor,
   highlightCacheStats,
   highlightLines,
-  LANGUAGE_GRAMMAR,
+  isHighlightable,
   MAX_HIGHLIGHT_CHARS,
-  REQUIRED_GRAMMARS,
   SLICE_CACHE_LINES,
   TOKEN_CLASSES,
   type HighlightResult,
 } from '../src/ui-server/highlight';
-import { loadManifest } from '../src/ui-server/highlight/grammars';
+import { classifyTree, syntaxRegionsFor } from '../src/extraction/syntax-tokens';
+import { getParser, initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars';
 import { LANGUAGES } from '../src/types';
 import { decodeLine, type Token } from '../ui/src/lib/highlight';
 import { assignRefs, type LineRef } from '../ui/src/lib/symbol-model';
 
-/**
- * The pruned grammars live in `dist/textmate`, written by `npm run build`. A
- * source tree that has only ever been type-checked has none, and the right
- * behaviour there is plain text — which is itself asserted below, so the
- * grammar-dependent cases skip rather than fail.
- */
-const HAS_GRAMMARS = loadManifest() !== null;
-const withGrammars = HAS_GRAMMARS ? it : it.skip;
-
 function tokensOf(result: HighlightResult, line: number): Token[] {
   return decodeLine(result.lines[line] ?? [], result.classes);
 }
@@ -74,10 +69,14 @@ function claimedText(result: HighlightResult, line: number, ref: LineRef): strin
   return index === undefined ? undefined : tokens[index]?.text;
 }
 
-describe('the language table', () => {
-  it('has an entry for every language the engine indexes', () => {
+describe('which languages classify', () => {
+  it('answers for every language the engine indexes, without throwing', () => {
     for (const language of LANGUAGES) {
-      expect(LANGUAGE_GRAMMAR).toHaveProperty(language);
+      expect(() => grammarFor(language)).not.toThrow();
+    }
+    // The ones the classification is measured on all have a grammar.
+    for (const language of ['typescript', 'go', 'python', 'rust', 'swift', 'csharp', 'ruby', 'php']) {
+      expect(isHighlightable(language)).toBe(true);
     }
   });
 
@@ -87,33 +86,30 @@ describe('the language table', () => {
     expect(grammarFor('')).toBeNull();
   });
 
-  withGrammars('ships a grammar for every id the table names', () => {
-    const found = loadManifest();
-    expect(found).not.toBeNull();
-    for (const id of REQUIRED_GRAMMARS) {
-      expect(Object.keys((found as NonNullable<typeof found>).manifest.languages)).toContain(id);
-    }
+  it('reads a single-file component through its script block', () => {
+    // A .svelte file has no grammar of its own; its symbols live in <script>
+    // and the extractor hands those to TypeScript. The classifier follows.
+    expect(grammarFor('svelte')).toBe('typescript');
+    const regions = syntaxRegionsFor('<p>{x}</p>\n<script lang="ts">\nlet x = 1;\n</script>\n', 'svelte');
+    expect(regions).toHaveLength(1);
+    expect(regions?.[0]?.language).toBe('typescript');
   });
 
-  withGrammars('loads a grammar chain dependencies-first, so embedded blocks highlight', () => {
-    const found = loadManifest();
-    const vue = (found as NonNullable<typeof found>).manifest.languages['vue'] ?? [];
-    // The single-file component's own grammar is last; everything it embeds
-    // has to be registered before Shiki resolves `embeddedLangs`.
-    expect(vue[vue.length - 1]).toBe('vue');
-    expect(vue).toContain('typescript');
-    expect(vue.indexOf('typescript')).toBeLessThan(vue.length - 1);
+  it('has no grammar for the formats that only have file-level extraction', () => {
+    for (const language of ['yaml', 'xml', 'properties', 'twig', 'unknown']) {
+      expect(grammarFor(language)).toBeNull();
+    }
   });
 });
 
 describe('classification', () => {
   beforeAll(() => clearHighlightCache());
 
-  withGrammars('reads TypeScript with the four classes the theme paints', async () => {
+  it('reads TypeScript with the classes the theme paints', async () => {
     const result = await highlightLines(['const answer = 42; // note'], {
       language: 'typescript',
     });
-    expect(result.engine).toBe('shiki');
+    expect(result.engine).toBe('tree-sitter');
     expect(result.grammar).toBe('typescript');
     expect(result.classes).toEqual([...TOKEN_CLASSES]);
     const rendered = shape(result, 0);
@@ -123,7 +119,7 @@ describe('classification', () => {
     expect(rendered).toContain('comment:// note');
   });
 
-  withGrammars('reads a # comment as a comment in Python and as code in TypeScript', async () => {
+  it('reads a # comment as a comment in Python and as code in TypeScript', async () => {
     const python = await highlightLines(['x = 1  # note'], { language: 'python' });
     expect(shape(python, 0).at(-1)).toBe('comment:# note');
 
@@ -131,7 +127,7 @@ describe('classification', () => {
     expect(shape(ts, 0).at(-1)).not.toBe('comment:# note');
   });
 
-  withGrammars('carries a block comment across lines within one slice', async () => {
+  it('carries a block comment across lines within one slice', async () => {
     const result = await highlightLines(['/* open', 'still comment', 'done */ const x = 1;'], {
       language: 'typescript',
     });
@@ -140,21 +136,73 @@ describe('classification', () => {
     expect(shape(result, 2)).toContain('keyword:const');
   });
 
-  withGrammars('reads Go, which has its own idea of what a keyword is', async () => {
+  it('reads Go, which has its own idea of what a keyword is', async () => {
     const result = await highlightLines(['func Greet(name string) string {'], { language: 'go' });
     expect(shape(result, 0)).toContain('keyword:func');
-    expect(shape(result, 0)).toContain('ident:Greet');
+    expect(shape(result, 0)).toContain('def:Greet');
   });
 
-  withGrammars('reads ArkTS with the TypeScript grammar', async () => {
+  it('reads ArkTS with its own grammar, not TypeScript’s', async () => {
     const result = await highlightLines(['@Entry struct Index { build() {} }'], {
       language: 'arkts',
     });
-    expect(result.engine).toBe('shiki');
-    expect(result.grammar).toBe('typescript');
+    expect(result.engine).toBe('tree-sitter');
+    expect(result.grammar).toBe('arkts');
+  });
+
+  it('does not read a type annotation’s `string` as a string literal', async () => {
+    // An anonymous tree-sitter node's type IS its text, so `string` in a
+    // signature arrives as a node literally typed `string`. Reading that as a
+    // string literal greys out half of every signature in TypeScript and PHP.
+    for (const [language, line] of [
+      ['typescript', 'function put(key: string): void {}'],
+      ['php', '<?php function put(string $key): void {}'],
+    ] as const) {
+      const result = await highlightLines([line], { language });
+      expect(shape(result, 0)).toContain('type:string');
+      expect(shape(result, 0)).not.toContain('string:string');
+    }
   });
 
-  withGrammars('emits one entry per source line, always', async () => {
+  it('paints a built-in type the same way in every language', async () => {
+    // The grammars disagree: `string` is a `type_identifier` in Go and an
+    // anonymous token inside a `predefined_type` in TypeScript. Left alone that
+    // is one word painting two ways on the same screen.
+    for (const [language, line] of [
+      ['typescript', 'let a: string;'],
+      ['go', 'var a string'],
+      ['csharp', 'string a;'],
+      ['rust', 'let a: u32 = 1;'],
+    ] as const) {
+      const rendered = shape(await highlightLines([line], { language }), 0);
+      expect(rendered.some((t) => t.startsWith('type:'))).toBe(true);
+      expect(rendered.some((t) => t === 'keyword:string' || t === 'keyword:u32')).toBe(false);
+    }
+  });
+
+  it('keeps a template literal’s interpolated call as code, so it can link', async () => {
+    const line = 'const s = `n=${store.size()} done`;';
+    const result = await highlightLines([line], { language: 'typescript' });
+    expect(shape(result, 0)).toContain('ident:size');
+    expect(claimedText(result, 0, lineRef({ ident: 'size' }))).toBe('size');
+  });
+
+  it('marks a definition’s own name, from the extractor’s tables', async () => {
+    const cases: [string, string, string][] = [
+      ['typescript', 'export class Store {}', 'Store'],
+      ['python', 'def put(self):', 'put'],
+      ['rust', 'pub fn put(&self) {}', 'put'],
+      ['ruby', 'class Store', 'Store'],
+      ['csharp', 'public class Store {}', 'Store'],
+      ['swift', 'final class Store {}', 'Store'],
+    ];
+    for (const [language, line, name] of cases) {
+      const result = await highlightLines([line], { language });
+      expect(shape(result, 0)).toContain(`def:${name}`);
+    }
+  });
+
+  it('emits one entry per source line, always', async () => {
     const lines = ['a();', '', 'b();', ''];
     const result = await highlightLines(lines, { language: 'typescript' });
     // The code block indexes rows positionally: one short answer and every
@@ -162,6 +210,34 @@ describe('classification', () => {
     expect(result.lines).toHaveLength(lines.length);
     expect(result.lines[1]).toEqual([]);
   });
+
+  it('reproduces every line of a real file exactly', async () => {
+    // The code block renders these tokens and nothing else, so a dropped or
+    // duplicated character is a corrupted file on screen — silently.
+    const file = path.join(__dirname, '..', 'src', 'ui-server', 'api', 'source.ts');
+    const lines = fs.readFileSync(file, 'utf-8').split('\n');
+    const result = await highlightLines(lines, { language: 'typescript' });
+    expect(result.engine).toBe('tree-sitter');
+    result.lines.forEach((row, i) => {
+      expect(row.map(([, text]) => text).join('')).toBe(lines[i]);
+    });
+  });
+
+  it('classifies a component’s script and leaves its markup plain', async () => {
+    const lines = [
+      '<script lang="ts">',
+      '  let count = 0;',
+      '</script>',
+      '',
+      '<button onclick={bump}>{count}</button>',
+    ];
+    const result = await highlightLines(lines, { language: 'svelte' });
+    expect(result.engine).toBe('tree-sitter');
+    expect(shape(result, 1)).toContain('keyword:let');
+    // The markup still splits into identifiers, so a call site in it links.
+    expect(claimedText(result, 4, lineRef({ ident: 'bump' }))).toBe('bump');
+    expect(result.lines.map((row) => row.map(([, t]) => t).join(''))).toEqual(lines);
+  });
 });
 
 describe('the plain fallback', () => {
@@ -182,7 +258,7 @@ describe('the plain fallback', () => {
     expect(claimedText(result, 0, lineRef({ ident: 'withLock', col: 9 }))).toBe('withLock');
   });
 
-  it('refuses to tokenise a minified line rather than wedging on it', async () => {
+  it('refuses to classify a minified line rather than wedging on it', async () => {
     const enormous = 'a'.repeat(MAX_HIGHLIGHT_CHARS + 1);
     const result = await highlightLines([enormous], { language: 'javascript' });
     expect(result.engine).toBe('plain');
@@ -191,41 +267,17 @@ describe('the plain fallback', () => {
     expect(result.lines[0]?.map(([, text]) => text).join('')).toHaveLength(enormous.length);
   });
 
-  it('answers plain when a shipped grammar file is missing or unreadable', async () => {
-    // The install is half there: a manifest that names a grammar whose file
-    // never made it. The viewer must still get its source.
-    const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-textmate-'));
-    fs.writeFileSync(
-      path.join(dir, 'manifest.json'),
-      JSON.stringify({ shikiVersion: 'test', languages: { typescript: ['typescript'] } })
-    );
-
-    const previous = process.env.CODEGRAPH_TEXTMATE_PATH;
-    process.env.CODEGRAPH_TEXTMATE_PATH = dir;
-    // A fresh module registry: the highlighter and its grammar bookkeeping are
-    // created once per process, and this case is about that first attempt.
-    vi.resetModules();
-    try {
-      const mod = await import('../src/ui-server/highlight');
-      const result: HighlightResult = await mod.highlightLines(['const x = 1;'], {
-        language: 'typescript',
-      });
-      expect(result.engine).toBe('plain');
-      expect(result.reason).toBeTruthy();
-      expect(result.lines[0]?.map(([, text]) => text).join('')).toBe('const x = 1;');
-    } finally {
-      if (previous === undefined) delete process.env.CODEGRAPH_TEXTMATE_PATH;
-      else process.env.CODEGRAPH_TEXTMATE_PATH = previous;
-      fs.rmSync(dir, { recursive: true, force: true });
-      vi.resetModules();
-    }
+  it('answers plain for a component whose script block is empty', async () => {
+    const result = await highlightLines(['<p>hello</p>'], { language: 'svelte' });
+    expect(result.engine).toBe('plain');
+    expect(result.lines[0]?.map(([, text]) => text).join('')).toBe('<p>hello</p>');
   });
 });
 
 describe('graph links land on the right token', () => {
   beforeAll(() => clearHighlightCache());
 
-  withGrammars('marks the callee, not the receiver the recorded column points at', async () => {
+  it('marks the callee, not the receiver the recorded column points at', async () => {
     // The recorded column is the start of the calling EXPRESSION — `this` —
     // and the underline has to end up on `withLock`.
     const line = '    return this.indexMutex.withLock(async () => {';
@@ -235,7 +287,7 @@ describe('graph links land on the right token', () => {
     );
   });
 
-  withGrammars('lands on a real call site in the engine’s own src/index.ts', async () => {
+  it('lands on a real call site in the engine’s own src/index.ts', async () => {
     const file = path.join(__dirname, '..', 'src', 'index.ts');
     const source = fs.readFileSync(file, 'utf-8').split('\n');
     // A line the engine actually contains, found rather than hard-coded, so a
@@ -252,7 +304,7 @@ describe('graph links land on the right token', () => {
     );
   });
 
-  withGrammars('lands on a Go method call', async () => {
+  it('lands on a Go method call', async () => {
     const line = '\tresult := s.repo.FindByID(ctx, id)';
     const result = await highlightLines([line], { language: 'go' });
     expect(claimedText(result, 0, lineRef({ ident: 'FindByID', col: line.indexOf('s.repo') }))).toBe(
@@ -260,7 +312,7 @@ describe('graph links land on the right token', () => {
     );
   });
 
-  withGrammars('lands on a Python method call, not on the receiver of the same name', async () => {
+  it('lands on a Python method call, not on the receiver of the same name', async () => {
     const line = '    return self.store.join(self.store.path)';
     const result = await highlightLines([line], { language: 'python' });
     expect(claimedText(result, 0, lineRef({ ident: 'join', col: line.indexOf('self') }))).toBe(
@@ -268,7 +320,7 @@ describe('graph links land on the right token', () => {
     );
   });
 
-  withGrammars('leaves a word inside a comment or a string alone', async () => {
+  it('leaves a word inside a comment or a string alone', async () => {
     const result = await highlightLines(
       ['  // call render here', '  const s = "render";'],
       { language: 'typescript' }
@@ -277,7 +329,7 @@ describe('graph links land on the right token', () => {
     expect(claimedText(result, 1, lineRef({ ident: 'render' }))).toBeUndefined();
   });
 
-  withGrammars('keeps every identifier separately claimable', async () => {
+  it('keeps every identifier separately claimable', async () => {
     const result = await highlightLines(['render(); render();'], { language: 'typescript' });
     const tokens = tokensOf(result, 0);
     const claimed = assignRefs(tokens, [
@@ -287,7 +339,13 @@ describe('graph links land on the right token', () => {
     expect(claimed.size).toBe(2);
   });
 
-  withGrammars('reproduces the line exactly — the code block renders these tokens', async () => {
+  it('keeps a type name claimable — it is a distinct class, not an excluded one', async () => {
+    const result = await highlightLines(['let store: Store = make();'], { language: 'typescript' });
+    expect(shape(result, 0)).toContain('type:Store');
+    expect(claimedText(result, 0, lineRef({ ident: 'Store' }))).toBe('Store');
+  });
+
+  it('reproduces the line exactly — the code block renders these tokens', async () => {
     const line = '  const s = `a ${b.c()} d`; // 1 + 2';
     const result = await highlightLines([line], { language: 'typescript' });
     expect(
@@ -299,7 +357,27 @@ describe('graph links land on the right token', () => {
 });
 
 describe('cost', () => {
-  withGrammars('answers a cached slice without re-tokenising it', async () => {
+  it('classifies three thousand lines of TypeScript well inside the budget', async () => {
+    clearHighlightCache();
+    const lines = fs
+      .readFileSync(path.join(__dirname, '..', 'src', 'extraction', 'tree-sitter.ts'), 'utf-8')
+      .split('\n')
+      .slice(0, 3000);
+    // Warm the grammar load, which is a one-off per language per process.
+    await highlightLines(lines.slice(0, 5), { language: 'typescript' });
+    clearHighlightCache();
+
+    const started = Date.now();
+    const result = await highlightLines(lines, { language: 'typescript' });
+    const elapsed = Date.now() - started;
+
+    expect(result.engine).toBe('tree-sitter');
+    // The whole point of CG-57's swap: the TextMate grammar took ~700 ms here.
+    // Generous against a loaded CI box; the dev Mac measures 24–41 ms.
+    expect(elapsed).toBeLessThan(400);
+  });
+
+  it('answers a cached slice without re-classifying it', async () => {
     clearHighlightCache();
     const lines = fs
       .readFileSync(path.join(__dirname, '..', 'src', 'ui-server', 'api', 'source.ts'), 'utf-8')
@@ -313,7 +391,7 @@ describe('cost', () => {
     const second = await highlightLines(lines, { language: 'typescript', cacheKey: 'a:1:9999' });
     const warmMs = Date.now() - warm;
 
-    expect(second.engine).toBe('shiki');
+    expect(second.engine).toBe('tree-sitter');
     // The cache is what makes a re-render free: every resize, theme flip and
     // step back through the trail re-asks for the same slice.
     expect(warmMs).toBeLessThan(Math.max(20, coldMs / 4));
@@ -333,7 +411,7 @@ describe('cost', () => {
     expect(stats.lines).toBeLessThanOrEqual(SLICE_CACHE_LINES);
   });
 
-  withGrammars('keys the cache on the content, so an edited file re-highlights', async () => {
+  it('keys the cache on the content, so an edited file re-classifies', async () => {
     clearHighlightCache();
     const first = await highlightLines(['const a = 1;'], {
       language: 'typescript',
@@ -347,3 +425,34 @@ describe('cost', () => {
     expect(second.lines[0]?.map(([, t]) => t).join('')).toBe('const bbb = 2;');
   });
 });
+
+describe('the classifier itself', () => {
+  it('covers the source with ordered, non-overlapping spans', async () => {
+    const source = fs
+      .readFileSync(path.join(__dirname, '..', 'src', 'ui-server', 'api', 'flow.ts'), 'utf-8')
+      .slice(0, 40_000);
+    await initGrammars();
+    await loadGrammarsForLanguages(['typescript']);
+    const parser = getParser('typescript');
+    expect(parser).not.toBeNull();
+    const tree = (parser as NonNullable<typeof parser>).parse(source);
+    const spans = classifyTree((tree as NonNullable<typeof tree>).rootNode, source, 'typescript');
+
+    expect(spans.length).toBeGreaterThan(1000);
+    let previous = 0;
+    for (const span of spans) {
+      expect(span.start).toBeGreaterThanOrEqual(previous);
+      expect(span.end).toBeGreaterThan(span.start);
+      previous = span.end;
+    }
+    expect(previous).toBeLessThanOrEqual(source.length);
+    // Everything the walk did not claim is whitespace the caller fills in.
+    const uncovered: string[] = [];
+    let at = 0;
+    for (const span of spans) {
+      if (span.start > at) uncovered.push(source.slice(at, span.start));
+      at = span.end;
+    }
+    expect(uncovered.every((gap) => gap.trim() === '')).toBe(true);
+  });
+});

__tests__/ui-server-api.test.ts

@@ -671,6 +671,8 @@ describe('GET /api/source', () => {
       'string',
       'keyword',
       'number',
+      'type',
+      'def',
     ]);
     expect(body.highlight.lines).toHaveLength(body.lines.length);
     // Every line's tokens reproduce that line exactly — the code block renders

__tests__/ui-symbol-model.test.ts

@@ -484,7 +484,7 @@ describe('showsBody', () => {
 
 describe('client-side token decoding', () => {
   // The classification itself is the server's job (`src/ui-server/highlight/`,
-  // real TextMate grammars); what is worth pinning here is the decoding — the
+  // the engine's own tree-sitter parse); what is worth pinning here is the decoding — the
   // columns the call-site overlay matches against, and the plain fallback that
   // has to keep links working when no grammar covers a file.
   const CLASSES = ['other', 'ident', 'comment', 'string', 'keyword', 'number'];
@@ -552,7 +552,7 @@ describe('client-side token decoding', () => {
 
   it('keys a slice by real file line, not by offset into the slice', () => {
     const byLine = tokensByLine(['a();', 'b();'], 120, {
-      engine: 'shiki',
+      engine: 'tree-sitter',
       grammar: 'typescript',
       classes: CLASSES,
       lines: [

docs/design/cg57-highlighting-parity.md

@@ -0,0 +1,95 @@
+# Highlighting parity: Shiki → the engine's own tree-sitter parse (CG-57)
+
+The viewer's code block used to be classified by a second highlighter — Shiki with 56 pruned
+TextMate grammars shipped in `dist/textmate/` — over source the engine had already parsed with a
+real grammar. CG-57 takes the classification off that tree instead. This file records what the swap
+changed, measured rather than asserted, so nobody has to re-derive it from a diff.
+
+Screenshots, one per language, before on the left and after on the right, same stylesheet:
+[`cg57-highlighting-parity/`](./cg57-highlighting-parity/) — `typescript.png`, `go.png`,
+`python.png`, `rust.png`, `swift.png`, `csharp.png`, `ruby.png`, `php.png`.
+
+## What it costs
+
+3 000 lines, cold, dev Mac (M-series), parse + classify + wire:
+
+| | TypeScript | Go | Python | Rust | Swift | C# | Ruby | PHP |
+|---|---|---|---|---|---|---|---|---|
+| Shiki + TextMate | ~700 ms | 43–57 ms | 35–47 ms | — | — | — | — | — |
+| Engine tree-sitter | 24–41 ms | ~30 ms | 25–29 ms | 18–19 ms | 25–27 ms | 20–25 ms | 14–16 ms | 20–22 ms |
+
+The task's budget was **< 100 ms per 3 000-line file warm**; every language clears it *cold*.
+TypeScript is the number that mattered: its TextMate grammar was 5–7× every other one and the cost
+was regex *execution*, not compilation, so nothing about the old module could have fixed it. The
+slice cache still exists — a re-render (resize, theme flip, stepping back through the trail) should
+cost nothing at all, and the whole-file view pages the same file repeatedly.
+
+## What it changes on screen
+
+Per-character comparison over ~40 lines of realistic source per language, counting only
+non-whitespace characters, and treating `ident` / `other` / `type` as one bucket because all three
+paint at plain ink:
+
+| language | painted identically | what moved |
+|---|---|---|
+| TypeScript | 91.3% | 33 `def`, 40 interpolation chars now code, 2 punctuation |
+| Go | 91.5% | 37 built-in type words, 13 `def` |
+| Python | 93.2% | 26 `def`, 15 keyword (`is not`, `__future__`) |
+| Rust | 96.3% | 21 `def`, 3 keyword |
+| Swift | 93.3% | 18 `def`, 14 keyword (`throws`/`rethrows`) |
+| C# | 88.3% | 30 built-in type words, 23 `def`, 31 interpolation chars now code |
+| Ruby | 83.8% | 23 `def`, 35 interpolation chars now code, 14 symbol literals, 3 keyword |
+| PHP | 85.9% | 33 built-in type words, 29 `def`, 15 phpdoc tag chars, 12 keyword |
+
+Every remaining difference is one of five deliberate categories:
+
+1. **`ident` → `def`.** The definition's own name now carries weight 600, everywhere rather than
+   only on the line the Symbol view opened at. It comes from the extractors' own definition tables
+   (`functionTypes`, `classTypes`, `methodTypes`, …) plus each language's `nameField`, so it cannot
+   drift from what indexing considers a definition.
+2. **`string` → code, inside an interpolation.** A template literal's `${…}`, an f-string's `{…}`,
+   Ruby's `#{…}` and C#'s `$"{…}"` are classified as code. This is the one difference that is not
+   cosmetic: the call-site overlay deliberately refuses to claim a token classed `string`, so
+   **calls inside interpolated strings now link and did not before.**
+3. **`keyword` → `type`, on built-in type words.** `string`, `int`, `u32`, `void`. The grammars
+   disagree with each other about what a built-in type is — tree-sitter-go calls `string` a
+   `type_identifier`, tree-sitter-typescript wraps it in a `predefined_type` whose child is an
+   anonymous token spelled `string` — and TextMate scoped them inconsistently too (plain in
+   TypeScript, `storage.type` in Go). They now all paint at plain ink, like a user-defined type
+   name, in every language.
+4. **Keyword-set corrections.** Python's `is not`, Rust's and Swift's modifiers, and Ruby's `new`
+   (which is a method, not a keyword — TextMate's `keyword.operator.new` matched it anyway).
+5. **`keyword` → `comment`, on phpdoc tags.** `@var` and friends recede with the comment they are
+   in, which is what the near-monochrome ramp asks for.
+
+## What is no longer highlighted
+
+Nine formats have extraction but no tree-sitter grammar. Three of them — `.svelte`, `.vue`,
+`.astro` — are classified through their `<script>` blocks with TypeScript or JavaScript, the same
+delegation the extractors do, so every symbol the engine indexed in those files is highlighted and
+the surrounding markup is not. The other six (Liquid, Razor, YAML, Twig, XML, `.properties`) render
+plain, where Shiki had grammars for them.
+
+That is a real, deliberate loss, and it is the alternative to a worse one. `tree-sitter-wasms`
+ships an `html` grammar that would cover most of them, but the ABI-13 builds in that package are
+the known cause of a shared-WASM-heap corruption that silently drops edges for *every other*
+language in the same process (see `VENDORED_WASM_LANGS` in `src/extraction/grammars.ts`), and the
+viewer runs in a process someone leaves open all day. Adding unvetted grammars to buy tag colouring
+on config files is not a trade worth making. Identifiers are still split out on those files, so the
+graph's call-site links land exactly as they do everywhere else — highlighting is the part that
+degrades, never the linking.
+
+## Reproducing this
+
+There is no committed harness: the "before" side needs the deleted Shiki module. Rebuild it from
+the last commit that had it —
+
+```
+git worktree add /tmp/cg48-baseline <ref-with-shiki>
+ln -s "$PWD/node_modules" /tmp/cg48-baseline/node_modules   # @shikijs/* must still be installed
+( cd /tmp/cg48-baseline && npx tsc && node scripts/prune-grammars.mjs )
+```
+
+— then run both `dist/ui-server/highlight/index.js` modules over the same lines and compare
+`classes[id]` per character. The screenshots were rendered from the same two token streams through
+the viewer's own token CSS at `--force-device-scale-factor=2`.

docs/design/codegraph-ui-design-spec.md

@@ -321,22 +321,32 @@ with `src/index.ts` selected, 15 links and 4 dimmed boxes, matching the canvas).
 - Svelte 5 (≥ 5.25) + Vite (workspace `ui/`), Svelte Flow `@xyflow/svelte` ^1.6 for the Map and Flow canvases only (custom nodes/edges,
   hidden handles for port spreading, local selection state — the pattern in docker-app's `StackGraph.svelte`); `@dagrejs/dagre` only as a
   fallback if crossing quality demands it (never ELK). Symbol view = DOM + one SVG overlay (`ResizeObserver` re-layout).
-- Shiki (JavaScript regex engine, lazy grammars, custom near-monochrome theme as in §2.2) server-side in `/api/source`; tree-sitter-derived
-  tokens replace it in phase 3.
-  - *As built (CG-43).* `@shikijs/core` + `@shikijs/engine-javascript` are runtime dependencies (~5 MB installed, no wasm, no native
-    module); `@shikijs/langs` is a **devDependency** and `npm run build:textmate` (`scripts/prune-grammars.mjs`) writes only the
-    closure the engine's 40-odd languages reach — 56 grammars, 2.6 MB — into **`dist/textmate/`**, checked by `scripts/check-ui-build.mjs`.
-    Shipping all 722 grammars would have been 11 MB.
-  - The theme classifies rather than colours: its foregrounds are sentinels the server maps to class names (`comment`, `string`,
-    `keyword`, `number`, `ident`, `other`), and the viewer paints them from the CSS custom properties above — so **one token stream
-    serves light and dark** with no refetch when `prefers-color-scheme` flips, and the ramp lives only in `ui/src/app.css`.
+- Syntax classification comes off **the engine's own tree-sitter parse** — no highlighter dependency, no second grammar set.
+  - *As built (CG-43, replaced in CG-57).* The first cut ran Shiki with 56 pruned TextMate grammars in `dist/textmate/`. That is
+    gone: `@shikijs/*` is off the dependency list, `scripts/prune-grammars.mjs` and `npm run build:textmate` are deleted, and
+    `scripts/check-ui-build.mjs` now asserts the tree-sitter grammars in `dist/extraction/wasm/` instead. A `.ts` file is read by
+    exactly the grammar that decided what its symbols are, so the viewer and the graph can never disagree about it.
+  - Eight token classes on the wire: `comment`, `string`, `number`, `keyword`, `type`, `def`, `ident`, `other`. Rules, not scope
+    tables — a node whose type mentions `comment` is a comment; inside a string every leaf is string *except* below an
+    interpolation, where code resumes (so `${user.name()}` still links); an **anonymous** leaf is a keyword when its text is a bare
+    word and punctuation otherwise; a **named** leaf is an identifier, a type name, or — from the extractors' own definition
+    tables — the name a definition declares. `punct` is folded into `other`: they paint identically and splitting them would
+    roughly double the token count on a dense line.
+  - The classification is a class NAME, never a colour, and the viewer paints it from the CSS custom properties above — so **one
+    token stream serves light and dark** with no refetch when `prefers-color-scheme` flips, and the ramp lives only in
+    `ui/src/app.css`. `type` is a distinct class painted at plain ink: the colouring is near-monochrome and a type name is not one
+    of the four things it moves off plain ink.
   - Every code token is split into identifier runs before it goes on the wire, so the graph's call-site overlay claims a token the
-    highlighter produced rather than re-cutting a line — which is what keeps a link landing on the callee's own name whatever
+    classifier produced rather than re-cutting a line — which is what keeps a link landing on the callee's own name whatever
     boundaries a grammar chose, and keeps links working in the plain-text fallback.
-  - Measured on this machine (Shiki 4.4.3, JS regex engine, 3 000 lines cold): Python 35–47 ms, Go 43–57 ms, **TypeScript ~700 ms** —
-    the TS TextMate grammar is 5–7× the cost of any other and the oniguruma-wasm engine would run it in ~120 ms. Slices are therefore
-    cached by content hash + range, so a re-render (resize, theme flip, stepping back through the trail) is a map lookup (< 10 ms);
-    a symbol-sized slice (~280 lines of TS) is ~50 ms cold. Phase 1 only ever requests one symbol's range.
+  - Single-file components (`.svelte`, `.vue`, `.astro`) have no grammar of their own; their `<script>` blocks — where every
+    indexed symbol in those files lives — are classified as TypeScript or JavaScript, exactly the delegation the extractors
+    already do. The surrounding markup, and the config formats with file-level extraction only (YAML, XML, Twig, properties),
+    render plain with their identifiers still split out, so links land there too.
+  - Measured on this machine, 3 000 lines cold: **TypeScript 24–41 ms** (it was ~700 ms under Shiki, whose TS grammar cost 5–7×
+    every other one), Go ~30 ms, Python 25–29 ms, and Rust/Ruby/PHP/C#/Swift 14–27 ms. Slices are still cached by content hash +
+    range, so a re-render (resize, theme flip, stepping back through the trail) is a map lookup. Side-by-side parity screenshots
+    for the eight gate languages: `docs/design/cg57-highlighting-parity/`.
 - No native modules; no runtime dependency for the UI itself; the CLI serves **`dist/viewer/`** over `node:http`, loopback only.
   (Not `dist/ui/` — `src/ui/` is the engine's *terminal* ui and tsc already compiles it there; see `ui/README.md`.)
 

package-lock.json

@@ -13,8 +13,6 @@
       ],
       "dependencies": {
         "@clack/prompts": "^1.3.0",
-        "@shikijs/core": "^4.4.3",
-        "@shikijs/engine-javascript": "^4.4.3",
         "commander": "^14.0.2",
         "fast-string-width": "^3.0.2",
         "fast-wrap-ansi": "^0.2.0",
@@ -29,7 +27,6 @@
         "codegraph": "dist/bin/codegraph.js"
       },
       "devDependencies": {
-        "@shikijs/langs": "^4.4.3",
         "@types/better-sqlite3": "^7.6.0",
         "@types/node": "^20.19.30",
         "@types/picomatch": "^4.0.2",
@@ -930,82 +927,6 @@
         "win32"
       ]
     },
-    "node_modules/@shikijs/core": {
-      "version": "4.4.3",
-      "resolved": "https://registry.npmjs.org/@shikijs/core/-/core-4.4.3.tgz",
-      "integrity": "sha512-QCR4q2ZO/ILJEuwiBMel4wdcTDb1JGwfjKTxPDF6x8ixOaluPrVqIn06C99AcRPhmYlBR56d/Fb+GN58GzExpg==",
-      "license": "MIT",
-      "dependencies": {
-        "@shikijs/primitive": "4.4.3",
-        "@shikijs/types": "4.4.3",
-        "@shikijs/vscode-textmate": "^10.0.2",
-        "@types/hast": "^3.0.5",
-        "hast-util-to-html": "^9.0.5"
-      },
-      "engines": {
-        "node": ">=20"
-      }
-    },
-    "node_modules/@shikijs/engine-javascript": {
-      "version": "4.4.3",
-      "resolved": "https://registry.npmjs.org/@shikijs/engine-javascript/-/engine-javascript-4.4.3.tgz",
-      "integrity": "sha512-FbOjFJp9VLdo1Wevs10BBtVxiTWwNLqZh5Gkhjgda/ioL15YOgeSl9n+6XMa3qRlPQzfhFNe641SrynFHYG0nQ==",
-      "license": "MIT",
-      "dependencies": {
-        "@shikijs/types": "4.4.3",
-        "@shikijs/vscode-textmate": "^10.0.2",
-        "oniguruma-to-es": "^4.3.6"
-      },
-      "engines": {
-        "node": ">=20"
-      }
-    },
-    "node_modules/@shikijs/langs": {
-      "version": "4.4.3",
-      "resolved": "https://registry.npmjs.org/@shikijs/langs/-/langs-4.4.3.tgz",
-      "integrity": "sha512-ePic0yfAJGOF83D5wBHK/00EjK65oahBYxFk5epgq33WRv7X9UuxLEV8PtR0szC0z8dl7INIpIodB99JRFlR+A==",
-      "dev": true,
-      "license": "MIT",
-      "dependencies": {
-        "@shikijs/types": "4.4.3"
-      },
-      "engines": {
-        "node": ">=20"
-      }
-    },
-    "node_modules/@shikijs/primitive": {
-      "version": "4.4.3",
-      "resolved": "https://registry.npmjs.org/@shikijs/primitive/-/primitive-4.4.3.tgz",
-      "integrity": "sha512-m0wBeLDQDeIxRdUmrCPdQqfuUamDwRL5isCfYbguKD6NiaKpVbsv+3J81DyIKgNW5h4WAIIr8T4EkgQrBBxvaQ==",
-      "license": "MIT",
-      "dependencies": {
-        "@shikijs/types": "4.4.3",
-        "@shikijs/vscode-textmate": "^10.0.2",
-        "@types/hast": "^3.0.5"
-      },
-      "engines": {
-        "node": ">=20"
-      }
-    },
-    "node_modules/@shikijs/types": {
-      "version": "4.4.3",
-      "resolved": "https://registry.npmjs.org/@shikijs/types/-/types-4.4.3.tgz",
-      "integrity": "sha512-UEJxmRR++MAGR6hugn0vgVS2W/6lWAts84FFSrnlH9sP0LNol7E5+NQ792pH8liWUhyMyjhTgSUH3k7iD7tc5g==",
-      "license": "MIT",
-      "dependencies": {
-        "@shikijs/vscode-textmate": "^10.0.2",
-        "@types/hast": "^3.0.5"
-      },
-      "engines": {
-        "node": ">=20"
-      }
-    },
-    "node_modules/@shikijs/vscode-textmate": {
-      "version": "10.0.2",
-      "resolved": "https://registry.npmjs.org/@shikijs/vscode-textmate/-/vscode-textmate-10.0.2.tgz",
-      "integrity": "sha512-83yeghZ2xxin3Nj8z1NMd/NCuca+gsYXswywDy5bHvwlWL8tpTQmzGeUuHd9FC3E/SBEMvzJRwWEOz5gGes9Qg==",
-      "license": "MIT"
-    },
     "node_modules/@svelte-put/shortcut": {
       "version": "4.2.0",
       "resolved": "https://registry.npmjs.org/@svelte-put/shortcut/-/shortcut-4.2.0.tgz",
@@ -1108,24 +1029,6 @@
       "dev": true,
       "license": "MIT"
     },
-    "node_modules/@types/hast": {
-      "version": "3.0.5",
-      "resolved": "https://registry.npmjs.org/@types/hast/-/hast-3.0.5.tgz",
-      "integrity": "sha512-rp/ezSWaD1m44dPKICGhiskI13nVr7qTloFwDa/IYkhhf5nzwP+zIQcIJh3WIFSBOy/H1PzB40jPjMDksN4F+g==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/unist": "*"
-      }
-    },
-    "node_modules/@types/mdast": {
-      "version": "4.0.4",
-      "resolved": "https://registry.npmjs.org/@types/mdast/-/mdast-4.0.4.tgz",
-      "integrity": "sha512-kGaNbPh1k7AFzgpud/gMdvIm5xuECykRR+JnWKQno9TAXVa6WIVCGTPvYGekIDL4uwCZQSYbUxNBSb1aUo79oA==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/unist": "*"
-      }
-    },
     "node_modules/@types/node": {
       "version": "20.19.33",
       "resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.33.tgz",
@@ -1151,18 +1054,6 @@
       "dev": true,
       "license": "MIT"
     },
-    "node_modules/@types/unist": {
-      "version": "3.0.3",
-      "resolved": "https://registry.npmjs.org/@types/unist/-/unist-3.0.3.tgz",
-      "integrity": "sha512-ko/gIFJRv177XgZsZcBwnqJN5x/Gien8qNOn0D5bQU/zAzVf9Zt3BlcUiLqhV9y4ARk0GbT3tnUiPNgnTXzc/Q==",
-      "license": "MIT"
-    },
-    "node_modules/@ungap/structured-clone": {
-      "version": "1.3.4",
-      "resolved": "https://registry.npmjs.org/@ungap/structured-clone/-/structured-clone-1.3.4.tgz",
-      "integrity": "sha512-JL+CF0GeLHyPWI0rXu7UnxgiuOm9UQWzadi0OYOJNhNO2q6EZElpwlgXkNkfU1PzANDHq3YcwKVZprdvS+BrbQ==",
-      "license": "ISC"
-    },
     "node_modules/@vitest/expect": {
       "version": "2.1.9",
       "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-2.1.9.tgz",
@@ -1362,16 +1253,6 @@
         "node": ">=8"
       }
     },
-    "node_modules/ccount": {
-      "version": "2.0.1",
-      "resolved": "https://registry.npmjs.org/ccount/-/ccount-2.0.1.tgz",
-      "integrity": "sha512-eyrF0jiFpY+3drT6383f1qhkbGsLSifNAjA61IUjZjmLCWjItY6LB9ft9YhoDgwfmclB2zhu51Lc7+95b8NRAg==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
     "node_modules/chai": {
       "version": "5.3.3",
       "resolved": "https://registry.npmjs.org/chai/-/chai-5.3.3.tgz",
@@ -1389,26 +1270,6 @@
         "node": ">=18"
       }
     },
-    "node_modules/character-entities-html4": {
-      "version": "2.1.0",
-      "resolved": "https://registry.npmjs.org/character-entities-html4/-/character-entities-html4-2.1.0.tgz",
-      "integrity": "sha512-1v7fgQRj6hnSwFpq1Eu0ynr/CDEw0rXo2B61qXrLNdHZmPKgb7fqS1a2JwF0rISo9q77jDI8VMEHoApn8qDoZA==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
-    "node_modules/character-entities-legacy": {
-      "version": "3.0.0",
-      "resolved": "https://registry.npmjs.org/character-entities-legacy/-/character-entities-legacy-3.0.0.tgz",
-      "integrity": "sha512-RpPp0asT/6ufRm//AJVwpViZbGM/MkjQFxJccQRHmISF/22NBtsHqAWmL+/pmkPWoIUJdWyeVleTl1wydHATVQ==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
     "node_modules/check-error": {
       "version": "2.1.3",
       "resolved": "https://registry.npmjs.org/check-error/-/check-error-2.1.3.tgz",
@@ -1449,16 +1310,6 @@
       "resolved": "ui",
       "link": true
     },
-    "node_modules/comma-separated-tokens": {
-      "version": "2.0.3",
-      "resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.3.tgz",
-      "integrity": "sha512-Fu4hJdvzeylCfQPp9SGWidpzrMs7tTrlu6Vb8XGaRGck8QSNZJJp538Wrb60Lax4fPwR64ViY468OIUTbRlGZg==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
     "node_modules/commander": {
       "version": "14.0.3",
       "resolved": "https://registry.npmjs.org/commander/-/commander-14.0.3.tgz",
@@ -1621,15 +1472,6 @@
         "node": ">=0.10.0"
       }
     },
-    "node_modules/dequal": {
-      "version": "2.0.3",
-      "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz",
-      "integrity": "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=6"
-      }
-    },
     "node_modules/devalue": {
       "version": "5.9.1",
       "resolved": "https://registry.npmjs.org/devalue/-/devalue-5.9.1.tgz",
@@ -1637,19 +1479,6 @@
       "dev": true,
       "license": "MIT"
     },
-    "node_modules/devlop": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/devlop/-/devlop-1.1.0.tgz",
-      "integrity": "sha512-RWmIqhcFf1lRYBvNmr7qTNuyCt/7/ns2jbpp1+PalgE/rDQcBT0fioSMUpJ93irlUhC5hrg4cYqe6U+0ImW0rA==",
-      "license": "MIT",
-      "dependencies": {
-        "dequal": "^2.0.0"
-      },
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
     "node_modules/es-module-lexer": {
       "version": "1.7.0",
       "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz",
@@ -1798,52 +1627,6 @@
         "node": "^8.16.0 || ^10.6.0 || >=11.0.0"
       }
     },
-    "node_modules/hast-util-to-html": {
-      "version": "9.0.5",
-      "resolved": "https://registry.npmjs.org/hast-util-to-html/-/hast-util-to-html-9.0.5.tgz",
-      "integrity": "sha512-OguPdidb+fbHQSU4Q4ZiLKnzWo8Wwsf5bZfbvu7//a9oTYoqD/fWpe96NuHkoS9h0ccGOTe0C4NGXdtS0iObOw==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/hast": "^3.0.0",
-        "@types/unist": "^3.0.0",
-        "ccount": "^2.0.0",
-        "comma-separated-tokens": "^2.0.0",
-        "hast-util-whitespace": "^3.0.0",
-        "html-void-elements": "^3.0.0",
-        "mdast-util-to-hast": "^13.0.0",
-        "property-information": "^7.0.0",
-        "space-separated-tokens": "^2.0.0",
-        "stringify-entities": "^4.0.0",
-        "zwitch": "^2.0.4"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/unified"
-      }
-    },
-    "node_modules/hast-util-whitespace": {
-      "version": "3.0.0",
-      "resolved": "https://registry.npmjs.org/hast-util-whitespace/-/hast-util-whitespace-3.0.0.tgz",
-      "integrity": "sha512-88JUN06ipLwsnv+dVn+OIYOvAuvBMy/Qoi6O7mQHxdPXpjy+Cd6xRkWwux7DKO+4sYILtLBRIKgsdpS2gQc7qw==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/hast": "^3.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/unified"
-      }
-    },
-    "node_modules/html-void-elements": {
-      "version": "3.0.0",
-      "resolved": "https://registry.npmjs.org/html-void-elements/-/html-void-elements-3.0.0.tgz",
-      "integrity": "sha512-bEqo66MRXsUGxWHV5IP0PUiAWwoEjba4VCzg0LjFJBpchPaTfyfCKTG6bc5F8ucKec3q5y6qOdGyYTSBEvhCrg==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
     "node_modules/ignore": {
       "version": "7.0.5",
       "resolved": "https://registry.npmjs.org/ignore/-/ignore-7.0.5.tgz",
@@ -1893,116 +1676,6 @@
         "@jridgewell/sourcemap-codec": "^1.5.5"
       }
     },
-    "node_modules/mdast-util-to-hast": {
-      "version": "13.2.1",
-      "resolved": "https://registry.npmjs.org/mdast-util-to-hast/-/mdast-util-to-hast-13.2.1.tgz",
-      "integrity": "sha512-cctsq2wp5vTsLIcaymblUriiTcZd0CwWtCbLvrOzYCDZoWyMNV8sZ7krj09FSnsiJi3WVsHLM4k6Dq/yaPyCXA==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/hast": "^3.0.0",
-        "@types/mdast": "^4.0.0",
-        "@ungap/structured-clone": "^1.0.0",
-        "devlop": "^1.0.0",
-        "micromark-util-sanitize-uri": "^2.0.0",
-        "trim-lines": "^3.0.0",
-        "unist-util-position": "^5.0.0",
-        "unist-util-visit": "^5.0.0",
-        "vfile": "^6.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/unified"
-      }
-    },
-    "node_modules/micromark-util-character": {
-      "version": "2.1.1",
-      "resolved": "https://registry.npmjs.org/micromark-util-character/-/micromark-util-character-2.1.1.tgz",
-      "integrity": "sha512-wv8tdUTJ3thSFFFJKtpYKOYiGP2+v96Hvk4Tu8KpCAsTMs6yi+nVmGh1syvSCsaxz45J6Jbw+9DD6g97+NV67Q==",
-      "funding": [
-        {
-          "type": "GitHub Sponsors",
-          "url": "https://github.com/sponsors/unifiedjs"
-        },
-        {
-          "type": "OpenCollective",
-          "url": "https://opencollective.com/unified"
-        }
-      ],
-      "license": "MIT",
-      "dependencies": {
-        "micromark-util-symbol": "^2.0.0",
-        "micromark-util-types": "^2.0.0"
-      }
-    },
-    "node_modules/micromark-util-encode": {
-      "version": "2.0.1",
-      "resolved": "https://registry.npmjs.org/micromark-util-encode/-/micromark-util-encode-2.0.1.tgz",
-      "integrity": "sha512-c3cVx2y4KqUnwopcO9b/SCdo2O67LwJJ/UyqGfbigahfegL9myoEFoDYZgkT7f36T0bLrM9hZTAaAyH+PCAXjw==",
-      "funding": [
-        {
-          "type": "GitHub Sponsors",
-          "url": "https://github.com/sponsors/unifiedjs"
-        },
-        {
-          "type": "OpenCollective",
-          "url": "https://opencollective.com/unified"
-        }
-      ],
-      "license": "MIT"
-    },
-    "node_modules/micromark-util-sanitize-uri": {
-      "version": "2.0.1",
-      "resolved": "https://registry.npmjs.org/micromark-util-sanitize-uri/-/micromark-util-sanitize-uri-2.0.1.tgz",
-      "integrity": "sha512-9N9IomZ/YuGGZZmQec1MbgxtlgougxTodVwDzzEouPKo3qFWvymFHWcnDi2vzV1ff6kas9ucW+o3yzJK9YB1AQ==",
-      "funding": [
-        {
-          "type": "GitHub Sponsors",
-          "url": "https://github.com/sponsors/unifiedjs"
-        },
-        {
-          "type": "OpenCollective",
-          "url": "https://opencollective.com/unified"
-        }
-      ],
-      "license": "MIT",
-      "dependencies": {
-        "micromark-util-character": "^2.0.0",
-        "micromark-util-encode": "^2.0.0",
-        "micromark-util-symbol": "^2.0.0"
-      }
-    },
-    "node_modules/micromark-util-symbol": {
-      "version": "2.0.1",
-      "resolved": "https://registry.npmjs.org/micromark-util-symbol/-/micromark-util-symbol-2.0.1.tgz",
-      "integrity": "sha512-vs5t8Apaud9N28kgCrRUdEed4UJ+wWNvicHLPxCa9ENlYuAY31M0ETy5y1vA33YoNPDFTghEbnh6efaE8h4x0Q==",
-      "funding": [
-        {
-          "type": "GitHub Sponsors",
-          "url": "https://github.com/sponsors/unifiedjs"
-        },
-        {
-          "type": "OpenCollective",
-          "url": "https://opencollective.com/unified"
-        }
-      ],
-      "license": "MIT"
-    },
-    "node_modules/micromark-util-types": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/micromark-util-types/-/micromark-util-types-2.0.2.tgz",
-      "integrity": "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA==",
-      "funding": [
-        {
-          "type": "GitHub Sponsors",
-          "url": "https://github.com/sponsors/unifiedjs"
-        },
-        {
-          "type": "OpenCollective",
-          "url": "https://opencollective.com/unified"
-        }
-      ],
-      "license": "MIT"
-    },
     "node_modules/mri": {
       "version": "1.2.0",
       "resolved": "https://registry.npmjs.org/mri/-/mri-1.2.0.tgz",
@@ -2053,23 +1726,6 @@
         "node": ">=12.20.0"
       }
     },
-    "node_modules/oniguruma-parser": {
-      "version": "0.12.2",
-      "resolved": "https://registry.npmjs.org/oniguruma-parser/-/oniguruma-parser-0.12.2.tgz",
-      "integrity": "sha512-6HVa5oIrgMC6aA6WF6XyyqbhRPJrKR02L20+2+zpDtO5QAzGHAUGw5TKQvwi5vctNnRHkJYmjAhRVQF2EKdTQw==",
-      "license": "MIT"
-    },
-    "node_modules/oniguruma-to-es": {
-      "version": "4.3.6",
-      "resolved": "https://registry.npmjs.org/oniguruma-to-es/-/oniguruma-to-es-4.3.6.tgz",
-      "integrity": "sha512-csuQ9x3Yr0cEIs/Zgx/OEt9iBw9vqIunAPQkx19R/fiMq2oGVTgcMqO/V3Ybqefr1TBvosI6jU539ksaBULJyA==",
-      "license": "MIT",
-      "dependencies": {
-        "oniguruma-parser": "^0.12.2",
-        "regex": "^6.1.0",
-        "regex-recursion": "^6.0.2"
-      }
-    },
     "node_modules/pathe": {
       "version": "1.1.2",
       "resolved": "https://registry.npmjs.org/pathe/-/pathe-1.1.2.tgz",
@@ -2136,16 +1792,6 @@
         "node": "^10 || ^12 || >=14"
       }
     },
-    "node_modules/property-information": {
-      "version": "7.2.0",
-      "resolved": "https://registry.npmjs.org/property-information/-/property-information-7.2.0.tgz",
-      "integrity": "sha512-IAtzIB6sUiWaJYrX9smp3V46pBGbBeLFRGdh25kg1334VcBlD8HzhPeNIWQH9zhGmo2itIe25EHt9dQP7G5hmg==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
     "node_modules/readdirp": {
       "version": "4.1.2",
       "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-4.1.2.tgz",
@@ -2160,30 +1806,6 @@
         "url": "https://paulmillr.com/funding/"
       }
     },
-    "node_modules/regex": {
-      "version": "6.1.0",
-      "resolved": "https://registry.npmjs.org/regex/-/regex-6.1.0.tgz",
-      "integrity": "sha512-6VwtthbV4o/7+OaAF9I5L5V3llLEsoPyq9P1JVXkedTP33c7MfCG0/5NOPcSJn0TzXcG9YUrR0gQSWioew3LDg==",
-      "license": "MIT",
-      "dependencies": {
-        "regex-utilities": "^2.3.0"
-      }
-    },
-    "node_modules/regex-recursion": {
-      "version": "6.0.2",
-      "resolved": "https://registry.npmjs.org/regex-recursion/-/regex-recursion-6.0.2.tgz",
-      "integrity": "sha512-0YCaSCq2VRIebiaUviZNs0cBz1kg5kVS2UKUfNIx8YVs1cN3AV7NTctO5FOKBA+UT2BPJIWZauYHPqJODG50cg==",
-      "license": "MIT",
-      "dependencies": {
-        "regex-utilities": "^2.3.0"
-      }
-    },
-    "node_modules/regex-utilities": {
-      "version": "2.3.0",
-      "resolved": "https://registry.npmjs.org/regex-utilities/-/regex-utilities-2.3.0.tgz",
-      "integrity": "sha512-8VhliFJAWRaUiVvREIiW2NXXTmHs4vMNnSzuJVhscgmGav3g9VDxLrQndI3dZZVVdp0ZO/5v0xmX516/7M9cng==",
-      "license": "MIT"
-    },
     "node_modules/rollup": {
       "version": "4.57.1",
       "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.57.1.tgz",
@@ -2265,16 +1887,6 @@
         "node": ">=0.10.0"
       }
     },
-    "node_modules/space-separated-tokens": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/space-separated-tokens/-/space-separated-tokens-2.0.2.tgz",
-      "integrity": "sha512-PEGlAwrG8yXGXRjW32fGbg66JAlOAwbObuqVoJpv/mRgoWDQfgH1wDPvtzWyUSNAXBGSk8h755YDbbcEy3SH2Q==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
     "node_modules/stackback": {
       "version": "0.0.2",
       "resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz",
@@ -2289,20 +1901,6 @@
       "dev": true,
       "license": "MIT"
     },
-    "node_modules/stringify-entities": {
-      "version": "4.0.4",
-      "resolved": "https://registry.npmjs.org/stringify-entities/-/stringify-entities-4.0.4.tgz",
-      "integrity": "sha512-IwfBptatlO+QCJUo19AqvrPNqlVMpW9YEL2LIVY+Rpv2qsjCGxaDLNRgeGsQWJhfItebuJhsGSLjaBbNSQ+ieg==",
-      "license": "MIT",
-      "dependencies": {
-        "character-entities-html4": "^2.0.0",
-        "character-entities-legacy": "^3.0.0"
-      },
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
     "node_modules/svelte": {
       "version": "5.56.10",
       "resolved": "https://registry.npmjs.org/svelte/-/svelte-5.56.10.tgz",
@@ -2427,16 +2025,6 @@
         "tree-sitter-wasms": "^0.1.11"
       }
     },
-    "node_modules/trim-lines": {
-      "version": "3.0.1",
-      "resolved": "https://registry.npmjs.org/trim-lines/-/trim-lines-3.0.1.tgz",
-      "integrity": "sha512-kRj8B+YHZCc9kQYdWfJB2/oUl9rA99qbowYYBtr4ui4mZyAQ2JpvVBd/6U2YloATfqBhBTSMhTpgBHtU0Mf3Rg==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
     "node_modules/typescript": {
       "version": "5.9.3",
       "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
@@ -2459,102 +2047,6 @@
       "dev": true,
       "license": "MIT"
     },
-    "node_modules/unist-util-is": {
-      "version": "6.0.1",
-      "resolved": "https://registry.npmjs.org/unist-util-is/-/unist-util-is-6.0.1.tgz",
-      "integrity": "sha512-LsiILbtBETkDz8I9p1dQ0uyRUWuaQzd/cuEeS1hoRSyW5E5XGmTzlwY1OrNzzakGowI9Dr/I8HVaw4hTtnxy8g==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/unist": "^3.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/unified"
-      }
-    },
-    "node_modules/unist-util-position": {
-      "version": "5.0.0",
-      "resolved": "https://registry.npmjs.org/unist-util-position/-/unist-util-position-5.0.0.tgz",
-      "integrity": "sha512-fucsC7HjXvkB5R3kTCO7kUjRdrS0BJt3M/FPxmHMBOm8JQi2BsHAHFsy27E0EolP8rp0NzXsJ+jNPyDWvOJZPA==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/unist": "^3.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/unified"
-      }
-    },
-    "node_modules/unist-util-stringify-position": {
-      "version": "4.0.0",
-      "resolved": "https://registry.npmjs.org/unist-util-stringify-position/-/unist-util-stringify-position-4.0.0.tgz",
-      "integrity": "sha512-0ASV06AAoKCDkS2+xw5RXJywruurpbC4JZSm7nr7MOt1ojAzvyyaO+UxZf18j8FCF6kmzCZKcAgN/yu2gm2XgQ==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/unist": "^3.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/unified"
-      }
-    },
-    "node_modules/unist-util-visit": {
-      "version": "5.1.0",
-      "resolved": "https://registry.npmjs.org/unist-util-visit/-/unist-util-visit-5.1.0.tgz",
-      "integrity": "sha512-m+vIdyeCOpdr/QeQCu2EzxX/ohgS8KbnPDgFni4dQsfSCtpz8UqDyY5GjRru8PDKuYn7Fq19j1CQ+nJSsGKOzg==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/unist": "^3.0.0",
-        "unist-util-is": "^6.0.0",
-        "unist-util-visit-parents": "^6.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/unified"
-      }
-    },
-    "node_modules/unist-util-visit-parents": {
-      "version": "6.0.2",
-      "resolved": "https://registry.npmjs.org/unist-util-visit-parents/-/unist-util-visit-parents-6.0.2.tgz",
-      "integrity": "sha512-goh1s1TBrqSqukSc8wrjwWhL0hiJxgA8m4kFxGlQ+8FYQ3C/m11FcTs4YYem7V664AhHVvgoQLk890Ssdsr2IQ==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/unist": "^3.0.0",
-        "unist-util-is": "^6.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/unified"
-      }
-    },
-    "node_modules/vfile": {
-      "version": "6.0.3",
-      "resolved": "https://registry.npmjs.org/vfile/-/vfile-6.0.3.tgz",
-      "integrity": "sha512-KzIbH/9tXat2u30jf+smMwFCsno4wHVdNmzFyL+T/L3UGqqk6JKfVqOFOZEpZSHADH1k40ab6NUIXZq422ov3Q==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/unist": "^3.0.0",
-        "vfile-message": "^4.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/unified"
-      }
-    },
-    "node_modules/vfile-message": {
-      "version": "4.0.3",
-      "resolved": "https://registry.npmjs.org/vfile-message/-/vfile-message-4.0.3.tgz",
-      "integrity": "sha512-QTHzsGd1EhbZs4AsQ20JX1rC3cOlt/IWJruk893DfLRr57lcnOeMaWG4K0JrRta4mIJZKth2Au3mM3u03/JWKw==",
-      "license": "MIT",
-      "dependencies": {
-        "@types/unist": "^3.0.0",
-        "unist-util-stringify-position": "^4.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/unified"
-      }
-    },
     "node_modules/vite": {
       "version": "5.4.21",
       "resolved": "https://registry.npmjs.org/vite/-/vite-5.4.21.tgz",
@@ -2763,16 +2255,6 @@
       "dev": true,
       "license": "MIT"
     },
-    "node_modules/zwitch": {
-      "version": "2.0.4",
-      "resolved": "https://registry.npmjs.org/zwitch/-/zwitch-2.0.4.tgz",
-      "integrity": "sha512-bXE4cR/kVZhKZX/RjPEflHaKVhUVl85noU3v6b8apfQEc1x4A+zBxjZ4lN8LqGd6WZ3dl98pY4o717VFmoPp+A==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/wooorm"
-      }
-    },
     "ui": {
       "name": "codegraph-ui",
       "version": "0.0.0",

package.json

@@ -20,7 +20,7 @@
     "ui"
   ],
   "scripts": {
-    "build": "tsc && npm run copy-assets && npm run build:textmate && npm run build:ui && node -e \"require('fs').chmodSync('dist/bin/codegraph.js', 0o755)\"",
+    "build": "tsc && npm run copy-assets && npm run build:ui && node -e \"require('fs').chmodSync('dist/bin/codegraph.js', 0o755)\"",
     "build:ui": "npm run build --workspace ui && node scripts/check-ui-build.mjs",
     "preuninstall": "node dist/bin/uninstall.js",
     "copy-assets": "node -e \"const fs=require('fs');fs.mkdirSync('dist/db',{recursive:true});fs.copyFileSync('src/db/schema.sql','dist/db/schema.sql');fs.mkdirSync('dist/extraction/wasm',{recursive:true});fs.readdirSync('src/extraction/wasm').filter(f=>f.endsWith('.wasm')).forEach(f=>fs.copyFileSync('src/extraction/wasm/'+f,'dist/extraction/wasm/'+f))\"",
@@ -31,8 +31,7 @@
     "test:watch": "vitest",
     "test:eval": "vitest run __tests__/evaluation/",
     "eval": "npm run build && npx tsx __tests__/evaluation/runner.ts",
-    "clean": "node -e \"const fs=require('fs');fs.rmSync('dist',{recursive:true,force:true})\"",
-    "build:textmate": "node scripts/prune-grammars.mjs"
+    "clean": "node -e \"const fs=require('fs');fs.rmSync('dist',{recursive:true,force:true})\""
   },
   "keywords": [
     "code-intelligence",
@@ -43,8 +42,6 @@
   "license": "MIT",
   "dependencies": {
     "@clack/prompts": "^1.3.0",
-    "@shikijs/core": "^4.4.3",
-    "@shikijs/engine-javascript": "^4.4.3",
     "commander": "^14.0.2",
     "fast-string-width": "^3.0.2",
     "fast-wrap-ansi": "^0.2.0",
@@ -56,7 +53,6 @@
     "web-tree-sitter": "^0.25.3"
   },
   "devDependencies": {
-    "@shikijs/langs": "^4.4.3",
     "@types/better-sqlite3": "^7.6.0",
     "@types/node": "^20.19.30",
     "@types/picomatch": "^4.0.2",

scripts/check-ui-build.mjs

@@ -13,16 +13,17 @@
  * where tsc puts the TERMINAL ui, so a mis-pointed outDir silently deletes
  * modules the CLI requires at startup.
  *
- * The pruned TextMate grammars in dist/textmate/ are checked the same way and
- * for the same reason: without them every file the viewer shows falls back to
- * unhighlighted text, which looks like a styling bug rather than a missing
- * build step.
+ * The tree-sitter grammars in dist/extraction/wasm/ are checked the same way
+ * and for the same reason. They are copied by `npm run copy-assets`, they are
+ * what both indexing and the viewer's syntax classification parse with, and
+ * their absence is survivable at runtime — source is served unhighlighted —
+ * which is exactly why it has to fail here: nothing downstream would complain.
  *
  * Usage: node scripts/check-ui-build.mjs [--root <dir>]
  *   --root  directory holding dist/ (default: the repo root). The release
  *           bundler points this at its staging dir to verify the copy.
  */
-import { existsSync, readFileSync, statSync } from 'node:fs';
+import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs';
 import { dirname, join, resolve, sep } from 'node:path';
 import { fileURLToPath } from 'node:url';
 
@@ -95,36 +96,62 @@ for (const compiled of [join('bin', 'codegraph.js'), 'index.js', join('ui', 'shi
   }
 }
 
-// The pruned syntax grammars (scripts/prune-grammars.mjs). Their absence is
-// survivable at runtime — source is served unhighlighted — which is exactly why
-// it has to fail here: nothing downstream would ever complain.
-const textmateDir = join(root, 'dist', 'textmate');
-const manifestPath = join(textmateDir, 'manifest.json');
-if (!existsSync(manifestPath)) {
+// The vendored tree-sitter grammars (`npm run copy-assets`). The viewer reads
+// every file with the same grammar the engine indexed it with, so a missing
+// wasm is both an extraction gap and a silently unhighlighted screen.
+const wasmDir = join(root, 'dist', 'extraction', 'wasm');
+
+/**
+ * The grammars the syntax classification is gated on — the eight languages
+ * CG-57 measured parity against, plus the two the TS family needs. Every one is
+ * vendored (see VENDORED_WASM_LANGS), so all of them must be in this directory
+ * rather than resolved out of node_modules.
+ */
+const GATE_GRAMMARS = [
+  'tree-sitter-typescript.wasm',
+  'tree-sitter-tsx.wasm',
+  'tree-sitter-javascript.wasm',
+  'tree-sitter-go.wasm',
+  'tree-sitter-python.wasm',
+  'tree-sitter-rust.wasm',
+  'tree-sitter-swift.wasm',
+  'tree-sitter-c_sharp.wasm',
+  'tree-sitter-ruby.wasm',
+  'tree-sitter-php.wasm',
+];
+
+if (!existsSync(wasmDir)) {
   fail(
-    `missing ${manifestPath}`,
+    `missing ${wasmDir}`,
     staged
-      ? 'this bundle was assembled before the syntax grammars were added, or dist/textmate was not copied'
-      : 'run `npm run build:textmate` (it needs @shikijs/langs from devDependencies)'
+      ? 'dist/extraction/wasm was not copied into the bundle — re-run scripts/build-bundle.sh'
+      : 'run `npm run copy-assets` (it copies src/extraction/wasm/*.wasm into dist/)'
   );
 }
 
-const manifest = JSON.parse(readFileSync(manifestPath, 'utf8'));
-const languages = Object.keys(manifest.languages ?? {});
-if (languages.length === 0) fail('dist/textmate/manifest.json lists no languages');
+// Against the source tree, the source directory IS the list — nothing to drift.
+// Inside a staged bundle there is no src/, so the gate list carries it.
+const expectedGrammars = new Set(GATE_GRAMMARS);
+const srcWasmDir = join(root, 'src', 'extraction', 'wasm');
+if (!staged && existsSync(srcWasmDir)) {
+  for (const name of readdirSync(srcWasmDir)) {
+    if (name.endsWith('.wasm')) expectedGrammars.add(name);
+  }
+}
 
-const grammarFiles = new Set(Object.values(manifest.languages).flat());
-const missingGrammars = [...grammarFiles].filter(
-  (name) => !existsSync(join(textmateDir, `${name}.json`))
+const missingGrammars = [...expectedGrammars].filter(
+  (name) => !existsSync(join(wasmDir, name))
 );
 if (missingGrammars.length > 0) {
   fail(
-    `dist/textmate is missing ${missingGrammars.length} grammar file(s): ${missingGrammars.join(', ')}`,
-    'the prune step was interrupted or dist/textmate was copied incompletely'
+    `dist/extraction/wasm is missing ${missingGrammars.length} grammar(s): ${missingGrammars.join(', ')}`,
+    'the copy-assets step was interrupted or dist/extraction/wasm was copied incompletely'
   );
 }
 
+const grammarCount = readdirSync(wasmDir).filter((n) => n.endsWith('.wasm')).length;
+
 console.log(
   `[check-ui-build] dist/viewer ok (index.html + ${assets} referenced asset(s)); ` +
-    `dist/textmate ok (${languages.length} languages, ${grammarFiles.size} grammars); dist/ engine intact`
+    `dist/extraction/wasm ok (${grammarCount} grammars); dist/ engine intact`
 );

scripts/prune-grammars.mjs

@@ -1,110 +0,0 @@
-#!/usr/bin/env node
-/**
- * Write the TextMate grammars the viewer needs into `dist/textmate/`.
- *
- * Shiki carries about 700 grammars, 11 MB of JSON. The engine indexes about 40
- * languages. Shipping the other 660 to every user of a code-intelligence CLI is
- * not a trade worth making, so `@shikijs/langs` stays a devDependency and this
- * step copies out exactly the closure the viewer can reach: every grammar named
- * in `src/ui-server/highlight/languages.ts`, plus every grammar those embed
- * (`vue` needs html, css, typescript, json and four Vue-specific ones before it
- * will highlight a single-file component).
- *
- * Run from `npm run build`, after `tsc`, because the language table is read
- * from the compiled `dist/ui-server/highlight/languages.js` rather than being
- * duplicated here — one source of truth for what ships.
- *
- * Output:
- *   dist/textmate/manifest.json      grammar id -> files to load, deps first
- *   dist/textmate/<name>.json        one TextMate grammar, verbatim
- */
-
-import { createRequire } from 'node:module';
-import * as fs from 'node:fs';
-import * as path from 'node:path';
-import { fileURLToPath } from 'node:url';
-
-const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
-const OUT = path.join(ROOT, 'dist', 'textmate');
-const require = createRequire(import.meta.url);
-
-function fail(message) {
-  process.stderr.write(`[prune-grammars] ${message}\n`);
-  process.exit(1);
-}
-
-const languagesModule = path.join(ROOT, 'dist', 'ui-server', 'highlight', 'languages.js');
-if (!fs.existsSync(languagesModule)) {
-  fail(`${path.relative(ROOT, languagesModule)} is missing — run tsc before this script.`);
-}
-const { REQUIRED_GRAMMARS } = require(languagesModule);
-if (!Array.isArray(REQUIRED_GRAMMARS) || REQUIRED_GRAMMARS.length === 0) {
-  fail('REQUIRED_GRAMMARS is empty — the language table did not compile as expected.');
-}
-
-const shikiVersion = JSON.parse(
-  fs.readFileSync(path.join(ROOT, 'node_modules', '@shikijs', 'langs', 'package.json'), 'utf-8')
-).version;
-
-/**
- * Load one Shiki language module and return its registrations.
- *
- * The default export is already the flattened chain — embedded grammars first,
- * the language itself last — which is exactly the order Shiki's registry needs
- * to resolve `embeddedLangs`. Keeping that order is the whole reason the
- * manifest stores a list rather than a single filename.
- */
-async function loadChain(id) {
-  const mod = await import(`@shikijs/langs/${id}`);
-  const chain = mod.default;
-  if (!Array.isArray(chain) || chain.length === 0) {
-    fail(`@shikijs/langs/${id} did not export a grammar array.`);
-  }
-  return chain;
-}
-
-fs.rmSync(OUT, { recursive: true, force: true });
-fs.mkdirSync(OUT, { recursive: true });
-
-const manifest = { shikiVersion, languages: {} };
-const written = new Map();
-let bytes = 0;
-
-for (const id of REQUIRED_GRAMMARS) {
-  let chain;
-  try {
-    chain = await loadChain(id);
-  } catch (err) {
-    fail(`could not load the ${id} grammar: ${err?.message ?? err}`);
-  }
-
-  const files = [];
-  for (const grammar of chain) {
-    // A chain can name the same dependency more than once (Vue reaches
-    // JavaScript four different ways). Registering it twice is wasted work and
-    // a confusing manifest; the FIRST occurrence is the one that keeps the
-    // dependencies-before-dependents ordering intact.
-    // `name` is the grammar's own id and is unique across the bundle, so two
-    // languages that embed html write (and share) exactly one html.json.
-    const file = grammar.name;
-    if (typeof file !== 'string' || !/^[\w.+-]+$/.test(file)) {
-      fail(`the ${id} chain contains a grammar with an unusable name: ${JSON.stringify(file)}`);
-    }
-    if (files.includes(file)) continue;
-    if (!written.has(file)) {
-      const json = JSON.stringify(grammar);
-      fs.writeFileSync(path.join(OUT, `${file}.json`), json);
-      written.set(file, json.length);
-      bytes += json.length;
-    }
-    files.push(file);
-  }
-  manifest.languages[id] = files;
-}
-
-fs.writeFileSync(path.join(OUT, 'manifest.json'), JSON.stringify(manifest, null, 2));
-
-process.stdout.write(
-  `[prune-grammars] ${REQUIRED_GRAMMARS.length} languages -> ${written.size} grammars, ` +
-    `${(bytes / 1024 / 1024).toFixed(1)} MB in dist/textmate (shiki ${shikiVersion})\n`
-);

src/extraction/grammars.ts

@@ -561,6 +561,19 @@ function looksLikeObjc(source: string): boolean {
   return /@(?:interface|implementation|protocol|synthesize)\b/.test(sample);
 }
 
+/**
+ * Whether a language has a tree-sitter grammar of its own.
+ *
+ * Narrower than {@link isLanguageSupported}, which also answers true for the
+ * formats handled by custom extractors (SFCs, Liquid, Razor, YAML, XML,
+ * properties) — those have extraction but no grammar, so anything that needs to
+ * PARSE the file (the viewer's syntax classification, for one) has to ask this
+ * instead.
+ */
+export function hasTreeSitterGrammar(language: string | undefined | null): boolean {
+  return !!language && language in WASM_GRAMMAR_FILES;
+}
+
 /**
  * Check if a language is supported (has a grammar defined).
  * Returns true if the grammar exists, even if not yet loaded.

src/extraction/syntax-tokens.ts

@@ -0,0 +1,465 @@
+/**
+ * Syntax classification from the engine's own tree-sitter parse (CG-57).
+ *
+ * The viewer used to run a second highlighter (Shiki + 56 pruned TextMate
+ * grammars) over source the engine had already parsed with a real grammar. This
+ * takes the classification off the tree instead, which removes the second
+ * dependency, the second grammar set, and — the part that actually mattered —
+ * the second opinion: a `.ts` file is now read by exactly the grammar that
+ * decided what its symbols are.
+ *
+ * ## What comes out
+ *
+ * A flat, ordered, non-overlapping list of {@link SyntaxSpan}s over the source
+ * string. Gaps between spans are whitespace and are the caller's to fill. The
+ * classes are deliberately few, because the design's code colouring is
+ * near-monochrome: comments recede, strings and numbers recede one step less,
+ * keywords carry weight rather than hue, and the only colour in the body is a
+ * call site the graph resolved.
+ *
+ * ## How a node becomes a class
+ *
+ * The rules are language-agnostic on purpose — the engine indexes 40-odd
+ * languages and a per-grammar scope table would be 40 tables to keep true:
+ *
+ * * a node whose type mentions `comment` is a comment, whole, undescended;
+ * * inside a string node every leaf is string, *except* below an interpolation,
+ *   where the code starts again (so `${user.name()}` still links);
+ * * a numeric literal node is a number;
+ * * an **anonymous** leaf is a keyword when its text is a bare word and
+ *   punctuation otherwise — this is what makes `func`, `fn`, `def`, `END-IF`
+ *   and `Sub` all land as keywords without naming any of them;
+ * * a **named** leaf whose text is identifier-shaped is an identifier, unless
+ *   the grammar called it a type name, or the extractor's own definition tables
+ *   say it is the name of a definition.
+ *
+ * The last of those is the one place per-language knowledge is used, and it is
+ * reused rather than restated: {@link EXTRACTORS} already names every node type
+ * that declares something in each language, plus the field its name hangs on.
+ */
+
+import type { Node as SyntaxNode } from 'web-tree-sitter';
+import { Language } from '../types';
+import { EXTRACTORS } from './languages';
+import { getParser, loadGrammarsForLanguages } from './grammars';
+import type { LanguageExtractor } from './tree-sitter-types';
+
+/* ------------------------------------------------------------- the classes -- */
+
+/**
+ * Every class a token can carry, in wire order.
+ *
+ * `other` is punctuation and whitespace both. The design spec lists them apart
+ * (`punct` vs the gaps) but they paint identically — plain ink — and splitting
+ * them would roughly double the token count on a dense line to express a
+ * difference nothing draws.
+ */
+export const SYNTAX_TOKEN_CLASSES = [
+  'other',
+  'ident',
+  'comment',
+  'string',
+  'keyword',
+  'number',
+  'type',
+  'def',
+] as const;
+
+export type SyntaxTokenClass = (typeof SYNTAX_TOKEN_CLASSES)[number];
+
+/** A classified run of the source, by JS string index. Half-open. */
+export interface SyntaxSpan {
+  start: number;
+  end: number;
+  cls: SyntaxTokenClass;
+}
+
+/* ---------------------------------------------------------- node-type tests -- */
+
+/**
+ * Anything a grammar calls a comment.
+ *
+ * Substring rather than equality because the spelling is per-grammar:
+ * `comment`, `line_comment`, `block_comment`, `doc_comment`, `html_comment`,
+ * `comment_directive`, `preproc_comment`.
+ */
+function isCommentType(type: string): boolean {
+  return type.includes('comment');
+}
+
+/**
+ * A node whose leaves are string content unless an interpolation interrupts.
+ *
+ * `string` covers the bulk (`string_literal`, `interpreted_string_literal`,
+ * `raw_string_literal`, `encapsed_string`, `string_content`); the rest are the
+ * spellings that avoid the word — Rust/Go/C character literals, shell and PHP
+ * heredocs, and regular expressions, which recede for the same reason a string
+ * does.
+ */
+function isStringType(type: string): boolean {
+  return (
+    type.includes('string') ||
+    type.includes('heredoc') ||
+    type.includes('regex') ||
+    type === 'char_literal' ||
+    type === 'character' ||
+    type === 'character_literal' ||
+    type === 'rune_literal' ||
+    type === 'quoted_attribute_value'
+  );
+}
+
+/**
+ * Where code resumes inside a string.
+ *
+ * A template literal's `${…}` and an f-string's `{…}` hold real expressions,
+ * and the graph records call sites inside them. Swallowing the whole literal as
+ * one string token would drop those links — the overlay refuses to claim a
+ * token classed `string`, deliberately, so that a word inside a message never
+ * gets underlined.
+ */
+function isInterpolationType(type: string): boolean {
+  return (
+    type.includes('interpolation') ||
+    type.includes('substitution') ||
+    type === 'template_substitution' ||
+    type === 'string_interpolation' ||
+    type === 'format_expression'
+  );
+}
+
+/** A numeric literal, plus the language constants a theme groups with them. */
+function isNumberType(type: string): boolean {
+  return (
+    type === 'number' ||
+    type === 'integer' ||
+    type === 'float' ||
+    type === 'number_literal' ||
+    type === 'integer_literal' ||
+    type === 'float_literal' ||
+    type === 'decimal_integer_literal' ||
+    type === 'decimal_floating_point_literal' ||
+    type === 'hex_integer_literal' ||
+    type === 'real_literal' ||
+    type === 'numeric_literal' ||
+    type === 'int_literal' ||
+    type === 'imaginary_literal'
+  );
+}
+
+/** A named type reference — `type_identifier` and the equivalents. */
+function isTypeNameType(type: string): boolean {
+  return type.includes('type_identifier') || type === 'type_name' || type === 'class_type';
+}
+
+/**
+ * Built-in type words — `string`, `int`, `u32`, `void`.
+ *
+ * These are emitted WHOLE and undescended, and they carry the same `type` class
+ * a user-defined type name gets. Both halves of that matter, because the
+ * grammars disagree with each other about what a built-in type even is:
+ * tree-sitter-go calls `string` a `type_identifier` (so it would be a type),
+ * tree-sitter-typescript wraps it in a `predefined_type` whose child is an
+ * anonymous token spelled `string` (so it would be a keyword). Reading the
+ * wrapper rather than its children is what stops the same word from painting
+ * two different ways in two languages on the same screen.
+ */
+const BUILTIN_TYPE_TYPES: ReadonlySet<string> = new Set([
+  'primitive_type',
+  'predefined_type',
+  'builtin_type',
+  'sized_type_specifier',
+]);
+
+/** Literal constants a theme groups with numbers (`constant.language`). */
+const CONSTANT_TYPES: ReadonlySet<string> = new Set([
+  'true',
+  'false',
+  'null',
+  'nil',
+  'none',
+  'undefined',
+  'null_literal',
+  'nil_literal',
+  'boolean_literal',
+  'true_literal',
+  'false_literal',
+]);
+
+/**
+ * Identifier-shaped text, in the loosest sense every indexed language agrees on.
+ *
+ * The high range is there because `\w` is ASCII-only in JavaScript and a symbol
+ * name can be Chinese, Japanese or Cyrillic; a call site in those repositories
+ * has to be linkable too. Hyphens are in because COBOL and Erlang spell words
+ * with them (`END-IF`, `is_record`).
+ */
+const IDENT_SHAPE = /^[A-Za-z_$À-￿][\w$À-￿-]*$/;
+
+/** A bare word — what separates a keyword from punctuation among anonymous nodes. */
+const WORD_SHAPE = /^[A-Za-z_][A-Za-z_0-9-]*$/;
+
+/* ------------------------------------------------------- definition names -- */
+
+/**
+ * Every node type that declares something, per language, from the extractors.
+ *
+ * This is the single piece of per-language knowledge the classifier uses, and
+ * it is borrowed rather than restated: the same lists drive extraction, so a
+ * language that learns a new declaration form gets its name bolded here for
+ * free — and cannot drift, because there is only one list.
+ */
+function definitionTypesFor(extractor: LanguageExtractor): ReadonlySet<string> {
+  return new Set([
+    ...extractor.functionTypes,
+    ...extractor.classTypes,
+    ...extractor.methodTypes,
+    ...extractor.interfaceTypes,
+    ...extractor.structTypes,
+    ...extractor.enumTypes,
+    ...extractor.typeAliasTypes,
+    ...(extractor.unionTypes ?? []),
+    ...(extractor.extraClassNodeTypes ?? []),
+  ]);
+}
+
+/* ------------------------------------------------------------- the walker -- */
+
+interface WalkContext {
+  source: string;
+  out: SyntaxSpan[];
+  defTypes: ReadonlySet<string>;
+  nameField: string;
+  /** Start indices of nodes that are a definition's own name. */
+  defStarts: Set<number>;
+  offset: number;
+}
+
+/**
+ * Classify one parsed tree into spans.
+ *
+ * Exported for tests and for anything that already holds a tree; the usual
+ * entry point is {@link tokenizeSource}, which parses first.
+ */
+export function classifyTree(
+  root: SyntaxNode,
+  source: string,
+  language: Language,
+  offset = 0
+): SyntaxSpan[] {
+  const extractor = EXTRACTORS[language];
+  const ctx: WalkContext = {
+    source,
+    out: [],
+    defTypes: extractor ? definitionTypesFor(extractor) : new Set<string>(),
+    nameField: extractor?.nameField ?? 'name',
+    defStarts: new Set<number>(),
+    offset,
+  };
+  visit(root, ctx, false);
+  return ctx.out;
+}
+
+function visit(node: SyntaxNode, ctx: WalkContext, inString: boolean): void {
+  const type = node.type;
+
+  if (node.isNamed && isCommentType(type)) {
+    emit(ctx, node.startIndex, node.endIndex, 'comment');
+    return;
+  }
+
+  if (node.isNamed && BUILTIN_TYPE_TYPES.has(type)) {
+    emit(ctx, node.startIndex, node.endIndex, 'type');
+    return;
+  }
+
+  // Record the definition's own name BEFORE descending — the name node is a
+  // descendant, so the mark has to be in place by the time the walk reaches it.
+  if (ctx.defTypes.has(type)) {
+    const name = node.childForFieldName(ctx.nameField);
+    if (name) ctx.defStarts.add(name.startIndex);
+  }
+
+  const childCount = node.childCount;
+  if (childCount === 0) {
+    emit(ctx, node.startIndex, node.endIndex, leafClass(node, ctx, inString));
+    return;
+  }
+
+  const nested = isInterpolationType(type) ? false : inString || isStringType(type);
+
+  for (let i = 0; i < childCount; i++) {
+    const child = node.child(i);
+    if (child) visit(child, ctx, nested);
+  }
+}
+
+function leafClass(node: SyntaxNode, ctx: WalkContext, inString: boolean): SyntaxTokenClass {
+  const type = node.type;
+
+  // An ANONYMOUS node's `type` is its own literal text, so none of the
+  // type-name tests below may be applied to one: `key: string` in TypeScript or
+  // PHP is a token whose type is the word `string`, and reading that as a
+  // string literal greys out half of every signature. Anonymous means keyword
+  // or punctuation, decided on shape alone — which is also what makes `func`,
+  // `fn`, `def`, `Sub` and `END-IF` all land right without naming any of them.
+  if (!node.isNamed) {
+    if (inString) return 'string';
+    if (CONSTANT_TYPES.has(type)) return 'number';
+    return WORD_SHAPE.test(type) ? 'keyword' : 'other';
+  }
+
+  if (inString || isStringType(type)) return 'string';
+  if (isNumberType(type) || CONSTANT_TYPES.has(type)) return 'number';
+
+  const text = ctx.source.slice(node.startIndex, node.endIndex);
+  // Ahead of the type tests: a class name is a `type_identifier` in half these
+  // grammars and a plain `identifier` in the other half, and the design bolds
+  // the thing being DECLARED either way.
+  if (ctx.defStarts.has(node.startIndex) && IDENT_SHAPE.test(text)) return 'def';
+  if (isTypeNameType(type)) return 'type';
+  return IDENT_SHAPE.test(text) ? 'ident' : 'other';
+}
+
+/**
+ * Append a span, skipping empties and merging a run of the same class.
+ *
+ * Zero-width nodes are real: every grammar with a layout-sensitive scanner
+ * (Python's `_newline`, Erlang's, Swift's) emits them, and a zero-width span
+ * would put an empty token on the wire for nothing.
+ */
+function emit(ctx: WalkContext, start: number, end: number, cls: SyntaxTokenClass): void {
+  if (end <= start) return;
+  const last = ctx.out[ctx.out.length - 1];
+  const from = start + ctx.offset;
+  if (last && last.cls === cls && last.end === from) {
+    last.end = end + ctx.offset;
+    return;
+  }
+  ctx.out.push({ start: from, end: end + ctx.offset, cls });
+}
+
+/* --------------------------------------------------------------- regions -- */
+
+/**
+ * A stretch of a file written in a different language from the file itself.
+ *
+ * Single-file components are the only case: a `.svelte`, `.vue` or `.astro`
+ * file has no tree-sitter grammar of its own here, but its `<script>` block —
+ * where every symbol the engine indexed in that file lives — is ordinary
+ * TypeScript or JavaScript. The extractors already delegate exactly this way,
+ * so the viewer reads a component's code with the same grammar the graph was
+ * built from. The surrounding markup stays unclassified, which under a
+ * near-monochrome theme costs the recession on tag names and attribute strings
+ * and nothing else.
+ */
+export interface SyntaxRegion {
+  start: number;
+  end: number;
+  language: Language;
+}
+
+const SCRIPT_BLOCK = /<script(\s[^>]*)?>([\s\S]*?)<\/script>/gi;
+const TS_LANG_ATTR = /lang\s*=\s*["'](ts|typescript)["']/i;
+/** Astro's frontmatter: a `---` fence at the very top of the file. */
+const ASTRO_FRONTMATTER = /^(---\r?\n)([\s\S]*?)\r?\n---/;
+
+/**
+ * The sub-language regions of a file, or null when the file is one language.
+ *
+ * Null and an empty array mean different things: null is "parse the whole file
+ * as `language`", empty is "this file has a grammar for none of it".
+ */
+export function syntaxRegionsFor(source: string, language: Language): SyntaxRegion[] | null {
+  if (language !== 'svelte' && language !== 'vue' && language !== 'astro') return null;
+
+  const regions: SyntaxRegion[] = [];
+  if (language === 'astro') {
+    const front = ASTRO_FRONTMATTER.exec(source);
+    if (front && front[2]) {
+      const start = (front[1] as string).length;
+      regions.push({ start, end: start + (front[2] as string).length, language: 'typescript' });
+    }
+  }
+
+  SCRIPT_BLOCK.lastIndex = 0;
+  let match: RegExpExecArray | null;
+  while ((match = SCRIPT_BLOCK.exec(source)) !== null) {
+    const body = match[2] ?? '';
+    if (body.trim() === '') continue;
+    const start = match.index + match[0].length - body.length - '</script>'.length;
+    regions.push({
+      start,
+      end: start + body.length,
+      language: TS_LANG_ATTR.test(match[1] ?? '') ? 'typescript' : 'javascript',
+    });
+  }
+  return regions;
+}
+
+/* --------------------------------------------------------------- the API -- */
+
+export interface TokenizeResult {
+  spans: SyntaxSpan[];
+  /** The grammar(s) that produced them, for the payload's `grammar` field. */
+  grammars: string[];
+}
+
+/**
+ * Parse `source` and classify it.
+ *
+ * Returns null when nothing in the file has a grammar — a plain answer, which
+ * every caller here already knows how to serve. Never throws: a grammar that
+ * fails to load or a parse that comes back empty is the same outcome as not
+ * having one.
+ */
+export async function tokenizeSource(
+  source: string,
+  language: Language
+): Promise<TokenizeResult | null> {
+  const regions = syntaxRegionsFor(source, language);
+  if (regions === null) {
+    const spans = await tokenizeRegion(source, language, 0);
+    return spans ? { spans, grammars: [language] } : null;
+  }
+  if (regions.length === 0) return null;
+
+  const spans: SyntaxSpan[] = [];
+  const grammars = new Set<string>();
+  for (const region of regions) {
+    const part = await tokenizeRegion(
+      source.slice(region.start, region.end),
+      region.language,
+      region.start
+    );
+    if (!part) continue;
+    grammars.add(region.language);
+    spans.push(...part);
+  }
+  if (spans.length === 0) return null;
+  spans.sort((a, b) => a.start - b.start);
+  return { spans, grammars: [...grammars] };
+}
+
+async function tokenizeRegion(
+  source: string,
+  language: Language,
+  offset: number
+): Promise<SyntaxSpan[] | null> {
+  try {
+    await loadGrammarsForLanguages([language]);
+    const parser = getParser(language);
+    if (!parser) return null;
+    const tree = parser.parse(source);
+    if (!tree?.rootNode) return null;
+    try {
+      return classifyTree(tree.rootNode, source, language, offset);
+    } finally {
+      tree.delete();
+    }
+  } catch {
+    // A grammar that will not load, or a parse that threw: the caller serves
+    // the source unclassified, which is the whole point of the plain path.
+    return null;
+  }
+}

src/ui-server/api/filecode.ts

@@ -18,7 +18,7 @@
  *   before a single page of source has arrived.
  *
  * The source itself does NOT ride along. A 6 800-line TypeScript file is ~1.5 s
- * of TextMate tokenising and megabytes of JSON; the viewer pages it through
+ * of parsing and megabytes of JSON; the viewer pages it through
  * `/api/source` as the reader scrolls, which is also what lets the graph
  * facts — ports, arcs, rail rows — be complete from the first frame while the
  * text fills in behind them.

src/ui-server/highlight/grammars.ts

@@ -1,86 +0,0 @@
-/**
- * Finding and reading the pruned TextMate grammars on disk.
- *
- * Shiki ships 700-odd grammars; the engine indexes 40-odd languages. The build
- * writes only the closure those 40 need — including the grammars they embed, so
- * a `.vue` file still gets its `<script lang="ts">` — into `dist/textmate/`,
- * and `@shikijs/langs` stays a devDependency that never reaches a user's disk.
- * See `scripts/prune-grammars.mjs`.
- *
- * They are located the way `db/index.ts` finds `schema.sql` and `assets.ts`
- * finds the viewer: relative to `__dirname`, never to `process.cwd()`, which is
- * whatever directory the user happened to be standing in.
- *
- * `dist/textmate`, not `dist/highlight` — `src/ui-server/highlight/` is this
- * module and tsc already owns `dist/ui-server/highlight/`. The same collision
- * that put the viewer in `dist/viewer` rather than `dist/ui`.
- */
-
-import * as fs from 'fs';
-import * as path from 'path';
-
-/** Overrides where the grammars are read from. For tests and for packagers. */
-export const TEXTMATE_PATH_ENV = 'CODEGRAPH_TEXTMATE_PATH';
-
-/** What `scripts/prune-grammars.mjs` writes beside the grammar files. */
-export interface GrammarManifest {
-  /** Shiki version the grammars were pruned from — surfaced when one fails. */
-  shikiVersion: string;
-  /** Grammar id → the files to load, dependencies first, the grammar itself last. */
-  languages: Record<string, string[]>;
-}
-
-/**
- * Candidate locations, most-specific first.
- *
- * 1. The `CODEGRAPH_TEXTMATE_PATH` override.
- * 2. `<__dirname>/../../textmate` — the shipped layout
- *    (`dist/ui-server/highlight/` → `dist/textmate/`).
- * 3. `<__dirname>/../../../dist/textmate` — running the TypeScript straight out
- *    of `src/` (vitest, tsx), where `__dirname` is `src/ui-server/highlight/`.
- */
-export function grammarDirCandidates(): string[] {
-  const override = process.env[TEXTMATE_PATH_ENV]?.trim();
-  const candidates = [
-    path.join(__dirname, '..', '..', 'textmate'),
-    path.join(__dirname, '..', '..', '..', 'dist', 'textmate'),
-  ];
-  return override ? [path.resolve(override), ...candidates] : candidates;
-}
-
-/**
- * The grammar directory and its manifest, or null when the build did not run.
- *
- * Null is a normal outcome, not an error: a source checkout that has only had
- * `tsc` run against it has no `dist/textmate`, and the right answer there is
- * plain text, not a 500 on a request for source.
- */
-export function loadManifest(): { dir: string; manifest: GrammarManifest } | null {
-  for (const dir of grammarDirCandidates()) {
-    try {
-      const raw = fs.readFileSync(path.join(dir, 'manifest.json'), 'utf-8');
-      const manifest = JSON.parse(raw) as GrammarManifest;
-      if (manifest && typeof manifest === 'object' && manifest.languages) return { dir, manifest };
-    } catch {
-      // Not here — try the next candidate.
-    }
-  }
-  return null;
-}
-
-/**
- * Read the grammar registrations one language needs, dependencies first.
- *
- * Shiki resolves a grammar's `embeddedLangs` against what is already in its
- * registry, so the order the manifest records matters: `vue` must arrive after
- * the `html`, `css` and `typescript` it embeds, or the embedded blocks come
- * back unhighlighted.
- */
-export function readGrammarChain(dir: string, manifest: GrammarManifest, id: string): unknown[] {
-  const files = manifest.languages[id];
-  if (!files) return [];
-  return files.map((file) => {
-    const full = path.join(dir, `${file}.json`);
-    return JSON.parse(fs.readFileSync(full, 'utf-8')) as unknown;
-  });
-}

src/ui-server/highlight/index.ts

@@ -1,57 +1,61 @@
 /**
- * Server-side syntax classification for the viewer's code block (CG-43).
+ * Server-side syntax classification for the viewer's code block.
  *
- * The viewer used to lex on the client with a hand-rolled dialect table. This
- * replaces it with real TextMate grammars, run once here, so a Go file reads as
- * Go rather than as "something with braces". Three properties keep that from
- * becoming a liability:
+ * The classes come off the engine's OWN tree-sitter parse (CG-57). Until then
+ * the viewer ran a second highlighter — Shiki, plus 56 pruned TextMate grammars
+ * shipped beside the binary — over source the engine had already parsed with a
+ * real grammar. That is gone: one grammar set, one opinion about what a `.ts`
+ * file is, nothing extra in the bundle, and roughly an order of magnitude off
+ * the cost on the language that used to be worst (see below).
  *
- * * **It never fails a request.** A missing grammar, a corrupt grammar file, an
- *   ESM import that did not resolve, a slice too big to be worth tokenising —
- *   every one of them answers `engine: 'plain'` with a reason and the source
- *   still goes out. Highlighting is the part that degrades; nothing else does.
+ * Three properties are unchanged, because they are what make this safe to
+ * depend on:
+ *
+ * * **It never fails a request.** A grammar that will not load, a parse that
+ *   throws, a language nobody wrote a grammar for, a slice too big to be worth
+ *   parsing — every one of them answers `engine: 'plain'` with a reason and the
+ *   source still goes out. Highlighting is the part that degrades; nothing else
+ *   does.
  * * **Identifiers survive whatever token boundaries the grammar chose.** Every
  *   code token is split into identifier runs before it goes on the wire, which
  *   is what lets the viewer wrap a call site as a link by *claiming a token*
- *   rather than re-tokenising the line on top of the highlighter's answer.
- * * **The classification is a class name, not a colour.** See `theme.ts` — the
- *   viewer paints from CSS custom properties, so one token stream serves light
- *   and dark and the design tokens live in exactly one place.
+ *   rather than re-tokenising the line on top of the classifier's answer.
+ * * **The classification is a class name, not a colour.** The viewer paints
+ *   from CSS custom properties, so one token stream serves light and dark and
+ *   the design tokens live in exactly one place.
  *
  * ## Cost, measured
  *
- * Shiki's JavaScript regex engine (no oniguruma wasm, no native module) runs at
- * roughly 17 us/line on Go, 34 us/line on Python and 230 us/line on TypeScript,
- * whose TextMate grammar is by a wide margin the most expensive one here. A
- * 3 000-line TypeScript file is therefore ~700 ms cold, which is why
- * {@link SLICE_CACHE_LIMIT} exists: a slice is keyed by the file's content hash
- * and its line range, so every re-render — a theme flip, a resize, stepping
- * back to a symbol — is a map lookup. Phase 1 only ever asks for one symbol's
- * range (tens of lines); the whole-file view is CG-52 and tree-sitter tokens
- * from the engine's own parse replace this module entirely in CG-57.
+ * On the dev Mac, 3 000 lines, cold (parse + classify + wire):
+ *
+ * | | TypeScript | Go | Python |
+ * |---|---|---|---|
+ * | Shiki (was) | ~700 ms | 43–57 ms | 35–47 ms |
+ * | tree-sitter (now) | 24–41 ms | ~30 ms | 25–29 ms |
+ *
+ * Rust, Ruby, PHP, C# and Swift all land between 14 and 27 ms on the same
+ * measurement. TypeScript's TextMate grammar was 5–7× every other one and the
+ * cost was regex *execution*, not compilation — nothing about the old module
+ * could have fixed it, and it is now the same order as everything else. The
+ * slice cache still exists, because a re-render (a theme flip, a resize,
+ * stepping back through the trail) should cost nothing at all, and because a
+ * whole-file view pages the same file repeatedly.
  */
 
-import type {
-  ShikiCoreModule,
-  ShikiHighlighter,
-  ShikiJavaScriptEngineModule,
-  ShikiThemedToken,
-} from './shiki-types';
-import { CLASS_ID, MONO_THEME, TOKEN_CLASSES, classOf, type TokenClassName } from './theme';
+import { SYNTAX_TOKEN_CLASSES, tokenizeSource, type SyntaxTokenClass } from '../../extraction/syntax-tokens';
+import type { Language } from '../../types';
 import { grammarFor } from './languages';
-import { loadManifest, readGrammarChain, type GrammarManifest } from './grammars';
 
-export { TOKEN_CLASSES } from './theme';
-export { LANGUAGE_GRAMMAR, REQUIRED_GRAMMARS, grammarFor } from './languages';
-export { TEXTMATE_PATH_ENV } from './grammars';
+export { COMPONENT_LANGUAGES, grammarFor, isHighlightable } from './languages';
+export { SYNTAX_TOKEN_CLASSES as TOKEN_CLASSES } from '../../extraction/syntax-tokens';
 
 /** One token on the wire: its class id, then its text. */
 export type WireToken = [number, string];
 
 export interface HighlightResult {
-  /** `shiki` when a grammar produced the classes; `plain` when nothing did. */
-  engine: 'shiki' | 'plain';
-  /** The TextMate grammar used, or null. */
+  /** `tree-sitter` when a grammar produced the classes; `plain` when nothing did. */
+  engine: 'tree-sitter' | 'plain';
+  /** The grammar the source was read with, or null. */
   grammar: string | null;
   /** Class names, indexed by the first element of every {@link WireToken}. */
   classes: readonly string[];
@@ -62,25 +66,25 @@ export interface HighlightResult {
 }
 
 /**
- * Lines above this are not tokenised.
+ * Lines above this are not classified.
  *
  * Matches `MAX_SOURCE_LINES`, so anything the source endpoint will serve, this
- * will try to highlight.
+ * will try to classify.
  */
 export const MAX_HIGHLIGHT_LINES = 4000;
 
 /**
- * Characters above this are not tokenised.
+ * Characters above this are not classified.
  *
  * The line cap alone does not bound the work: one minified bundle line can be
- * two megabytes, and a TextMate scanner walks it character by character. This
- * is the guard that keeps a single request from wedging a single-threaded
- * loopback server, and it is generous — 600 kB is far more source than any
- * screen renders.
+ * two megabytes, and a parser walks it character by character. This is the
+ * guard that keeps a single request from wedging a single-threaded loopback
+ * server, and it is generous — 600 kB is far more source than any screen
+ * renders.
  */
 export const MAX_HIGHLIGHT_CHARS = 600_000;
 
-/** Highlighted slices kept in memory. Most are one symbol's body. */
+/** Classified slices kept in memory. Most are one symbol's body. */
 export const SLICE_CACHE_LIMIT = 96;
 
 /**
@@ -94,105 +98,23 @@ export const SLICE_CACHE_LIMIT = 96;
  */
 export const SLICE_CACHE_LINES = 20_000;
 
-/* ----------------------------------------------------------- the runtime -- */
-
-/**
- * tsc compiles `import()` to `require()` under `module: commonjs`, which fails
- * for an ESM-only package. Same escape hatch `src/bin/codegraph.ts` uses.
- */
-const importESM = new Function('specifier', 'return import(specifier)') as (
-  specifier: string
-) => Promise<unknown>;
+/** Class name → its index in {@link SYNTAX_TOKEN_CLASSES}, which is what the wire carries. */
+const CLASS_ID = Object.fromEntries(
+  SYNTAX_TOKEN_CLASSES.map((name, index) => [name, index])
+) as Record<SyntaxTokenClass, number>;
 
 /**
- * Import an ESM-only package from this CommonJS build.
+ * Classes that are never merged with their neighbour.
  *
- * The `new Function` route is the one that runs in production. It does NOT run
- * under Vitest, whose module runner evaluates this file without a dynamic-import
- * callback ("A dynamic import callback was not specified") — there, the
- * transformed `import()` below is the working one, and in the shipped CommonJS
- * build it is the one that cannot work. Each covers exactly the other's gap;
- * neither alone is enough, which is why both are here.
+ * Every identifier-shaped token has to stay claimable on its own — the overlay
+ * wraps exactly one of them as a call-site link, and two merged into one token
+ * would underline both or neither.
  */
-async function loadEsm<T>(specifier: string): Promise<T> {
-  try {
-    return (await importESM(specifier)) as T;
-  } catch (err) {
-    if (!(err instanceof Error) || !/dynamic import callback/i.test(err.message)) throw err;
-    return (await import(/* @vite-ignore */ specifier)) as T;
-  }
-}
-
-interface Runtime {
-  highlighter: ShikiHighlighter;
-  dir: string;
-  manifest: GrammarManifest;
-}
-
-let runtimePromise: Promise<Runtime | null> | null = null;
-/** Why the runtime is unavailable, for the `reason` on a plain answer. */
-let runtimeFailure: string | null = null;
-
-async function getRuntime(): Promise<Runtime | null> {
-  if (!runtimePromise) runtimePromise = createRuntime();
-  return runtimePromise;
-}
-
-async function createRuntime(): Promise<Runtime | null> {
-  const found = loadManifest();
-  if (!found) {
-    runtimeFailure =
-      'No syntax grammars are installed with this build, so source is shown unhighlighted.';
-    return null;
-  }
-  try {
-    const core = await loadEsm<ShikiCoreModule>('@shikijs/core');
-    const engineModule = await loadEsm<ShikiJavaScriptEngineModule>('@shikijs/engine-javascript');
-    const highlighter = core.createHighlighterCoreSync({
-      themes: [MONO_THEME],
-      langs: [],
-      // The JavaScript regex engine, deliberately: no oniguruma wasm and no
-      // native module, so the viewer adds nothing to the install that has to
-      // be compiled or fetched per platform. `forgiving` skips the handful of
-      // Oniguruma-only patterns it cannot translate rather than refusing the
-      // whole grammar over them.
-      engine: engineModule.createJavaScriptRegexEngine({ forgiving: true, cache: new Map() }),
-    });
-    return { highlighter, dir: found.dir, manifest: found.manifest };
-  } catch (err) {
-    runtimeFailure = `Syntax highlighting is unavailable (${
-      err instanceof Error ? err.message : String(err)
-    }).`;
-    return null;
-  }
-}
-
-/** Grammar ids already handed to the highlighter, and the ones that failed. */
-const loadedGrammars = new Set<string>();
-const brokenGrammars = new Map<string, string>();
-
-function ensureGrammar(runtime: Runtime, id: string): string | null {
-  if (loadedGrammars.has(id)) return null;
-  const broken = brokenGrammars.get(id);
-  if (broken !== undefined) return broken;
-  try {
-    const chain = readGrammarChain(runtime.dir, runtime.manifest, id);
-    if (chain.length === 0) {
-      const reason = `No ${id} grammar shipped with this build, so it is shown unhighlighted.`;
-      brokenGrammars.set(id, reason);
-      return reason;
-    }
-    runtime.highlighter.loadLanguageSync(chain);
-    loadedGrammars.add(id);
-    return null;
-  } catch (err) {
-    const reason = `The ${id} grammar could not be loaded (${
-      err instanceof Error ? err.message : String(err)
-    }).`;
-    brokenGrammars.set(id, reason);
-    return reason;
-  }
-}
+const UNMERGEABLE: ReadonlySet<SyntaxTokenClass> = new Set<SyntaxTokenClass>([
+  'ident',
+  'type',
+  'def',
+]);
 
 /* -------------------------------------------------------------- the cache -- */
 
@@ -242,7 +164,7 @@ export interface HighlightOptions {
   language?: string | null;
   /**
    * A key that changes whenever the text does — the file's content hash plus
-   * the requested range. Omit it and the slice is tokenised every time.
+   * the requested range. Omit it and the slice is classified every time.
    */
   cacheKey?: string;
 }
@@ -265,13 +187,14 @@ export async function highlightLines(
     if (hit) return hit;
   }
 
-  const result = await highlightUncached(lines, grammar);
+  const result = await highlightUncached(lines, options.language ?? null, grammar);
   if (key) cachePut(key, result);
   return result;
 }
 
 async function highlightUncached(
   lines: readonly string[],
+  language: string | null,
   grammar: string | null
 ): Promise<HighlightResult> {
   if (!grammar) {
@@ -280,55 +203,84 @@ async function highlightUncached(
   if (lines.length > MAX_HIGHLIGHT_LINES) {
     return plain(lines, grammar, `Too many lines to highlight (over ${MAX_HIGHLIGHT_LINES}).`);
   }
-  let chars = 0;
-  for (const line of lines) chars += line.length + 1;
-  if (chars > MAX_HIGHLIGHT_CHARS) {
+  const text = lines.join('\n');
+  if (text.length > MAX_HIGHLIGHT_CHARS) {
     return plain(lines, grammar, 'Too much text on too few lines to highlight (minified?).');
   }
 
-  const runtime = await getRuntime();
-  if (!runtime) return plain(lines, grammar, runtimeFailure ?? undefined);
-
-  const failure = ensureGrammar(runtime, grammar);
-  if (failure) return plain(lines, grammar, failure);
-
-  let tokenized: ShikiThemedToken[][];
-  try {
-    tokenized = runtime.highlighter.codeToTokensBase(lines.join('\n'), {
-      lang: grammar,
-      theme: MONO_THEME.name,
-    });
-  } catch (err) {
-    // A grammar that throws once will throw again on the next request for the
-    // same file type, so it is retired rather than retried.
-    const reason = `The ${grammar} grammar failed on this file (${
-      err instanceof Error ? err.message : String(err)
-    }).`;
-    brokenGrammars.set(grammar, reason);
-    loadedGrammars.delete(grammar);
-    return plain(lines, grammar, reason);
+  const tokenized = await tokenizeSource(text, language as Language);
+  if (!tokenized || tokenized.spans.length === 0) {
+    return plain(lines, grammar, `The ${grammar} grammar is not available in this build.`);
   }
 
-  // A trailing empty line, or a grammar that answered short, must not shift the
-  // viewer's line numbering — the rows are indexed positionally.
-  const out: WireToken[][] = lines.map((line, i) => {
-    const row = tokenized[i];
-    return row ? atomize(row) : atomizePlain(line);
-  });
-
-  return { engine: 'shiki', grammar, classes: TOKEN_CLASSES, lines: out };
+  return {
+    engine: 'tree-sitter',
+    grammar: tokenized.grammars.join('+') || grammar,
+    classes: SYNTAX_TOKEN_CLASSES,
+    lines: toWireLines(lines, text, tokenized.spans),
+  };
 }
 
 function plain(lines: readonly string[], grammar: string | null, reason?: string): HighlightResult {
   return {
     engine: 'plain',
     grammar,
-    classes: TOKEN_CLASSES,
+    classes: SYNTAX_TOKEN_CLASSES,
     lines: lines.map(atomizePlain),
     ...(reason ? { reason } : {}),
   };
 }
 
+/* -------------------------------------------------------- spans to lines -- */
+
+/**
+ * Cut the classifier's spans into one token list per source line.
+ *
+ * The classifier answers over the whole slice, in string offsets, and leaves
+ * the gaps between spans unclassified — those are whitespace and the layout
+ * separators no grammar names. Here they become plain tokens, multi-line spans
+ * (a block comment, a heredoc) are split at the newlines, and every line ends
+ * up with a token list whose texts concatenate back to exactly that line.
+ *
+ * One entry per line, always: the code block indexes rows positionally, so a
+ * short answer would render every line below it against the wrong source.
+ */
+function toWireLines(
+  lines: readonly string[],
+  text: string,
+  spans: readonly { start: number; end: number; cls: SyntaxTokenClass }[]
+): WireToken[][] {
+  const pieces: { start: number; end: number; cls: SyntaxTokenClass }[] = [];
+  let cursor = 0;
+  for (const span of spans) {
+    if (span.end <= cursor) continue;
+    const start = Math.max(span.start, cursor);
+    if (start > cursor) pieces.push({ start: cursor, end: start, cls: 'other' });
+    pieces.push({ start, end: span.end, cls: span.cls });
+    cursor = span.end;
+  }
+  if (cursor < text.length) pieces.push({ start: cursor, end: text.length, cls: 'other' });
+
+  const out: WireToken[][] = [];
+  let lineStart = 0;
+  let first = 0;
+  for (const line of lines) {
+    const lineEnd = lineStart + line.length;
+    const row: WireToken[] = [];
+    while (first < pieces.length && (pieces[first] as { end: number }).end <= lineStart) first += 1;
+    for (let i = first; i < pieces.length; i++) {
+      const piece = pieces[i] as { start: number; end: number; cls: SyntaxTokenClass };
+      if (piece.start >= lineEnd) break;
+      const from = Math.max(piece.start, lineStart);
+      const to = Math.min(piece.end, lineEnd);
+      if (to > from) pushPiece(row, text.slice(from, to), piece.cls);
+    }
+    out.push(row);
+    lineStart = lineEnd + 1; // the '\n' the join put back
+  }
+  return out;
+}
+
 /* ---------------------------------------------------------- atomisation -- */
 
 /**
@@ -341,29 +293,23 @@ function plain(lines: readonly string[], grammar: string | null, reason?: string
 const IDENT = /[A-Za-z_$À-￿][\w$À-￿]*/g;
 
 /**
- * Split a grammar's tokens into identifier runs, merging everything else.
+ * Add one classified run to a line, splitting it into identifier atoms.
  *
  * This is the step that makes the graph's call-site links independent of how a
- * grammar chose to chunk a line. TextMate is free to emit `this.mutex.withLock`
- * as one token, three, or five, and the viewer has to be able to wrap exactly
- * `withLock`; giving it identifier-sized atoms up front means the overlay only
- * ever *claims* a token, never re-cuts one.
+ * grammar chose to chunk a line: the viewer has to be able to wrap exactly
+ * `withLock` in `this.mutex.withLock`, and giving it identifier-sized atoms up
+ * front means the overlay only ever *claims* a token, never re-cuts one.
  *
  * Comments and strings are left whole on purpose: no edge points inside one,
  * and a doc comment split into forty atoms is forty times the wire bytes for
  * nothing.
  */
-function atomize(tokens: readonly ShikiThemedToken[]): WireToken[] {
-  const out: WireToken[] = [];
-  for (const token of tokens) {
-    const cls = classOf(token.color);
-    if (cls === 'comment' || cls === 'string') {
-      push(out, cls, token.content);
-      continue;
-    }
-    splitIdentifiers(out, token.content, cls);
+function pushPiece(row: WireToken[], text: string, cls: SyntaxTokenClass): void {
+  if (cls === 'comment' || cls === 'string') {
+    push(row, cls, text);
+    return;
   }
-  return out;
+  splitIdentifiers(row, text, cls);
 }
 
 function atomizePlain(line: string): WireToken[] {
@@ -375,31 +321,35 @@ function atomizePlain(line: string): WireToken[] {
 /**
  * Emit `text` as alternating non-identifier and identifier runs.
  *
- * An identifier inside a token the grammar called a keyword keeps the keyword
+ * An identifier inside a run the grammar called a keyword keeps the keyword
  * class — `func` should still carry its weight — while the overlay's matcher
- * looks at a token's *text*, not its class, so a language whose grammar scopes
- * type names as `storage.type` still links.
+ * looks at a token's *text*, not its class, so a language whose grammar calls a
+ * declared type name something unexpected still links.
  */
-function splitIdentifiers(out: WireToken[], text: string, cls: TokenClassName): void {
+function splitIdentifiers(out: WireToken[], text: string, cls: SyntaxTokenClass): void {
   if (text === '') return;
   IDENT.lastIndex = 0;
   let at = 0;
   let match: RegExpExecArray | null;
   while ((match = IDENT.exec(text)) !== null) {
-    if (match.index > at) push(out, cls === 'ident' ? 'other' : cls, text.slice(at, match.index));
+    if (match.index > at) push(out, gapClass(cls), text.slice(at, match.index));
     push(out, cls === 'other' ? 'ident' : cls, match[0]);
     at = match.index + match[0].length;
   }
-  if (at < text.length) push(out, cls === 'ident' ? 'other' : cls, text.slice(at));
+  if (at < text.length) push(out, gapClass(cls), text.slice(at));
+}
+
+/** The class for the non-identifier remainder of a run. */
+function gapClass(cls: SyntaxTokenClass): SyntaxTokenClass {
+  return UNMERGEABLE.has(cls) ? 'other' : cls;
 }
 
 /** Append, merging into the previous token when it carries the same class. */
-function push(out: WireToken[], cls: TokenClassName, text: string): void {
+function push(out: WireToken[], cls: SyntaxTokenClass, text: string): void {
   if (text === '') return;
   const id = CLASS_ID[cls];
   const last = out[out.length - 1];
-  // Identifiers are never merged: each one has to stay claimable on its own.
-  if (last && last[0] === id && id !== CLASS_ID.ident) {
+  if (last && last[0] === id && !UNMERGEABLE.has(cls)) {
     last[1] += text;
     return;
   }

src/ui-server/highlight/languages.ts

@@ -1,87 +1,34 @@
 /**
- * Engine `Language` → TextMate grammar, and the closure of grammars that has
- * to ship for those to load.
+ * Which engine language a file's source is classified with (CG-57).
  *
- * The engine indexes 40-odd languages; Shiki carries 700-odd grammars. Shipping
- * all of them would put 11 MB of JSON in the bundle to serve 40, so the build
- * prunes them (`scripts/prune-grammars.mjs`) to exactly the closure this table
- * names — which is why the table lives in its own module: the build script
- * reads the compiled `dist/ui-server/highlight/languages.js` rather than keeping
- * a second copy of the mapping that could drift from the runtime's.
+ * There is no second grammar table any more. The viewer reads a file with the
+ * grammar the *engine* parsed it with, so this is a question about coverage
+ * rather than about mapping: a language the extractor has a tree-sitter grammar
+ * for classifies; one it does not renders plain, with its identifiers still
+ * split out so the graph's call-site links land exactly as they do everywhere
+ * else. Highlighting is the part that degrades, never the linking.
  *
- * A language with no entry (or with `null`) is not an error. It renders as
- * plain text with its identifiers still split out, so the graph's call-site
- * links land exactly as they do everywhere else — highlighting is the part that
- * degrades, never the linking.
+ * The three single-file-component formats are the exception worth naming. A
+ * `.svelte`, `.vue` or `.astro` file has no grammar of its own here — the
+ * extractors pull the `<script>` block out and hand it to TypeScript or
+ * JavaScript — and the classifier does exactly the same thing, so a component's
+ * code is read by the grammar its symbols came from while the surrounding
+ * markup stays plain.
  */
 
 import type { Language } from '../../types';
+import { hasTreeSitterGrammar } from '../../extraction/grammars';
 
 /**
- * The grammar each indexed language is read with.
+ * Formats whose source is classified through their embedded script blocks.
  *
- * Three of these are deliberate approximations, marked below: Shiki has no
- * ColdFusion grammar, and the three CFML dialects the engine distinguishes are
- * each a close relative of something it does have. An approximate keyword set
- * is a better answer than no colouring at all, and nothing downstream depends
- * on the grammar being exact — the links come from the graph.
+ * Kept beside `syntaxRegionsFor`, which decides where those blocks are — this
+ * list only has to agree about *which* formats have them.
  */
-export const LANGUAGE_GRAMMAR: Record<Language, string | null> = {
-  typescript: 'typescript',
-  javascript: 'javascript',
-  tsx: 'tsx',
-  jsx: 'jsx',
-  // ArkTS is TypeScript plus HarmonyOS decorators — the TS grammar reads it.
-  arkts: 'typescript',
-  python: 'python',
-  go: 'go',
-  rust: 'rust',
-  java: 'java',
-  c: 'c',
-  cpp: 'cpp',
-  csharp: 'csharp',
-  razor: 'razor',
-  php: 'php',
-  ruby: 'ruby',
-  swift: 'swift',
-  kotlin: 'kotlin',
-  dart: 'dart',
-  svelte: 'svelte',
-  vue: 'vue',
-  astro: 'astro',
-  liquid: 'liquid',
-  pascal: 'pascal',
-  scala: 'scala',
-  lua: 'lua',
-  luau: 'luau',
-  objc: 'objective-c',
-  r: 'r',
-  solidity: 'solidity',
-  nix: 'nix',
-  yaml: 'yaml',
-  twig: 'twig',
-  xml: 'xml',
-  properties: 'properties',
-  // Approximations — no CFML grammar exists. Tag soup reads as HTML, cfscript
-  // is a JavaScript-shaped dialect, and a <cfquery> body is SQL.
-  cfml: 'html',
-  cfscript: 'javascript',
-  cfquery: 'sql',
-  cobol: 'cobol',
-  vbnet: 'vb',
-  erlang: 'erlang',
-  terraform: 'terraform',
-  // Not a language, the absence of one: a file no extractor claimed.
-  unknown: null,
-};
-
-/** Every grammar the build must prune to, de-duplicated, in a stable order. */
-export const REQUIRED_GRAMMARS: readonly string[] = [
-  ...new Set(Object.values(LANGUAGE_GRAMMAR).filter((id): id is string => id !== null)),
-].sort();
+export const COMPONENT_LANGUAGES: readonly Language[] = ['svelte', 'vue', 'astro'];
 
 /**
- * The grammar for an indexed language, or null when it has none.
+ * The grammar a file of this language is read with, or null when it has none.
  *
  * Accepts the raw string off a `FileRecord` rather than a `Language`, because
  * an index written by an older engine can hold a language this build has since
@@ -89,5 +36,12 @@ export const REQUIRED_GRAMMARS: readonly string[] = [
  */
 export function grammarFor(language: string | undefined | null): string | null {
   if (!language) return null;
-  return LANGUAGE_GRAMMAR[language as Language] ?? null;
+  const lang = language as Language;
+  if (COMPONENT_LANGUAGES.includes(lang)) return 'typescript';
+  return hasTreeSitterGrammar(lang) ? lang : null;
+}
+
+/** Whether a file of this language classifies at all. For tests and diagnostics. */
+export function isHighlightable(language: string | undefined | null): boolean {
+  return grammarFor(language) !== null;
 }

src/ui-server/highlight/shiki-types.ts

@@ -1,40 +0,0 @@
-/**
- * The slice of Shiki's surface this server uses, declared locally.
- *
- * `@shikijs/core` is ESM-only and the engine compiles to CommonJS, so it is
- * loaded through the same `new Function('return import(...)')` escape hatch the
- * CLI uses for `@clack/prompts` — which means tsc never sees the import and
- * cannot type it. Rather than fight `.d.mts` resolution under
- * `module: commonjs`, the four shapes actually touched are written out here.
- * They are checked against the real package by the highlighter's tests: a
- * signature change shows up as a failing highlight, not as a silent `any`.
- */
-
-export interface ShikiThemedToken {
-  content: string;
-  color?: string;
-  fontStyle?: number;
-}
-
-export interface ShikiHighlighter {
-  loadLanguageSync(lang: unknown): void;
-  getLoadedLanguages(): string[];
-  codeToTokensBase(code: string, options: { lang: string; theme: string }): ShikiThemedToken[][];
-  dispose?(): void;
-}
-
-export interface ShikiCoreModule {
-  createHighlighterCoreSync(options: {
-    themes: unknown[];
-    langs: unknown[];
-    engine: unknown;
-  }): ShikiHighlighter;
-}
-
-export interface ShikiJavaScriptEngineModule {
-  createJavaScriptRegexEngine(options?: {
-    forgiving?: boolean;
-    target?: 'auto' | 'ES2025' | 'ES2024' | 'ES2018';
-    cache?: Map<string, RegExp | Error> | null;
-  }): unknown;
-}

src/ui-server/highlight/theme.ts

@@ -1,115 +0,0 @@
-/**
- * The near-monochrome code theme (design spec §2.2).
- *
- * The colouring is deliberately almost absent: comments and strings recede,
- * keywords carry weight rather than hue, and the ONLY colour in the body is a
- * resolved call site. A six-colour syntax theme buries exactly the thing the
- * screen exists to show.
- *
- * ## Why the theme's colours are sentinels, not colours
- *
- * A TextMate theme classifies by mapping scopes to colours, so that is how the
- * classification is *expressed* — but the values here are placeholders that
- * mean "comment", "string", "keyword", "number", nothing. The server turns each
- * one back into a class name; the viewer paints it from a CSS custom property.
- *
- * That indirection is load-bearing, not decoration:
- *
- * * **One token stream serves both modes.** The viewer flips light/dark from
- *   `prefers-color-scheme` with no reload and no refetch. Baking `#6a675d` into
- *   the payload would make dark mode a second request for the same source, and
- *   would put the design tokens in two places at once.
- * * **Contrast is fixed where the tokens live.** `ui/src/app.css` owns the
- *   ramp; a colour change there cannot leave the server's copy behind.
- *
- * The sentinels are arbitrary but must be distinct and must never be a colour a
- * grammar could plausibly emit through some other path, hence the `#00000n`
- * block: TextMate themes only ever return values *this* theme defines.
- */
-
-/** The classes a token can carry — the viewer's `TokenClass`, server side. */
-export const TOKEN_CLASSES = ['other', 'ident', 'comment', 'string', 'keyword', 'number'] as const;
-
-export type TokenClassName = (typeof TOKEN_CLASSES)[number];
-
-/** Class name → its index in {@link TOKEN_CLASSES}, which is what the wire carries. */
-export const CLASS_ID: Record<TokenClassName, number> = {
-  other: 0,
-  ident: 1,
-  comment: 2,
-  string: 3,
-  keyword: 4,
-  number: 5,
-};
-
-const FG_DEFAULT = '#000001';
-const FG_COMMENT = '#000002';
-const FG_STRING = '#000003';
-const FG_KEYWORD = '#000004';
-const FG_NUMBER = '#000005';
-
-/** Sentinel foreground → the class it stands for. */
-export const SENTINEL_CLASS: Record<string, TokenClassName> = {
-  [FG_DEFAULT]: 'other',
-  [FG_COMMENT]: 'comment',
-  [FG_STRING]: 'string',
-  [FG_KEYWORD]: 'keyword',
-  [FG_NUMBER]: 'number',
-};
-
-/**
- * The theme itself.
- *
- * Scope selection follows the spec exactly: `comment` recedes furthest,
- * `string`/`constant.numeric` sit one step in, `keyword`/`storage` stay ink and
- * gain weight, everything else is ink. Nothing sets a background — a token that
- * painted its own would fight the hovered-line and hot-line tints the rails use
- * to point at it.
- */
-export const MONO_THEME = {
-  name: 'codegraph-mono',
-  type: 'light' as const,
-  fg: FG_DEFAULT,
-  // TextMate wants a background; the viewer never reads it (the code block
-  // paints `--paper`), and it must not equal a foreground sentinel.
-  bg: '#ffffff',
-  settings: [
-    { settings: { foreground: FG_DEFAULT } },
-    { scope: ['comment', 'punctuation.definition.comment'], settings: { foreground: FG_COMMENT } },
-    {
-      scope: [
-        'string',
-        'string.template',
-        'punctuation.definition.string',
-        'constant.character.escape',
-      ],
-      settings: { foreground: FG_STRING },
-    },
-    {
-      scope: ['constant.numeric', 'constant.language', 'keyword.other.unit'],
-      settings: { foreground: FG_NUMBER },
-    },
-    {
-      scope: ['keyword', 'keyword.control', 'storage', 'storage.type', 'storage.modifier'],
-      settings: { foreground: FG_KEYWORD },
-    },
-    // `keyword.operator` is a keyword scope by name only: it covers `=`, `+`,
-    // `=>` and `?.`. Weighting punctuation buys nothing and costs the calm the
-    // rest of the block is built on, so it drops back to plain ink — while the
-    // operators that are actually WORDS (`new`, `typeof`, `instanceof`, `in`)
-    // keep their weight through the more specific rule below. Shiki resolves
-    // the longest matching scope, so the order here is the order of rescue,
-    // not of priority.
-    { scope: ['keyword.operator'], settings: { foreground: FG_DEFAULT } },
-    {
-      scope: ['keyword.operator.expression', 'keyword.operator.word', 'keyword.operator.new'],
-      settings: { foreground: FG_KEYWORD },
-    },
-  ],
-};
-
-/** The class a Shiki token's resolved colour stands for. */
-export function classOf(color: string | undefined): TokenClassName {
-  if (!color) return 'other';
-  return SENTINEL_CLASS[color.toLowerCase()] ?? 'other';
-}

ui/src/components/flow/FlowCard.svelte

@@ -8,7 +8,7 @@
   pinned here so the arrows land where the arithmetic said they would.
 
   The source window is the Symbol view's code block with the noise removed. It
-  keeps the two things that make the code readable: the server's TextMate
+  keeps the two things that make the code readable: the server's classified
   classification, and one accent link on the identifier the graph resolved. It
   drops gutter ports and multi-window folding, because a seven-line card has
   neither a gutter worth reading nor anything to fold.
@@ -213,7 +213,7 @@
   }
 
   /* Token classes — the same near-monochrome ramp the Symbol view paints
-     (design spec §2.2); the class names come from the server's theme. */
+     (design spec §2.2); the class names come from the server's classifier. */
   .t-c {
     color: var(--code-comment);
   }
@@ -227,6 +227,11 @@
     color: var(--ink-2);
   }
 
+  /* A definition's own name, from the extractor's tables. */
+  .t-def {
+    font-weight: 600;
+  }
+
   /* The only colour in the window: the call this card is opened at. */
   .ref {
     padding: 0;

ui/src/components/symbol/SourceBlock.svelte

@@ -4,10 +4,11 @@
 
   Two things make this more than a <pre>:
 
-  * Syntax classification arrives already done, from `/api/source` — real
-    TextMate grammars, run server-side, indexed by file line. The whole slice
-    is tokenised in one pass there, so a window that starts 200 lines into a
-    body still knows it is inside a block comment; nothing is re-lexed here.
+  * Syntax classification arrives already done, from `/api/source` — taken off
+    the engine's own tree-sitter parse, server-side, indexed by file line. The
+    whole slice is classified in one pass there, so a window that starts 200
+    lines into a body still knows it is inside a block comment; nothing is
+    re-lexed here.
   * Each ref is matched to an actual token rather than to a column, because the
     recorded column points at the start of the calling expression — see
     `assignRefs`. The overlay CLAIMS a token the highlighter produced; it never

ui/src/lib/api.ts

@@ -163,7 +163,7 @@ export interface WireSource {
   truncated?: boolean;
   reason?: string;
   /**
-   * The same lines, classified by the server's TextMate grammars — one entry
+   * The same lines, classified by the server's tree-sitter parse — one entry
    * per line, each a list of `[classId, text]` pairs indexed into `classes`.
    * Absent whenever `lines` is, and `engine: 'plain'` whenever no grammar
    * covers the file. See `lib/highlight.ts`.

ui/src/lib/filecode-model.ts

@@ -60,12 +60,13 @@ export const OVERSCAN_LINES = 24;
 /**
  * Source lines fetched in one page.
  *
- * Measured on this repo's own TypeScript with the shipped Shiki setup: a warm
- * grammar tokenises ~7 000 lines/second, so a page plus its lead-in is ~130 ms
- * of single-threaded server. Bigger pages mean fewer, longer stalls; smaller
- * ones mean the lead-in dominates. The scroll itself never waits on this —
- * ports, arcs and rail rows are already drawn from the graph, and the text
- * arrives behind them.
+ * Measured on this repo's own TypeScript with the shipped classifier: a loaded
+ * grammar classifies ~50 000 lines/second (CG-57 replaced the TextMate path,
+ * which managed ~4 000), so a page plus its lead-in is ~20 ms of
+ * single-threaded server. Bigger pages mean fewer, longer stalls; smaller ones
+ * mean the lead-in dominates. The scroll itself never waits on this — ports,
+ * arcs and rail rows are already drawn from the graph, and the text arrives
+ * behind them.
  */
 export const PAGE_LINES = 800;
 
@@ -73,7 +74,7 @@ export const PAGE_LINES = 800;
  * Lines fetched BEFORE a page and thrown away.
  *
  * A page that starts in the middle of a block comment, a template literal or a
- * JSX block does not know it: TextMate state is built by scanning from the top.
+ * JSX block does not know it: a parse starts from the top of what it is given.
  * Tokenising a run-up and discarding it is what keeps page 6 from rendering a
  * doc comment as code. The same trick the Flow strip's source windows use, at a
  * different scale — 150 lines covers every real comment block; a 3 000-line

ui/src/lib/highlight.ts

@@ -2,7 +2,8 @@
  * Turning the server's classified source into tokens the code block can draw.
  *
  * The classification itself happens on the server (`src/ui-server/highlight/`),
- * with real TextMate grammars via Shiki. What arrives is deliberately small:
+ * off the engine's own tree-sitter parse — the same grammar that decided what
+ * the file's symbols are. What arrives is deliberately small:
  * one array per line, each entry a `[classId, text]` pair, with the class names
  * carried alongside so the payload is self-describing. This module does two
  * things to it and nothing else — resolve the class ids to names, and compute
@@ -23,7 +24,18 @@
  * underline land on the callee's own name whatever boundaries a grammar chose.
  */
 
-export type TokenClass = 'other' | 'ident' | 'comment' | 'string' | 'keyword' | 'number';
+export type TokenClass =
+  | 'other'
+  | 'ident'
+  | 'comment'
+  | 'string'
+  | 'keyword'
+  | 'number'
+  /** A named type reference. Plain ink today — the class is here so a consumer
+   *  of this payload can style it without a second round of server work. */
+  | 'type'
+  /** The name a definition declares, from the extractor's own tables. */
+  | 'def';
 
 export interface Token {
   cls: TokenClass;
@@ -36,7 +48,7 @@ export interface Token {
 export type WireToken = [number, string];
 
 export interface WireHighlight {
-  engine: 'shiki' | 'plain';
+  engine: 'tree-sitter' | 'plain';
   grammar: string | null;
   classes: string[];
   lines: WireToken[][];
@@ -51,6 +63,8 @@ const CLASS_NAMES: ReadonlySet<string> = new Set<TokenClass>([
   'string',
   'keyword',
   'number',
+  'type',
+  'def',
 ]);
 
 /**
@@ -116,7 +130,15 @@ export function tokensByLine(
   return byLine;
 }
 
-/** The CSS class for a token, or null where the default ink is right. */
+/**
+ * The CSS class for a token, or null where the default ink is right.
+ *
+ * `type` deliberately returns null: the design's code colouring is
+ * near-monochrome and a type name is not one of the four things it moves off
+ * plain ink. It stays a distinct class on the wire because the classification
+ * is free once the tree has been walked, and re-deriving it in a consumer would
+ * not be.
+ */
 export function tokenClass(cls: TokenClass): string | null {
   switch (cls) {
     case 'comment':
@@ -127,6 +149,8 @@ export function tokenClass(cls: TokenClass): string | null {
       return 't-k';
     case 'number':
       return 't-n';
+    case 'def':
+      return 't-def';
     default:
       return null;
   }