feat: browser data platform for AI agents (v0.16.0.0) (#907)

* refactor: extract path-security.ts shared module

validateOutputPath, validateReadPath, and SAFE_DIRECTORIES were duplicated
across write-commands.ts, meta-commands.ts, and read-commands.ts. Extract
to a single shared module with re-exports for backward compatibility.

Also adds validateTempPath() for the upcoming GET /file endpoint (TEMP_DIR
only, not cwd, to prevent remote agents from reading project files).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: default paired agents to full access, split SCOPE_CONTROL

The trust boundary for paired agents is the pairing ceremony itself, not
the scope. An agent with write scope can already click anything and navigate
anywhere. Gating js/cookies behind --admin was security theater.

Changes:
- Default pair scopes: read+write+admin+meta (was read+write)
- New SCOPE_CONTROL for browser-wide destructive ops (stop, restart,
  disconnect, state, handoff, resume, connect)
- --admin flag now grants control scope (backward compat)
- New --restrict flag for limited access (e.g., --restrict read)
- Updated hint text: "re-pair with --control" instead of "--admin"

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add media and data commands for page content extraction

media command: discovers all img/video/audio/background-image elements
on the page. Returns JSON with URLs, dimensions, srcset, loading state,
HLS/DASH detection. Supports --images/--videos/--audio filters and
optional CSS selector scoping.

data command: extracts structured data embedded in pages (JSON-LD,
Open Graph, Twitter Cards, meta tags). One command returns product
prices, article metadata, social share info without DOM scraping.

Both are READ scope with untrusted content wrapping.
Shared media-extract.ts helper for reuse by the upcoming scrape command.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add download, scrape, and archive commands

download: fetch any URL or @ref element to disk using browser session
cookies via page.request.fetch(). Supports blob: URLs via in-page
base64 conversion. --base64 flag returns inline data URI (cap 10MB).
Detects HLS/DASH and rejects with yt-dlp hint.

scrape: bulk media download composing media discovery + download loop.
Sequential with 100ms delay, URL deduplication, configurable --limit.
Writes manifest.json with per-file metadata for machine consumption.

archive: saves complete page as MHTML via CDP Page.captureSnapshot.
No silent fallback -- errors clearly if CDP unavailable.

All three are WRITE scope (write to disk, blocked in watch mode).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add GET /file endpoint for remote agent file retrieval

Remote paired agents can now retrieve downloaded files over HTTP.
TEMP_DIR only (not cwd) to prevent project file exfiltration.

- Bearer token auth (root or scoped with read scope)
- Path validation via validateTempPath() (symlink-aware)
- 200MB size cap
- Extension-based MIME detection
- Zero-copy streaming via Bun.file()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add scroll --times N for automated repeated scrolling

Extends the scroll command with --times N flag for infinite feed
scraping. Scrolls N times with configurable --wait delay (default
1000ms) between each scroll for content loading.

Usage: scroll --times 10
       scroll --times 5 --wait 2000
       scroll --times 3 .feed-container

Composable with scrape: scroll to load content, then scrape images.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add network response body capture (--capture/--export/--bodies)

The killer feature for social media scraping. Extends the existing
network command to intercept API response bodies:

  network --capture [--filter graphql]  # start capturing
  network --capture stop                # stop
  network --export /tmp/api.jsonl       # export as JSONL
  network --bodies                      # show summary

Uses page.on('response') listener with URL pattern filtering.
SizeCappedBuffer (50MB total, 5MB per-entry cap) evicts oldest
entries when full. Binary responses stored as base64, text as-is.

This lets agents tap Instagram's GraphQL API, TikTok's hydration
data, and any SPA's internal API responses instead of fragile DOM
scraping.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add screenshot --base64 for inline image return

Returns data:image/png;base64,... instead of writing to disk.
Cap at 10MB. Works with all screenshot modes (element, clip, viewport).

Eliminates the two-step screenshot+file-serve dance for remote agents.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add data platform tests and media fixture

Tests for SizeCappedBuffer (eviction, export, summary), validateTempPath
(TEMP_DIR only, rejects cwd), command registration (all new commands in
correct scope sets), and MIME mapping source checks.

Rich HTML fixture with: standard images, lazy-loaded images, srcset,
video with sources + HLS, audio, CSS background-images, JSON-LD,
Open Graph, Twitter Cards, and meta tags.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: regenerate SKILL.md with Extraction category

Add Extraction category to browse command table ordering. Regenerate
SKILL.md files to include media, data, download, scrape, archive
commands in the generated documentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.16.0.0)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

browse/src/server.ts

@@ -32,6 +32,7 @@ import {
   rotateRoot, listTokens, serializeRegistry, restoreRegistry, recordCommand,
   isRootToken, checkConnectRateLimit, type TokenInfo,
 } from './token-registry';
+import { validateTempPath } from './path-security';
 import { resolveConfig, ensureStateDir, readVersionHash } from './config';
 import { emitActivity, subscribe, getActivityAfter, getActivityHistory, getSubscriberCount } from './activity';
 import { inspectElement, modifyStyle, resetModifications, getModificationHistory, detachSession, type InspectorResult } from './cdp-inspector';
@@ -1457,9 +1458,12 @@ async function start() {
         }
         try {
           const pairBody = await req.json() as any;
-          const scopes = pairBody.admin
-            ? ['read', 'write', 'admin', 'meta'] as const
-            : (pairBody.scopes || ['read', 'write']) as const;
+          // Default: full access (read+write+admin+meta). The trust boundary is
+          // the pairing ceremony itself, not the scope. --control adds browser-wide
+          // destructive commands (stop, restart, disconnect). --restrict limits scope.
+          const scopes = pairBody.control || pairBody.admin
+            ? ['read', 'write', 'admin', 'meta', 'control'] as const
+            : (pairBody.scopes || ['read', 'write', 'admin', 'meta']) as const;
           const setupKey = createSetupKey({
             clientId: pairBody.clientId,
             scopes: [...scopes],
@@ -2031,6 +2035,60 @@ async function start() {
         });
       }
 
+      // ─── File serving endpoint (for remote agents to retrieve downloaded files) ────
+      if (url.pathname === '/file' && req.method === 'GET') {
+        const tokenInfo = getTokenInfo(req);
+        if (!tokenInfo) {
+          return new Response(JSON.stringify({ error: 'Unauthorized' }), {
+            status: 401, headers: { 'Content-Type': 'application/json' },
+          });
+        }
+        const filePath = url.searchParams.get('path');
+        if (!filePath) {
+          return new Response(JSON.stringify({ error: 'Missing "path" query parameter' }), {
+            status: 400, headers: { 'Content-Type': 'application/json' },
+          });
+        }
+        try {
+          validateTempPath(filePath);
+        } catch (err: any) {
+          return new Response(JSON.stringify({ error: err.message }), {
+            status: 403, headers: { 'Content-Type': 'application/json' },
+          });
+        }
+        if (!fs.existsSync(filePath)) {
+          return new Response(JSON.stringify({ error: 'File not found' }), {
+            status: 404, headers: { 'Content-Type': 'application/json' },
+          });
+        }
+        const stat = fs.statSync(filePath);
+        if (stat.size > 200 * 1024 * 1024) {
+          return new Response(JSON.stringify({ error: 'File too large (max 200MB)' }), {
+            status: 413, headers: { 'Content-Type': 'application/json' },
+          });
+        }
+        const ext = path.extname(filePath).toLowerCase();
+        const MIME_MAP: Record<string, string> = {
+          '.png': 'image/png', '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg',
+          '.gif': 'image/gif', '.webp': 'image/webp', '.svg': 'image/svg+xml',
+          '.avif': 'image/avif',
+          '.mp4': 'video/mp4', '.webm': 'video/webm', '.mov': 'video/quicktime',
+          '.mp3': 'audio/mpeg', '.wav': 'audio/wav', '.ogg': 'audio/ogg',
+          '.pdf': 'application/pdf', '.json': 'application/json',
+          '.html': 'text/html', '.txt': 'text/plain', '.mhtml': 'message/rfc822',
+        };
+        const contentType = MIME_MAP[ext] || 'application/octet-stream';
+        resetIdleTimer();
+        return new Response(Bun.file(filePath), {
+          headers: {
+            'Content-Type': contentType,
+            'Content-Length': String(stat.size),
+            'Content-Disposition': `inline; filename="${path.basename(filePath)}"`,
+            'Cache-Control': 'no-cache',
+          },
+        });
+      }
+
       // ─── Command endpoint (accepts both root AND scoped tokens) ────
       // Must be checked BEFORE the blanket root-only auth gate below,
       // because scoped tokens from /connect are valid for /command.