README.md 8.0 KB
description: "Run Cua Driver computer-use tools from its native npm SDK, with durable screenshots and explicit host desktop permissions."
kind: "package-reference"
@deepseek-ai/dsh-experimental-computer-use-cua-driver-native
English | 中文
Summary
Use Cua Driver to inspect and operate desktop windows without installing its separate CLI or application. The native npm dependency runs inside the DSH host and exposes Cua Driver's own tools. Screenshots reach image-capable models through durable attachments. This published experimental package requires the launching host's desktop permissions and remains an explicit composition choice.
Table of Contents
- Use this package
- Understand the implementation
- Further Exploration
- Model Experience
- Known Limitations and Deferred Work
- Dev Note
Use this package
Mount the provider in a composition that already supplies the tool registry and system prompt.
Minimal configuration
- name: '@deepseek-ai/dsh-computer-use'
- name: '@deepseek-ai/dsh-experimental-computer-use-cua-driver-native'
The provider has no configuration fields. It loads the exact Cua Driver npm version declared in package.json and uses its same-process defaults. Native import, runtime initialization, malformed catalog, duplicate tool name, or occupied computer-use registration failures reject activation and roll back owned resources. The registered provider name is cua-driver-native.
Use an attachment store and a model route that explicitly declares image input to receive screenshots. The MCP result adapter owns image admission and diagnostic behavior; programmatic callers retain the canonical raw result when a model cannot receive its images. Calls use Cua Driver's upstream tool parameters and results.
Host requirements
The native dependency supplies platform binaries through npm optional dependencies. Keep optional dependencies enabled. Grant desktop permissions to the application that launches DSH; this provider neither installs a permission-owning app nor changes OS grants. The native runtime shares the host process, so native crashes can terminate that process. Use the installed MCP provider when the separate Cua Driver application should own permissions and execution.
Verify the installed SDK
From the repository root, run this opt-in check against the installed native dependency. It discovers tools, reads permission status with prompt: false, and verifies teardown; it captures no screenshots, sends no input, and requests no OS permissions. Clearing NODE_USE_ENV_PROXY prevents Node from installing the launching shell's proxy before test setup.
env -u NODE_USE_ENV_PROXY DSH_COMPUTER_USE_NATIVE_E2E=1 node node_modules/vitest/vitest.mjs run --config vitest.e2e.config.ts packages/experimental/computer-use-cua-driver-native/tests/native.e2e.ts
Understand the implementation
Implementation internals — click to expand
The provider reserves the shared computer-use registration before loading native code. A child plugin owns discovery, model tools, guidance, and the native runtime. The parent retains the registration until child teardown has removed tools, interrupted native calls and image-capability admission, awaited settlement, and completed native shutdown. Cancellation does not undo input already delivered to an application. | File | Role | |---|---| | [src/index.ts](src/index.ts) | Native runtime ownership, catalog validation, tool registration, and provider guidance | | — | No runtime invariant companion is published; resource ownership has no independently observed state to compare. | Tool definitions reuse the existing MCP result adapter. Cua Driver's JSON catalog determines the schemas; its raw result supplies canonical text, structured output, and image bytes. The computer-use service carries only the provider name and exclusive registration.Further Exploration
- Computer-use service — exclusive named registration.
- MCP client — shared result and image projection.
- Cua Driver SDK — upstream runtime API and host facilities.
Model Experience
System prompt
What the model sees
The provider contributes the following computer-use guidance while its native tools are mounted.
Native Cua Driver guidance
Cua Driver native computer-use tools operate the host desktop. Discover the exact app and window, then get a fresh window snapshot before acting. Use element_token from that snapshot, or coordinates from its screenshot. A new snapshot of that window invalidates its earlier element tokens. Select either target or the legacy pid/window_id fields; do not combine them.
Prefer background delivery. A refusal does not authorize a foreground retry. Verify the requested outcome from fresh state after an action; a delivered click alone does not prove the outcome. After cancellation, inspect current state before retrying because completed input is not rolled back. Other sessions and applications may change the same desktop.
On macOS, cursor-overlay operations may return facility_unavailable even when screenshots and input work.
Token effect
This fixed guidance adds system-prompt tokens while the provider is mounted. Upstream guidance resources are not automatically loaded.
KV Cache effect
The unchanged guidance preserves its repeated prompt prefix. Mounting, removing, or editing it changes that prefix and can reduce cache reuse.
Discovered Cua Driver tools and results
What the model sees
Tools use the cua_driver_native__ prefix followed by the upstream name and retain the upstream descriptions and input schemas. Upstream tool refusals become tool errors. Supported screenshots appear as durable image references beside result text; the canonical raw result remains available to programmatic callers.
Token effect
The discovered catalog adds tool definitions to each request. Accessibility trees, result text, and admitted screenshots add per-call context. Raw base64 remains in execution-local canonical values and is not copied into model history.
KV Cache effect
An unchanged catalog preserves its tool-definition prefix. Tool results append to Session history. Replacing the provider or its catalog changes the model-visible tools and can reduce prefix reuse.
Known Limitations and Deferred Work
The package preserves the upstream driver's platform and application limits.
- Host permissions and graphics session — npm installation does not grant desktop access or create a graphical session.
- Native cursor overlay — a headless macOS Node host can receive
facility_unavailablefor overlay operations while screenshots and background input remain usable. - Shared desktop — the provider does not reserve windows or complete workflows for a Session. Other callers and applications can change the same desktop between calls.
- Cancellation — an aborted call can have delivered input already; inspect fresh state before retrying. The provider waits for SDK shutdown during unload but does not promise native action rollback.
- Failed shutdown — if native shutdown fails, the registration remains occupied. Restart the host before mounting another computer-use provider.
- Experimental release — tool schemas follow the pinned upstream SDK and have no DSH stability promise.