agent-desktop-ffi
Direct C-ABI access to the same operations as the CLI. Build the cdylib with the
release-ffi profile:
cargo build --profile release-ffi -p agent-desktop-ffi
The output is target/release-ffi/libagent_desktop_ffi.dylib (.so on Linux,
agent_desktop_ffi.dll on Windows) plus a committed C header at
crates/ffi/include/agent_desktop.h. The default release profile uses
panic = "abort": a Rust panic inside an extern "C" function would SIGABRT
the host. release-ffi keeps panic = "unwind", which the panic traps need.
Four reference topics, loaded as needed:
- ownership.md: who allocates and who frees every
pointer the FFI hands back.
- error-handling.md: errno-style last-error
contract, error codes, enum validation, panic boundary.
- threading.md: host-thread contract, mutation
ordering across threads and processes, adapter-bound native handles,
AXIsProcessTrusted inheritance.
- build-and-link.md: prebuilt archives, ABI
handshake, struct size validation, C and Python examples.
Observe-act workflow (canonical path)
ad_init(AD_ABI_VERSION_MAJOR) // verify header ↔ dylib match
adapter = ad_adapter_create_with_session("s1") // or ad_adapter_create()
rc = ad_snapshot(adapter, "Finder", 0, 10, false, false, &json_out)
// parse json_out: locate snapshot-qualified refs in data.tree
ad_free_string(json_out)
// build action:
AdAction act = {0}; act.kind = AD_ACTION_KIND_CLICK;
rc = ad_execute_by_ref(adapter, "@s8f3k2p9:e5", &act, 0, &result_out)
ad_free_string(result_out)
ad_adapter_destroy(adapter)
ad_snapshot returns the CLI envelope {version, ok, command, data}.
data.tree carries snapshot-qualified refs for interactive elements. Read
data.complete before trusting a tree: false means the observation budget ran
out and the tree is partial, not an error. ad_snapshot reads the owner selected
by surface. On macOS, menu and menubar are application-owned, need no
document window, and omit data.window; window-owned snapshots include it.
Window targeting, skeleton mode and --root drill-down are not exposed over the
ABI. A bare @eN ref returns
AD_RESULT_ERR_INVALID_ARGS.
Core constraints
ABI handshake. Call ad_init(AD_ABI_VERSION_MAJOR) once after loading the
library. A mismatch returns AD_RESULT_ERR_INVALID_ARGS: abort. ad_abi_version()
returns the library's major for display. The major increments on a removed
symbol or a layout change; new ad_* symbols and error codes are additive.
Pin the exact libagent_desktop_ffi version you link against.
Session adapters. ad_adapter_create_with_session("session-id") scopes
refmap persistence to a session namespace, like CLI --session <id>. IDs are 1
to 64 ASCII alphanumeric, -, or _ characters. A null ID behaves like
ad_adapter_create(). An invalid ID returns null (read ad_last_error_*).
Ref actions. ad_execute_by_ref is the primary action entrypoint. It loads
the RefStore, resolves the ref strictly (STALE_REF when the snapshot has no
such ref or the element changed, AMBIGUOUS_TARGET on several candidates), runs
the live actionability preflight (visible, stable, enabled, supported action,
policy, editable), then dispatches. A disabled or unsupported target fails
before any platform call; on AD_RESULT_ERR_ACTION_FAILED the check report is
JSON in ad_last_error_details(). It auto-waits 5000 ms before the preflight;
ad_execute_by_ref_timeout takes timeout_ms (-1 = 5000 ms default, 0 =
single-shot). PressKey defaults to focus_fallback; every other action,
including TypeText, defaults to headless. Pass AD_POLICY_KIND_FOCUS_FALLBACK
(1) to allow focus without cursor movement, or AD_POLICY_KIND_HEADED (2) to allow
physical cursor and keyboard delivery. ad_execute_ref_action_exact_with_policy
always fails with AD_RESULT_ERR_ACTION_NOT_SUPPORTED: an exact entry has no
retained element token and no capture proof, so it cannot be re-identified.
Use ad_execute_by_ref instead.
Generation-safe direct APIs. Every direct targeting function takes
process-generation evidence: AdExactRefEntry, AdExactWindowInfo,
ad_list_windows_exact, ad_launch_app_exact, and the other *_exact
symbols. AdRefEntry, AdWindowInfo, and AdSurfaceInfo exist
only as fields embedded in the exact structs. ad_list_surfaces_exact keeps the
surface id. From AD_EXACT_SURFACE_INFO_VERSION 2, AdExactSurfaceInfo.unclassified
names the surface kinds its probe could not read (for example sheet,menu), or
is null. A missing sheet or menu next to a listed kind is unknown, not
absent. Only the Windows adapter fills it; macOS and Linux return null.
Native handles. ad_find_exact returns an AdNativeHandle. It re-identifies the
element before publishing it: it raises the landed handle against the selected entry's
recorded role, native id, name and description, and returns AD_RESULT_ERR_STALE_REF
instead of a handle when a different control occupies the anchored path.
ad_resolve_element_exact takes an exact ref entry and always fails with
AD_RESULT_ERR_ACTION_NOT_SUPPORTED for the same reason as the exact action call.
ad_execute_action (headless, no preflight) and
ad_execute_action_with_policy act on a handle directly, bypassing the ref-action
pipeline. Release every handle with ad_free_handle(adapter, &handle) on the
same adapter and thread, before ad_adapter_destroy, on every platform. It
zeroes handle.ptr, so a second call is a no-op.
Action results. AdActionResult.steps mirrors the CLI steps array for
activation-chain actions. Each AdActionStep has label and outcome strings,
mechanism (AD_STEP_MECHANISM_SEMANTIC_API = 1 or
AD_STEP_MECHANISM_PHYSICAL_SYNTHETIC = 2, valid only when has_mechanism),
and verified (valid only when has_verified). AdActionResult.disposition
carries the same delivery/retry pair as ad_last_error_delivery_semantics.
Release the result with ad_free_action_result(&out).
Display discovery. Call ad_list_displays before using
AdScreenshotTarget.screen_index. List order is the screenshot index order.
Each AdDisplayInfo carries a stable display ID, bounds, primary flag, and scale.
Release the list with ad_display_list_free.
Errors. A failing call sets thread-local last-error state; read it with the
ad_last_error_* accessors before the next failing call. ad_last_error_details()
returns structured JSON (the actionability report on ACTION_FAILED, candidate
summaries on AMBIGUOUS_TARGET) that can contain element names, values, and
window titles from the user's screen: keep it out of shared logs.
ad_last_error_delivery_semantics reports whether the failed action was
delivered and whether a retry is safe. Invalid enum discriminants return
AD_RESULT_ERR_INVALID_ARGS. See error-handling.md.
Threading. Entrypoints may be called from any host thread. Native handles
stay on the adapter and thread that created them; pass qualified refs across
threads. Desktop mutations take the same cross-process interaction lease as the
CLI; reads take no lease and carry finite deadlines. See
threading.md for the read/read, read/mutation, and
mutation/mutation matrix.
Tracing. File tracing (same JSONL contract as CLI --trace; segments carry
event, ts_ms, seq, and redacted fields) is active only when the session has
a manifest with trace: on, created by session start (or an equivalent manifest
on disk) before the adapter is created. ad_adapter_create_with_session alone
writes nothing. Each OS process appends to one segment,
~/.agent-desktop/sessions/<id>/trace/<pid>-<procTs>.jsonl, and a long-lived host
reuses it. ad_trace_show (JSON envelope) and ad_trace_export (self-contained
HTML file) read the adapter session's merged timeline; limit -1 = CLI default.
Log callback. ad_set_log_callback(cb) registers one process-wide callback
for diagnostics regardless of session manifest. It receives, as JSON, the
tracing events emitted inside later ad_* calls on the same thread; the FFI
never installs or replaces the host process's global tracing subscriber. cb
receives an int32_t level (1=ERROR ... 5=TRACE) and a const char *msg valid
only for the duration of the call. Pass NULL to unregister. Sensitive field
values (password, token, text, ...) are replaced with {"redacted":true}. The
callback must not unwind across the C ABI: catch C++ exceptions and Rust panics
inside it, because a violation may abort the host. It may fire from several host
threads at once and briefly after a NULL unregister: keep the callback and
any data it captures valid for the process lifetime.
Wait. ad_wait(adapter, args, &out) runs the CLI wait command. Set
mode.pause (AdOptionalU64, ms), mode.element, mode.window (waits for a
window with that title), mode.text, or one of mode.surfaces.menu,
.menu_closed, .notification; add the predicate group (predicate,
value, action, count) and scope.timeout_ms / scope.app. The CLI event
wait (--event, --window-id) is not exposed. Zero-initialize AdWaitArgs and
validate the struct size against AD_WAIT_ARGS_SIZE / ad_wait_args_size()
before calling. The output is a {version, ok, command, data} envelope freed with
ad_free_string. ad_wait blocks the calling thread up to scope.timeout_ms ms. A
concurrent ad_adapter_destroy does not invalidate an in-flight wait.
Text input privacy. On macOS, focus-fallback or headed text insertion may
briefly use the clipboard for non-ASCII text. For sensitive text, prefer
AD_ACTION_KIND_SET_VALUE with AD_POLICY_KIND_HEADLESS when the target
supports settable values.
ad_get_tree_exact vs ad_snapshot. ad_get_tree_exact returns a raw flat
breadth-first tree: no refs, no refmap persistence, no JSON envelope (free it
with ad_free_tree). Use it for custom traversal or UI inspection. For
observe-act agents that drive actions via ad_execute_by_ref, start with
ad_snapshot.
1---2name: agent-desktop-ffi3tags: ffi, c-bindings, cdylib, python, swift, node, go, rust-ffi4requirements:5 - agent-desktop-ffi6description: >7 C-ABI bindings over agent-desktop's PlatformAdapter. Consumers8 (Python ctypes, Swift, Node ffi-napi, Go cgo, C++, Ruby fiddle)9 link libagent_desktop_ffi.{dylib,so,dll} and call `ad_*` functions10 directly instead of spawning the CLI binary per call. The canonical11 observe-act workflow is: ad_init → ad_adapter_create[_with_session]12 → ad_snapshot → read qualified refs (@s8f3k2p9:e5) → ad_execute_by_ref → ad_free_string13 → ad_adapter_destroy.14metadata:15 source: https://skillmd.com/skills/vercel-labs/agent-desktop-ffi16---1718# agent-desktop-ffi1920Direct C-ABI access to the same operations as the CLI. Build the cdylib with the21`release-ffi` profile:2223```sh24cargo build --profile release-ffi -p agent-desktop-ffi25```2627The output is `target/release-ffi/libagent_desktop_ffi.dylib` (`.so` on Linux,28`agent_desktop_ffi.dll` on Windows) plus a committed C header at29`crates/ffi/include/agent_desktop.h`. The default `release` profile uses30`panic = "abort"`: a Rust panic inside an `extern "C"` function would `SIGABRT`31the host. `release-ffi` keeps `panic = "unwind"`, which the panic traps need.3233Four reference topics, loaded as needed:3435- [ownership.md](references/ownership.md): who allocates and who frees every36 pointer the FFI hands back.37- [error-handling.md](references/error-handling.md): errno-style last-error38 contract, error codes, enum validation, panic boundary.39- [threading.md](references/threading.md): host-thread contract, mutation40 ordering across threads and processes, adapter-bound native handles,41 AXIsProcessTrusted inheritance.42- [build-and-link.md](references/build-and-link.md): prebuilt archives, ABI43 handshake, struct size validation, C and Python examples.4445## Observe-act workflow (canonical path)4647```48ad_init(AD_ABI_VERSION_MAJOR) // verify header ↔ dylib match49adapter = ad_adapter_create_with_session("s1") // or ad_adapter_create()50rc = ad_snapshot(adapter, "Finder", 0, 10, false, false, &json_out)51// parse json_out: locate snapshot-qualified refs in data.tree52ad_free_string(json_out)53// build action:54AdAction act = {0}; act.kind = AD_ACTION_KIND_CLICK;55rc = ad_execute_by_ref(adapter, "@s8f3k2p9:e5", &act, 0, &result_out)56ad_free_string(result_out)57ad_adapter_destroy(adapter)58```5960`ad_snapshot` returns the CLI envelope `{version, ok, command, data}`.61`data.tree` carries snapshot-qualified refs for interactive elements. Read62`data.complete` before trusting a tree: `false` means the observation budget ran63out and the tree is partial, not an error. `ad_snapshot` reads the owner selected64by `surface`. On macOS, `menu` and `menubar` are application-owned, need no65document window, and omit `data.window`; window-owned snapshots include it.66Window targeting, skeleton mode and `--root` drill-down are not exposed over the67ABI. A bare `@eN` ref returns68`AD_RESULT_ERR_INVALID_ARGS`.6970## Core constraints7172- **ABI handshake.** Call `ad_init(AD_ABI_VERSION_MAJOR)` once after loading the73 library. A mismatch returns `AD_RESULT_ERR_INVALID_ARGS`: abort. `ad_abi_version()`74 returns the library's major for display. The major increments on a removed75 symbol or a layout change; new `ad_*` symbols and error codes are additive.76 Pin the exact `libagent_desktop_ffi` version you link against.7778- **Session adapters.** `ad_adapter_create_with_session("session-id")` scopes79 refmap persistence to a session namespace, like CLI `--session <id>`. IDs are 180 to 64 ASCII alphanumeric, `-`, or `_` characters. A null ID behaves like81 `ad_adapter_create()`. An invalid ID returns null (read `ad_last_error_*`).8283- **Ref actions.** `ad_execute_by_ref` is the primary action entrypoint. It loads84 the RefStore, resolves the ref strictly (`STALE_REF` when the snapshot has no85 such ref or the element changed, `AMBIGUOUS_TARGET` on several candidates), runs86 the live actionability preflight (visible, stable, enabled, supported action,87 policy, editable), then dispatches. A disabled or unsupported target fails88 before any platform call; on `AD_RESULT_ERR_ACTION_FAILED` the check report is89 JSON in `ad_last_error_details()`. It auto-waits 5000 ms before the preflight;90 `ad_execute_by_ref_timeout` takes `timeout_ms` (`-1` = 5000 ms default, `0` =91 single-shot). `PressKey` defaults to `focus_fallback`; every other action,92 including `TypeText`, defaults to `headless`. Pass `AD_POLICY_KIND_FOCUS_FALLBACK`93 (1) to allow focus without cursor movement, or `AD_POLICY_KIND_HEADED` (2) to allow94 physical cursor and keyboard delivery. `ad_execute_ref_action_exact_with_policy`95 always fails with `AD_RESULT_ERR_ACTION_NOT_SUPPORTED`: an exact entry has no96 retained element token and no capture proof, so it cannot be re-identified.97 Use `ad_execute_by_ref` instead.9899- **Generation-safe direct APIs.** Every direct targeting function takes100 process-generation evidence: `AdExactRefEntry`, `AdExactWindowInfo`,101 `ad_list_windows_exact`, `ad_launch_app_exact`, and the other `*_exact`102 symbols. `AdRefEntry`, `AdWindowInfo`, and `AdSurfaceInfo` exist103 only as fields embedded in the exact structs. `ad_list_surfaces_exact` keeps the104 surface `id`. From `AD_EXACT_SURFACE_INFO_VERSION` 2, `AdExactSurfaceInfo.unclassified`105 names the surface kinds its probe could not read (for example `sheet,menu`), or106 is null. A missing `sheet` or `menu` next to a listed kind is unknown, not107 absent. Only the Windows adapter fills it; macOS and Linux return null.108109- **Native handles.** `ad_find_exact` returns an `AdNativeHandle`. It re-identifies the110 element before publishing it: it raises the landed handle against the selected entry's111 recorded role, native id, name and description, and returns `AD_RESULT_ERR_STALE_REF`112 instead of a handle when a different control occupies the anchored path.113 `ad_resolve_element_exact` takes an exact ref entry and always fails with114 `AD_RESULT_ERR_ACTION_NOT_SUPPORTED` for the same reason as the exact action call.115 `ad_execute_action` (headless, no preflight) and116 `ad_execute_action_with_policy` act on a handle directly, bypassing the ref-action117 pipeline. Release every handle with `ad_free_handle(adapter, &handle)` on the118 same adapter and thread, before `ad_adapter_destroy`, on every platform. It119 zeroes `handle.ptr`, so a second call is a no-op.120121- **Action results.** `AdActionResult.steps` mirrors the CLI `steps` array for122 activation-chain actions. Each `AdActionStep` has `label` and `outcome` strings,123 `mechanism` (`AD_STEP_MECHANISM_SEMANTIC_API` = 1 or124 `AD_STEP_MECHANISM_PHYSICAL_SYNTHETIC` = 2, valid only when `has_mechanism`),125 and `verified` (valid only when `has_verified`). `AdActionResult.disposition`126 carries the same delivery/retry pair as `ad_last_error_delivery_semantics`.127 Release the result with `ad_free_action_result(&out)`.128129- **Display discovery.** Call `ad_list_displays` before using130 `AdScreenshotTarget.screen_index`. List order is the screenshot index order.131 Each `AdDisplayInfo` carries a stable display ID, bounds, primary flag, and scale.132 Release the list with `ad_display_list_free`.133134- **Errors.** A failing call sets thread-local last-error state; read it with the135 `ad_last_error_*` accessors before the next failing call. `ad_last_error_details()`136 returns structured JSON (the actionability report on `ACTION_FAILED`, candidate137 summaries on `AMBIGUOUS_TARGET`) that can contain element names, values, and138 window titles from the user's screen: keep it out of shared logs.139 `ad_last_error_delivery_semantics` reports whether the failed action was140 delivered and whether a retry is safe. Invalid enum discriminants return141 `AD_RESULT_ERR_INVALID_ARGS`. See [error-handling.md](references/error-handling.md).142143- **Threading.** Entrypoints may be called from any host thread. Native handles144 stay on the adapter and thread that created them; pass qualified refs across145 threads. Desktop mutations take the same cross-process interaction lease as the146 CLI; reads take no lease and carry finite deadlines. See147 [threading.md](references/threading.md) for the read/read, read/mutation, and148 mutation/mutation matrix.149150- **Tracing.** File tracing (same JSONL contract as CLI `--trace`; segments carry151 `event`, `ts_ms`, `seq`, and redacted fields) is active only when the session has152 a manifest with `trace: on`, created by `session start` (or an equivalent manifest153 on disk) before the adapter is created. `ad_adapter_create_with_session` alone154 writes nothing. Each OS process appends to one segment,155 `~/.agent-desktop/sessions/<id>/trace/<pid>-<procTs>.jsonl`, and a long-lived host156 reuses it. `ad_trace_show` (JSON envelope) and `ad_trace_export` (self-contained157 HTML file) read the adapter session's merged timeline; `limit` `-1` = CLI default.158159- **Log callback.** `ad_set_log_callback(cb)` registers one process-wide callback160 for diagnostics regardless of session manifest. It receives, as JSON, the161 `tracing` events emitted inside later `ad_*` calls on the same thread; the FFI162 never installs or replaces the host process's global tracing subscriber. `cb`163 receives an `int32_t` level (1=ERROR ... 5=TRACE) and a `const char *msg` valid164 only for the duration of the call. Pass `NULL` to unregister. Sensitive field165 values (password, token, text, ...) are replaced with `{"redacted":true}`. The166 callback must not unwind across the C ABI: catch C++ exceptions and Rust panics167 inside it, because a violation may abort the host. It may fire from several host168 threads at once and briefly after a `NULL` unregister: keep the callback and169 any data it captures valid for the process lifetime.170171- **Wait.** `ad_wait(adapter, args, &out)` runs the CLI `wait` command. Set172 `mode.pause` (`AdOptionalU64`, ms), `mode.element`, `mode.window` (waits for a173 window with that title), `mode.text`, or one of `mode.surfaces.menu`,174 `.menu_closed`, `.notification`; add the `predicate` group (`predicate`,175 `value`, `action`, `count`) and `scope.timeout_ms` / `scope.app`. The CLI event176 wait (`--event`, `--window-id`) is not exposed. Zero-initialize `AdWaitArgs` and177 validate the struct size against `AD_WAIT_ARGS_SIZE` / `ad_wait_args_size()`178 before calling. The output is a `{version, ok, command, data}` envelope freed with179 `ad_free_string`. `ad_wait` blocks the calling thread up to `scope.timeout_ms` ms. A180 concurrent `ad_adapter_destroy` does not invalidate an in-flight wait.181182- **Text input privacy.** On macOS, focus-fallback or headed text insertion may183 briefly use the clipboard for non-ASCII text. For sensitive text, prefer184 `AD_ACTION_KIND_SET_VALUE` with `AD_POLICY_KIND_HEADLESS` when the target185 supports settable values.186187- **`ad_get_tree_exact` vs `ad_snapshot`.** `ad_get_tree_exact` returns a raw flat188 breadth-first tree: no refs, no refmap persistence, no JSON envelope (free it189 with `ad_free_tree`). Use it for custom traversal or UI inspection. For190 observe-act agents that drive actions via `ad_execute_by_ref`, start with191 `ad_snapshot`.