Agent Desktop Ffi

agent-desktop-ffi

Direct C-ABI access to every PlatformAdapter operation. Build the cdylib with the workspace's release-ffi profile:

cargo build --profile release-ffi -p agent-desktop-ffi

The output is target/release-ffi/libagent_desktop_ffi.dylib (.so on Linux, .dll on Windows) plus a committed C header at crates/ffi/include/agent_desktop.h.

A Python ctypes smoke harness lives at tests/ffi-python/smoke.py and serves as a worked end-to-end example covering the ABI handshake, struct size validation, ad_version, and the snapshot pipeline leg. See tests/ffi-python/README.md for usage.

Four reference topics, loaded as needed:

• ownership.md — who allocates / who frees, for every *mut T the FFI hands back to the caller.
• error-handling.md — errno-style last-error contract, enum validation, panic boundary.
• threading.md — host-thread contract, cross-process mutation serialization, AXIsProcessTrusted inheritance, and adapter-bound native handles.
• build-and-link.md — ABI handshake, struct size validation, minimal C and Python examples, observe-act workflow, and prebuilt archive locations.

Observe-act workflow (canonical path)

ad_init(AD_ABI_VERSION_MAJOR)                    // verify header ↔ dylib match
adapter = ad_adapter_create_with_session("s1")   // or ad_adapter_create()
rc = ad_snapshot(adapter, "Finder", 0, 10, false, false, &json_out)
// parse json_out: locate snapshot-qualified refs in data.tree
ad_free_string(json_out)
// build action:
AdAction act = {0}; act.kind = AD_ACTION_KIND_CLICK;
rc = ad_execute_by_ref(adapter, "@s8f3k2p9:e5", NULL, &act, 0, &result_out)
ad_free_string(result_out)
ad_adapter_destroy(adapter)

ad_snapshot returns a {version, ok, command, data} JSON envelope identical to the CLI output. The data.tree field contains snapshot-qualified ref IDs for interactive elements. Pass a qualified ref, or a legacy bare ref plus its explicit snapshot_id, to ad_execute_by_ref to drive the pipeline (RefStore load → strict resolution → actionability preflight → dispatch).

Core constraints

• ABI handshake. Call ad_init(AD_ABI_VERSION_MAJOR) once after loading the dylib. A mismatch between the compiled-in constant and the loaded dylib returns AD_RESULT_ERR_INVALID_ARGS — abort rather than proceed. You can also read the raw dylib major via ad_abi_version() for diagnostic display. New ad_* symbols and new error codes are additive (no bump required); removed or layout-changed symbols increment the major.

• Session adapters. ad_adapter_create_with_session("session-id") associates the adapter with a session namespace for refmap persistence — the same as CLI --session <id>. A null snapshot_id is valid only for a qualified ref; legacy bare @eN refs require an explicit snapshot ID. Session IDs: 1–64 chars, ASCII alphanumeric / - / _. Invalid IDs return null (check ad_last_error_*).

• Structured session trace (no ABI change). File-based JSONL tracing activates only when the session has a manifest with trace: on from session start (CLI) or equivalent on-disk setup. ad_adapter_create_with_session alone does not create trace files. When tracing is active, command_context()-backed commands append to one segment per OS process under ~/.agent-desktop/sessions/<id>/trace/<pid>-<procTs>.jsonl. A long-lived host reuses the same segment filename for all calls in that process. For unstructured diagnostics regardless of session manifest, use ad_set_log_callback (below).

• Threading and mutation leases. Adapter entrypoints may be called from any host thread. Native handles remain bound to their creating adapter and thread. Desktop mutations acquire the same canonical cross-process interaction lease as the CLI; reads carry finite deadlines without taking the mutation lock. See threading.md for the Apple documentation basis and the read/read, read/mutation, and mutation/mutation ordering matrix.

• Release profile. cargo build --release produces panic = "abort" — any Rust panic inside an extern "C" fn will SIGABRT the host. Use --profile release-ffi to get the correct panic = "unwind" profile. CI enforces this.

• Last-error lifetime. Pointers returned by ad_last_error_* remain valid across any number of subsequent successful FFI calls on the same thread. Only the next failing call rotates them. Cache the pointer once, read it as many times as you need.

• ad_last_error_details. A fourth accessor, ad_last_error_details(), returns a borrowed JSON string carrying structured details (e.g. the actionability check report on ACTION_FAILED, candidate summaries on AMBIGUOUS_TARGET). The details may contain element names, values, and window titles from the user's screen — treat as sensitive diagnostics and avoid routing to shared log surfaces.

• Handle release. Every ad_resolve_element_exact / ad_find_exact result must be released with ad_free_handle(adapter, &handle) on the same adapter that produced it, before that adapter is destroyed. On macOS this balances the internal CFRetain; on Windows/Linux the call is a no-op but safe to issue. ad_free_handle zeroes handle.ptr so a follow-up call is a safe no-op.

• Primary ref-action path. ad_execute_by_ref is the recommended entrypoint for the observe-act loop: it loads the RefStore, looks up the ref in the refmap (STALE_REF on miss), runs strict element re-identification (STALE_REF / AMBIGUOUS_TARGET), runs the live actionability preflight, then dispatches. TypeText and PressKey default to focus_fallback policy (matching CLI type/press-key); all other actions default to headless. Pass AD_POLICY_KIND_HEADED (2) to opt in to cursor-based fallbacks.

• Generation-safe direct APIs. Legacy AdRefEntry and AdWindowInfo layouts remain available for binary compatibility, but they do not carry process-generation evidence and direct targeting functions fail closed. Use AdExactRefEntry, AdExactWindowInfo, ad_list_windows_exact, and the *_exact targeting symbols. Likewise, ad_list_surfaces_exact preserves SurfaceInfo.id; the legacy surface list is an observation-only projection that omits it.

• Display discovery. Call ad_list_displays before using AdScreenshotTarget.screen_index. List order is the screenshot index order; inspect each AdDisplayInfo for its stable display ID, bounds, primary flag, and scale, then release the handle with ad_display_list_free.

• Low-level action paths. ad_execute_action (headless, no preflight) and ad_execute_action_with_policy are raw escape hatches for callers holding a live AdNativeHandle from ad_resolve_element_exact / ad_find_exact. Use them when you need to bypass the ref-action pipeline.

• Ref-action preflight. ad_execute_by_ref and ad_execute_ref_action_exact_with_policy both resolve the element strictly and run the live actionability preflight (visible, stable, enabled, supported action, policy, editable) before dispatching — a disabled or unsupported target fails before any platform call. On AD_RESULT_ERR_ACTION_FAILED, the structured check report is available as JSON via ad_last_error_details().

• Action result steps. AdActionResult.steps mirrors the CLI steps array for activation-chain actions. Each entry has label and outcome strings and is owned by the result; release with ad_free_action_result(&out).

• Tracing / log callback. Two tracing surfaces coexist:

  1. Structured file trace — same JSONL contract as CLI --trace, gated by a trace: on session manifest. Segments include event, ts_ms, seq, and redacted fields. Requires session start (or equivalent manifest on disk) before creating the adapter; plain session-id adapters write nothing to disk.

  2. ad_set_log_callback(cb) — installs a tracing subscriber layer that delivers events as JSON to your callback. cb receives an int32_t level (1=ERROR … 5=TRACE) and a const char *msg valid only for the duration of the call. Pass NULL to unregister. The layer is installed on the first non-null call; if a foreign global subscriber already owns the process at that point, the install fails with AD_RESULT_ERR_INTERNAL and no events are ever delivered. Sensitive field values (password, token, text, …) are replaced with {"redacted":true} before formatting. A panicking callback is caught and silently discarded. The callback may fire from threads other than the registering thread, and may still fire briefly after a NULL unregister — keep the callback and any data it captures valid for the process lifetime.

• Wait. ad_wait(adapter, args, &out) runs the full CLI wait command (element-appear, window-appear, text-appear, menu-open/close, notification, element predicates). Zero-initialize AdWaitArgs, set the fields you need, and validate the struct size against AD_WAIT_ARGS_SIZE / ad_wait_args_size() before calling. The output is a {version, ok, command, data} JSON envelope freed with ad_free_string. ad_wait blocks the calling thread up to timeout_ms ms — ensure the adapter is not destroyed from another thread while it is running.

• Text input privacy. On macOS, focus-fallback or headed text insertion may briefly use the clipboard for non-ASCII text. For sensitive text, prefer AD_ACTION_KIND_SET_VALUE with AD_POLICY_KIND_HEADLESS when the target supports settable values.

• Enum discriminants. Every #[repr(i32)] enum field is validated at the C boundary — invalid discriminants return AD_RESULT_ERR_INVALID_ARGS instead of undefined behavior.

• ABI stability. The major version in AD_ABI_VERSION_MAJOR increments on any breaking change (removed symbol, incompatible layout). Additive changes (new symbols, new error codes) do not bump it. Before 1.0, pin the exact version of libagent_desktop_ffi you link against.

• ad_get_tree_exact vs ad_snapshot. ad_get_tree_exact returns a raw flat BFS tree without @e refs, no refmap persistence, and no JSON envelope — use it for custom traversal or UI inspection. For observe-act agents that drive actions via ad_execute_by_ref, always start with ad_snapshot.