Fused Storage

Storage & secrets in fused

Part of the Fused skill set — don't work from it alone. Fused is workspace ⊃ project ⊃ UDF. If this is a step in building or running a project, load fused-projects for the end-to-end model and fused-execute for running code over the data. See fused-guide for the full set.

These tools are the find → load → explore front of the workflow: locate data, understand its shape, and move bytes in/out — before you run code over it with execute_code. All are always-on MCP tools (no --enable-infra / --enable-destructive flags needed) — except delete_secret, which requires --enable-destructive — and back the same operations as the CLI's files … / secrets … commands (see fused-cli). They operate on the active environment's storage backend — real S3 on AWS, the local filesystem on the local backend — so the same calls work on either of them.

Inspect before you compute

The cheapest way to avoid a wasted execute_code run is to look first. Typical flow: list_files → count_files → get_file_schema, then write code against a known schema.

list_files(bucket?, prefix?, page_size=100, page_token?)

• No bucket → lists buckets: {"buckets": [...], "count": N}.
• With bucket → one page of keys under prefix plus a next_page_token. Paginate: follow next_page_token in successive calls until it is null. Do not assume one call returns everything — page_size defaults to 100.

list_files()                                   # what buckets exist?
list_files(bucket="my-data", prefix="events/") # first page of keys
list_files(bucket="my-data", prefix="events/", page_token="<next>")  # next page

count_files(bucket, prefix?, extensions?)

Counts objects under a prefix, optionally filtered by extension — cheaper than paging all keys when you only need a tally or a partition count.

count_files(bucket="my-data", prefix="events/", extensions=[".parquet"])

get_file_schema(bucket, key)

Column schema + metadata for a Parquet, Arrow IPC, or CSV file — read this to learn column names/dtypes (and row count for Parquet/CSV) before writing a query. Arrow IPC reports record-batch count rather than a row count. Use it to confirm a file's shape so your execute_code doesn't guess column names.

get_file(bucket, key, expires_in=3600)

Returns a presigned download URL (default 1 h). For fetching a result or sample to the caller. On the local backend this is a short-lived HMAC-signed http:// URL served by a local file server, so it behaves like the cloud path. Treat the URL as a bearer token — anyone with it can fetch the object until it expires.

upload_file(bucket, key, content, base64_encoded=False)

Write content to object storage.

• Text (CSV, JSON): pass content as a UTF-8 string.
• Binary (Parquet, Arrow, images): base64-encode and set base64_encoded=True.

For large or computed outputs, prefer writing from inside execute_code (the code has direct S3 access via the execution role) rather than round-tripping bytes through the MCP boundary — see fused-execute.

Secrets

Secrets let executing code reach databases/APIs without hard-coding credentials. The model is the execution principal reads the store directly — values are never injected into the execute_code payload.

put_secret(name, value) — mind the name prefix

On AWS the Lambda execution role can only read secrets under the environment's function_prefix (e.g. openfused-*). put_secret enforces this at write time: a name outside the prefix is rejected with an actionable error rather than becoming an invisible-at-runtime secret. So name secrets openfused-<thing>:

put_secret(name="openfused-pg-conn", value="postgresql://user:pass@host/db")

Then read it inside the execution using openfused.get_secret — works on both AWS and the local backend (via the fused shim):

import openfused

conn = openfused.get_secret("openfused-pg-conn")
result = ...  # use conn

(examples/duckdb_with_secret.py is a full DuckDB-over-Postgres example.)

get_secret(name) / list_secrets(prefix?)

get_secret returns the secret string (raises if absent or binary — text only). list_secrets returns [{"name", "arn", …}], prefix-filtered. On the local backend secrets live in the OS keychain (one JSON blob per environment, keyed by the resolved store path); access control is OS-keychain, not IAM.

Linux/WSL has no native keychain. Where no usable OS keychain exists, every local secret operation raises a RuntimeError naming both remedies: install the file-based fallback (pip install keyrings.alt, which stores secrets unencrypted on disk — dev only) or switch to the AWS backend for headless/CI use. Without keyrings.alt and without a keychain, secrets simply fail loud.

delete_secret(name) — gated behind --enable-destructive

Unlike the always-on trio above, delete_secret is a destructive tool: it is only registered when the server runs with --enable-destructive (the same gate as env_delete/infra_teardown). It raises if the secret does not exist and returns {"deleted": name} on success. On AWS the secret is scheduled for deletion with the default 30-day recovery window (recoverable via AWS tooling until it elapses); on the local backend the name is removed from the keychain map immediately. CLI equivalent: fused secrets delete <name> (prompts; --yes to skip). Do not "delete" by overwriting with an empty value — that leaves a readable (empty) secret in place.

Notes

• Apart from delete_secret (--enable-destructive), these tools never gate on feature flags; they are available in every server session.
• For destructive cleanup of cached/spilled result objects use cache_clear (covered in fused-execute), not these tools.
• CLI equivalents: fused files list|count|get|schema|upload and fused secrets get|put|list|delete (fused-cli).