Needle — on-device tool calling
Needle is a foundation model built for tiny devices: a single 121M-parameter
"Simple Attention Network" quantised to 2-bit, shipped as one ~35 MB weights
file (needle3.cact) plus an engine library under 1 MB. It runs offline on a
CPU — roughly 100 MB of RAM, hundreds of tokens per second on a laptop — with
no API key, no GPU and no network. It does three things:
- Tool calls — given your function schemas and a plain-English request, it
picks the right function and fills every argument from what was said. Ask
for two things and you get two calls in order; ask for something no tool
covers and you get an empty list, not a guess.
- Structured extraction — declare a shape, hand over messy text, get
typed fields back; the decode grammar guarantees the output parses.
- Text embeddings — a vector for a sentence (Needle 3 only).
Everything below was verified live against cactus-needle 3.0.2 (Python
API, CLI, and the standalone engine on macOS arm64, with source-level checks
of the Windows/Linux paths).
The model is tiny (121M params). It is reliable when tools are designed for
it — the rules in "Tool design" below are not optional polish, they are the
difference between a working agent and a flaky one.
Pick a surface
| Your situation |
Surface |
Read |
| Python app/script; agent loop that executes the tools it calls |
Python API — import needle |
references/python-api.md |
| One-shot call generation from a terminal; deterministic output |
needle run CLI (needs jax + a 242 MB checkpoint) |
references/cli.md |
| No Python at runtime — servers, Windows services, edge devices, embedding in C |
Standalone engine binary — ./needle --model needle3.cact |
references/engine-binary.md |
| The model picked the wrong tool / refused / looped / mangled args |
Tool design rules (read this before debugging anything else) |
references/tool-design.md |
| It errored or behaved oddly |
Symptom table |
references/troubleshooting.md |
Install
pip install cactus-needle # Python 3.9+; macOS, Linux, Windows
- On Windows, prefer
py -m pip install cactus-needle. The engine ships
as a prebuilt wheel (libneedle3.dll) — no compiler needed.
- First use auto-downloads the engine +
needle3.cact weights into
~/.cache/cactus-needle/v3/<engine-version>/ (~36 MB; on Windows:
%USERPROFILE%\.cache\cactus-needle\v3\<version>\). Set
NEEDLE3_LIB_PATH to override the library location.
- If the machine must never touch the network, pre-seed that cache from
another box, or use the standalone engine bundle (below) which needs no
Python at runtime.
- Telemetry is on by default. Disable before importing:
NEEDLE_TELEMETRY=0 and DO_NOT_TRACK=1.
- Multi-Python gotcha: on machines with several Pythons (Homebrew vs
python.org vs system), the
needle CLI lives in the interpreter's bin dir
that pip installed it. If import needle fails under python3, find the
right interpreter (ls */bin/needle, pip show cactus-needle) — or see
the re-exec pattern in references/troubleshooting.md.
Quick start (Python API)
import needle
@needle.tool
def get_weather(city: str) -> dict:
"""Get the current weather for a city."""
return {"city": city, "temp_c": 27, "sky": "clear"}
agent = needle.Needle(tools=[get_weather])
response = agent.run("what's it like in Lagos right now?")
print(response["results"]) # [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
The decorator reads the signature for argument types and the docstring for
the tool description (an Args: section documents parameters). run() picks
the tool, executes it in-process, feeds the return value back to the
model, and returns the final response. Full API — including Field
constraints, extraction and embeddings — in references/python-api.md.
Tool design — the rules that make a 121M model reliable
These came from live failure modes, not style guides. Details and code
patterns in references/tool-design.md.
- Give every tool
triggers. @needle.tool(triggers=["list databases", ...]). Without them, tool selection is flaky — the model intermittently
refuses a perfectly matching tool ("no connectivity or network tools
available", often with high confidence).
- Never make secrets tool arguments. Needle's strict grounding blocks
arguments it cannot verify against the input (
ungrounded password), and
secrets should not travel through an LLM anyway. Bake credentials into
constants; let the model pick what to do, not recite keys.
- Keep the result you return small.
run() feeds your tool's return
value verbatim back to the model. A multi-kilobyte result makes a 121M
model lose the thread and repeat unrelated calls until max_steps. Return
a compact digest and keep the full payload in a variable the caller reads.
- Two asks per turn, maximum. "Do X then Y" reliably yields two calls in
order; a three-part compound drops the third. Split into follow-up
questions.
- Judge success by
response["results"], not by the final turn. After
execution the final response has function_calls: [] and
type: "respond" — that is success, not refusal.
- Constrain arguments in the grammar, don't hope.
Field/Annotated
with ge/le, pattern, enum, const, or Literal types make invalid
values unrepresentable. The tiny model will otherwise fill host='mysql'
from the word "MySQL" in your prompt.
- The Python engine injects a date fact (
date: YYYY-MM-DD HH:MM) into
every prompt by default; the minutes drift between runs and can flip tool
selection. For reproducibility pass a fixed system= and
auto_date=False. The CLI and standalone engine don't inject dates.
Standalone engine (no Python at runtime)
The raw-C engine binary runs the same needle3.cact weights with no JAX, no
Python — ideal for servers and Windows boxes:
# macOS / Linux — download a platform bundle and place the weights beside it
python scripts/bootstrap_engine.py # auto-detects this platform
./macos-arm64/needle --model needle3.cact --tools tools.json \
--prompt "give me revenue breakdown by country"
# Windows
python scripts\bootstrap_engine.py --platform windows-x86_64
.\windows-x86_64\needle.exe --model needle3.cact --tools tools.json --prompt "..."
The binary is call-selection only: it emits the chosen call as JSON
(name, arguments, reasoning, confidence, throughput stats) and does not
execute anything — your code runs the tool. It is deterministic (greedy
decode), needs ~100 MB RAM, and can also serve HTTP (--serve, default port
8080: POST /complete {"input": "..."}, POST /reset). Platforms:
macos-arm64, linux-x86_64/arm64/armv7/riscv64/mipsel,
windows-x86_64/arm64, plus android, ios, tvos, watchos and wasm variants.
Full details in references/engine-binary.md.
Bundled scripts
scripts/bootstrap_engine.py — jax-free download of any platform's engine
bundle + weights, prints the exact run command. Cross-platform.
scripts/export_tools.py — dump the @needle.tool schemas from a Python
module into tools.json for the engine binary or needle run --tools.
Troubleshooting quick table
| Symptom |
Cause → fix |
ModuleNotFoundError: needle |
Wrong interpreter → references/troubleshooting.md §1 |
Result says ungrounded password/ungrounded <arg> |
Grounding blocked a secret/fabricated value → §2 |
Model fills nonsense (host='mysql') |
Unconstrained string arg → triggers + constraints → §3 |
| Same tool called repeatedly until max_steps |
Tool result too large fed back → digest+stash → §4 |
| Third request of a compound ask never happens |
2-ask limit → split the ask → §5 |
| Refusal with high confidence, no call |
Missing triggers / date-fact drift → §6 |
needle run/build demand jax |
Use the engine binary path instead → §7 |
| Garbled unicode in outputs |
Charset on the wrapped CLI / corrupted source data → §8 |
| Nothing happens on macOS double-click |
Window flashes closed — run from Terminal → §9 |
Expanded causes and fixes: references/troubleshooting.md.
1---2name: needle3description: Run Cactus Compute Needle — the on-device foundation model (121M params, 2-bit, ~35 MB weights + <1 MB engine) for tool calling, function calling, structured extraction and text embeddings that runs entirely offline on CPU across macOS, Linux and Windows. Use this skill whenever the user mentions needle, cactus-needle, cactus compute, .cact archives, on-device / offline / local-first / edge LLM tool calling or function calling, an agent that picks functions and fills arguments without a cloud API, running a tiny model on a server / phone / robot / Raspberry Pi, or needs a zero-dependency engine binary that serves function calls over HTTP. Also use it when a needle agent misbehaves (wrong tool, refused calls, repeated calls, "ungrounded" errors), when wiring OS CLIs or subprocesses as LLM-callable tools, or when fine-tuning or exporting a .cact archive.4metadata:5 source: https://skillmd.com/skills/drolu/needle6---78# Needle — on-device tool calling910Needle is a foundation model built for tiny devices: a single 121M-parameter11"Simple Attention Network" quantised to 2-bit, shipped as one ~35 MB weights12file (`needle3.cact`) plus an engine library under 1 MB. It runs offline on a13CPU — roughly 100 MB of RAM, hundreds of tokens per second on a laptop — with14no API key, no GPU and no network. It does three things:15161. **Tool calls** — given your function schemas and a plain-English request, it17 picks the right function and fills every argument from what was said. Ask18 for two things and you get two calls in order; ask for something no tool19 covers and you get an empty list, not a guess.202. **Structured extraction** — declare a shape, hand over messy text, get21 typed fields back; the decode grammar guarantees the output parses.223. **Text embeddings** — a vector for a sentence (Needle 3 only).2324Everything below was verified live against **cactus-needle 3.0.2** (Python25API, CLI, and the standalone engine on macOS arm64, with source-level checks26of the Windows/Linux paths).2728The model is tiny (121M params). It is reliable **when tools are designed for29it** — the rules in "Tool design" below are not optional polish, they are the30difference between a working agent and a flaky one.3132## Pick a surface3334| Your situation | Surface | Read |35|---|---|---|36| Python app/script; agent loop that **executes** the tools it calls | Python API — `import needle` | `references/python-api.md` |37| One-shot call generation from a terminal; deterministic output | `needle run` CLI (needs jax + a 242 MB checkpoint) | `references/cli.md` |38| No Python at runtime — servers, Windows services, edge devices, embedding in C | Standalone engine binary — `./needle --model needle3.cact` | `references/engine-binary.md` |39| The model picked the wrong tool / refused / looped / mangled args | Tool design rules (read this before debugging anything else) | `references/tool-design.md` |40| It errored or behaved oddly | Symptom table | `references/troubleshooting.md` |4142## Install4344```45pip install cactus-needle # Python 3.9+; macOS, Linux, Windows46```4748- On **Windows**, prefer `py -m pip install cactus-needle`. The engine ships49 as a prebuilt wheel (`libneedle3.dll`) — no compiler needed.50- First use auto-downloads the engine + `needle3.cact` weights into51 `~/.cache/cactus-needle/v3/<engine-version>/` (~36 MB; on Windows:52 `%USERPROFILE%\.cache\cactus-needle\v3\<version>\`). Set53 `NEEDLE3_LIB_PATH` to override the library location.54- If the machine must never touch the network, pre-seed that cache from55 another box, or use the standalone engine bundle (below) which needs no56 Python at runtime.57- Telemetry is on by default. Disable before importing:58 `NEEDLE_TELEMETRY=0` and `DO_NOT_TRACK=1`.59- **Multi-Python gotcha:** on machines with several Pythons (Homebrew vs60 python.org vs system), the `needle` CLI lives in the interpreter's bin dir61 that `pip install`ed it. If `import needle` fails under `python3`, find the62 right interpreter (`ls */bin/needle`, `pip show cactus-needle`) — or see63 the re-exec pattern in `references/troubleshooting.md`.6465## Quick start (Python API)6667```python68import needle6970@needle.tool71def get_weather(city: str) -> dict:72 """Get the current weather for a city."""73 return {"city": city, "temp_c": 27, "sky": "clear"}7475agent = needle.Needle(tools=[get_weather])76response = agent.run("what's it like in Lagos right now?")77print(response["results"]) # [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]78```7980The decorator reads the signature for argument types and the docstring for81the tool description (an `Args:` section documents parameters). `run()` picks82the tool, executes it **in-process**, feeds the return value back to the83model, and returns the final response. Full API — including `Field`84constraints, extraction and embeddings — in `references/python-api.md`.8586## Tool design — the rules that make a 121M model reliable8788These came from live failure modes, not style guides. Details and code89patterns in `references/tool-design.md`.90911. **Give every tool `triggers`.** `@needle.tool(triggers=["list databases",92 ...])`. Without them, tool selection is flaky — the model intermittently93 refuses a perfectly matching tool ("no connectivity or network tools94 available", often with high confidence).952. **Never make secrets tool arguments.** Needle's strict grounding blocks96 arguments it cannot verify against the input (`ungrounded password`), and97 secrets should not travel through an LLM anyway. Bake credentials into98 constants; let the model pick *what* to do, not recite keys.993. **Keep the result you return small.** `run()` feeds your tool's return100 value verbatim back to the model. A multi-kilobyte result makes a 121M101 model lose the thread and repeat unrelated calls until `max_steps`. Return102 a compact digest and keep the full payload in a variable the caller reads.1034. **Two asks per turn, maximum.** "Do X then Y" reliably yields two calls in104 order; a three-part compound drops the third. Split into follow-up105 questions.1065. **Judge success by `response["results"]`, not by the final turn.** After107 execution the final response has `function_calls: []` and108 `type: "respond"` — that is success, not refusal.1096. **Constrain arguments in the grammar, don't hope.** `Field`/`Annotated`110 with `ge/le`, `pattern`, `enum`, `const`, or `Literal` types make invalid111 values unrepresentable. The tiny model *will* otherwise fill `host='mysql'`112 from the word "MySQL" in your prompt.1137. **The Python engine injects a date fact** (`date: YYYY-MM-DD HH:MM`) into114 every prompt by default; the minutes drift between runs and can flip tool115 selection. For reproducibility pass a fixed `system=` and116 `auto_date=False`. The CLI and standalone engine don't inject dates.117118## Standalone engine (no Python at runtime)119120The raw-C engine binary runs the same `needle3.cact` weights with no JAX, no121Python — ideal for servers and Windows boxes:122123```bash124# macOS / Linux — download a platform bundle and place the weights beside it125python scripts/bootstrap_engine.py # auto-detects this platform126./macos-arm64/needle --model needle3.cact --tools tools.json \127 --prompt "give me revenue breakdown by country"128```129130```powershell131# Windows132python scripts\bootstrap_engine.py --platform windows-x86_64133.\windows-x86_64\needle.exe --model needle3.cact --tools tools.json --prompt "..."134```135136The binary is **call-selection only**: it emits the chosen call as JSON137(name, arguments, reasoning, confidence, throughput stats) and does **not**138execute anything — your code runs the tool. It is deterministic (greedy139decode), needs ~100 MB RAM, and can also serve HTTP (`--serve`, default port1408080: `POST /complete {"input": "..."}`, `POST /reset`). Platforms:141macos-arm64, linux-x86_64/arm64/armv7/riscv64/mipsel,142windows-x86_64/arm64, plus android, ios, tvos, watchos and wasm variants.143Full details in `references/engine-binary.md`.144145## Bundled scripts146147- `scripts/bootstrap_engine.py` — jax-free download of any platform's engine148 bundle + weights, prints the exact run command. Cross-platform.149- `scripts/export_tools.py` — dump the `@needle.tool` schemas from a Python150 module into `tools.json` for the engine binary or `needle run --tools`.151152## Troubleshooting quick table153154| Symptom | Cause → fix |155|---|---|156| `ModuleNotFoundError: needle` | Wrong interpreter → `references/troubleshooting.md` §1 |157| Result says `ungrounded password`/`ungrounded <arg>` | Grounding blocked a secret/fabricated value → §2 |158| Model fills nonsense (`host='mysql'`) | Unconstrained string arg → triggers + constraints → §3 |159| Same tool called repeatedly until max_steps | Tool result too large fed back → digest+stash → §4 |160| Third request of a compound ask never happens | 2-ask limit → split the ask → §5 |161| Refusal with high confidence, no call | Missing triggers / date-fact drift → §6 |162| `needle run`/`build` demand jax | Use the engine binary path instead → §7 |163| Garbled unicode in outputs | Charset on the wrapped CLI / corrupted source data → §8 |164| Nothing happens on macOS double-click | Window flashes closed — run from Terminal → §9 |165166Expanded causes and fixes: `references/troubleshooting.md`.