Test Afm Binary

Test AFM Binary

Test any pre-built afm binary with a menu of test suites. Validates the binary won't crash when relocated (pip/Homebrew install), then runs the selected tests.

Usage

• /test-afm-binary — interactive: asks for binary path, model, and test selection
• /test-afm-binary /path/to/afm — test the binary at the given path
• /test-afm-binary .build/arm64-apple-macosx/release/afm — test the current build

Instructions

Step 1: Resolve Binary Path

Ask for the binary path if not provided as an argument. Default: .build/arm64-apple-macosx/release/afm.

BIN="${1:-.build/arm64-apple-macosx/release/afm}"
[ -x "$BIN" ] || BIN=".build/release/afm"
BIN_ABS="$(cd "$(dirname "$BIN")" && pwd)/$(basename "$BIN")"
echo "Binary: $BIN_ABS"

If the binary doesn't exist or isn't executable, STOP and tell the user.

Step 2: Pre-Flight Safety Checks (MANDATORY — always run)

These checks run before any test suite. They catch fatal distribution bugs that would crash every pip/Homebrew user. If any check fails, STOP — do not proceed to testing.

Check A: Binary version

REPORTED=$($BIN_ABS --version 2>&1)
echo "Version: $REPORTED"

If the version shows only a base version without a SHA suffix (e.g., v0.9.8 instead of v0.9.8-62395ab), warn the user: this likely means the binary was built with an incremental swift build instead of ./Scripts/build-from-scratch.sh. The SHA injection only happens in the build script. This is a warning, not a blocker — the binary may still be valid for testing.

Check B: Metallib present

BIN_DIR="$(dirname "$BIN_ABS")"

# Check for metallib in either location (SPM bundle or loose file)
if [ -f "$BIN_DIR/AFMKit_AFMKitMLX.bundle/default.metallib" ]; then
  echo "PASS: Metallib in SPM bundle ($(du -h "$BIN_DIR/AFMKit_AFMKitMLX.bundle/default.metallib" | cut -f1))"
elif [ -f "$BIN_DIR/default.metallib" ]; then
  echo "PASS: Loose metallib ($(du -h "$BIN_DIR/default.metallib" | cut -f1))"
else
  echo "FAIL: No metallib found next to binary"
  echo "The binary will crash on first inference without default.metallib"
fi

Check C: Relocated binary does NOT crash

TMPDIR=$(mktemp -d)
cp "$BIN_ABS" "$TMPDIR/"

# Copy metallib as loose file (pip wheel layout)
if [ -f "$BIN_DIR/AFMKit_AFMKitMLX.bundle/default.metallib" ]; then
  cp "$BIN_DIR/AFMKit_AFMKitMLX.bundle/default.metallib" "$TMPDIR/"
elif [ -f "$BIN_DIR/default.metallib" ]; then
  cp "$BIN_DIR/default.metallib" "$TMPDIR/"
fi

MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache \
  "$TMPDIR/afm" mlx -m mlx-community/SmolLM3-3B-4bit -s "hello" --max-tokens 3 2>&1 | head -3
EXIT_CODE=${PIPESTATUS[0]}
rm -rf "$TMPDIR"

if [ "$EXIT_CODE" -ne 0 ]; then
  echo "FATAL: Relocated binary crashed (exit $EXIT_CODE)"
  echo "Bundle.module fatalError is still reachable — pip/Homebrew install will crash"
  echo "STOP. Fix MLXMetalLibrary.swift — it must NOT call Bundle.module"
else
  echo "PASS: Relocated binary runs without crash"
fi

If this fails, STOP IMMEDIATELY. Do not run any tests. The binary is broken for distribution.

Check D: No Bundle.module in source code

HITS=$(grep -r 'Bundle\.module' Sources/ --include='*.swift' | grep -v '^\s*//' | grep -v '// ' | wc -l | tr -d ' ')
if [ "$HITS" -gt 0 ]; then
  echo "FAIL: Found $HITS Bundle.module call(s) in source"
  grep -rn 'Bundle\.module' Sources/ --include='*.swift' | grep -v '//'
  echo "This WILL crash when installed via pip or Homebrew"
else
  echo "PASS: No Bundle.module calls in source"
fi

Check E: Info.plist embedded with privacy usage descriptions

macOS 26 SIGABRTs any process that requests privacy-sensitive APIs (Speech Recognition, microphone, camera, etc.) without a matching *UsageDescription key in the binary's embedded Info.plist. PR #107's Apple Speech feature triggers this on every afm speech / POST /v1/audio/transcriptions / chat input_audio call.

# Verify __TEXT,__info_plist section exists
if otool -l "$BIN_ABS" | grep -q '__info_plist'; then
  echo "PASS: __info_plist section present"
else
  echo "FAIL: Missing __TEXT,__info_plist section"
  echo "Check Package.swift linker flags and Sources/AFMCLI/Info.plist"
fi

# Verify NSSpeechRecognitionUsageDescription key is in the embedded plist
if strings "$BIN_ABS" | grep -q 'NSSpeechRecognitionUsageDescription'; then
  echo "PASS: NSSpeechRecognitionUsageDescription embedded"
else
  echo "FAIL: NSSpeechRecognitionUsageDescription missing"
  echo "afm speech / /v1/audio/transcriptions will SIGABRT on macOS 26"
fi

Note on testing Speech from an unattended context: If this skill is running inside Claude Code / an editor terminal / any parent process that does NOT have NSSpeechRecognitionUsageDescription, macOS 26 attributes the TCC subject to the parent and the child crashes even with a correct embedded plist. This is a test-environment artifact, not a binary bug. To verify Speech end-to-end, run afm speech -f <file.wav> from a fresh Terminal.app window (stock /System/Applications/Utilities/Terminal.app).

Present pre-flight results

If B, C, D, or E fail, STOP. Do not proceed.

Step 3: Select Model

Show available models and let the user pick:

MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache ./Scripts/list-models.sh

Use AskUserQuestion with the model list. Default recommendation: mlx-community/Qwen3.5-35B-A3B-4bit (19 GB, MoE, best coverage).

For quick smoke tests, suggest mlx-community/SmolLM3-3B-4bit (1.6 GB, fast).

Step 4: Select Tests

Use AskUserQuestion with multiSelect: true. Present these options:

Step 5: Run Selected Tests

For each selected test, set the correct environment and invoke. The binary path must be passed to every script.

Environment (always set):

export MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache

Parallelism rules:

• Unit tests (no server) → can run in parallel with anything
• Promptfoo (port 9999) → can run in parallel with assertions (port 9998)
• Batch validation (port 9999) → must NOT overlap with promptfoo
• Smart analysis (port 9877) → can run in parallel with assertions (port 9998)

Per-test invocation:

After each test completes, present its results immediately. Don't wait for all tests to finish before showing anything.

Step 6: Present Results Summary

After all selected tests complete, present a summary table:

Step 7: Open Promptfoo Web UI (if promptfoo tests were run)

After promptfoo evals complete, launch the interactive web interface:

promptfoo view -y &
# Opens browser at http://localhost:15500
# Shows all evaluations with interactive filtering, pass/fail drill-down, response comparison
# Results are persisted in ~/.promptfoo/promptfoo.db — all historical runs are visible
echo "Promptfoo UI running at http://localhost:15500 — press Ctrl+C to stop"

Leave the server running for the user to explore results. The web UI provides:

• Side-by-side comparison of outputs across server profiles (default vs adaptive-xml vs grammar)
• Drill-down into individual test failures with full request/response bodies
• Filtering by pass/fail status, test description, or provider
• Historical comparison with previous promptfoo runs

Step 8: Archive Results

TODAY=$(date +%Y-%m-%d)
ARCHIVE_DIR="test-reports/binary-test/$TODAY"
mkdir -p "$ARCHIVE_DIR"

# Copy all reports generated during this session
cp test-reports/assertions-report-*.html test-reports/assertions-report-*.jsonl "$ARCHIVE_DIR/" 2>/dev/null
cp test-reports/multi-assertions-report-*.html test-reports/multi-assertions-report-*.jsonl "$ARCHIVE_DIR/" 2>/dev/null
cp test-reports/smart-analysis-*.md "$ARCHIVE_DIR/" 2>/dev/null
cp test-reports/mlx-model-report-*.html test-reports/mlx-model-report-*.jsonl "$ARCHIVE_DIR/" 2>/dev/null

# Copy promptfoo results
PROMPTFOO_DIR="${AFM_PROMPTFOO_OUT_DIR:-/Volumes/edata/promptfoo/data/maclocal-api/current}"
cp "$PROMPTFOO_DIR"/*-mlx-community_*.json "$ARCHIVE_DIR/" 2>/dev/null

Write a SUMMARY.md in the archive directory with: binary path, version, model tested, platform, date, and a pass/fail table for every test suite run.

Interpreting Results

Server-Critical Suites (must be 100% pass — failures = server bug)

Model-Quality Suites (failures expected — not server bugs)

When to Investigate

• Any assertion failure in sections 0-8 → server bug, investigate immediately
• Relocated binary crash (Check C) → Bundle.module regression, fix before doing anything else
• All tool calls missing → wrong tool call format detection, check model_type in config.json
• NaN/garbage in long context → SDPA regression, check MLX version (must be pinned to 0.30.3)
• Streaming tool calls missing finish_reason → check MLXChatCompletionsController state machine

Quick Reference

# Smoke test the current build
/test-afm-binary .build/arm64-apple-macosx/release/afm

# Test a Homebrew-installed binary
/test-afm-binary $(brew --prefix afm-next)/bin/afm

# Test a pip-installed binary
/test-afm-binary $(python3 -c "import macafm_next; print(macafm_next.binary_path())")