Test Macafm

test-macafm

Run the maclocal-api test suite: automated pass/fail assertions and smart analysis (the smart suite's AI judge is opt-in — default off; ask the user before enabling it).

Triggers

Use this skill when the user asks to:

• Test or validate the server (e.g., "run the tests", "test AFM", "validate the build")
• Regression check after code changes
• Onboard a new model (verify it works correctly with the server)
• Release check before tagging or pushing
• Benchmark or profile model performance

First Questions to Ask

1. Model — Which model to test? (Ask if not specified. Default: whatever's loaded.)
2. Tier — smoke / standard / full? (Suggest based on context.)
3. Binary path — Default .build/release/afm. Ask if user has a custom build location.
4. Port — Default 9998. Ask if user's server is on a different port.
5. Server running? — Is the server already running, or should tests start it?

Tier Decision Tree

Quick guide:

• "Just run a quick test" → smoke
• "Test before merging" → standard
• "Full release validation" or "onboard new model" → full
• User doesn't specify → suggest standard

Execution Workflow

1. Build Check

# Ensure release build is current
swift build -c release

2. Start Server (if not running)

MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache \
  .build/release/afm mlx -m MODEL --port 9998 \
  --tool-call-parser afm_adaptive_xml \
  --enable-prefix-caching \
  --enable-grammar-constraints &
# Wait for server to be ready
until curl -sf http://127.0.0.1:9998/v1/models >/dev/null 2>&1; do sleep 1; done

Recommended flags for testing:

• --tool-call-parser afm_adaptive_xml — best tool call parser with JSON-in-XML fallback
• --enable-prefix-caching — 67-79% prompt token savings on repeated requests
• --enable-grammar-constraints — EBNF constrained decoding forces valid XML tool calls, improving success from 60% to 100% on realistic workloads

3. Run Automated Assertions

./Scripts/test-assertions.sh --tier TIER --model MODEL --port 9998

Interpret results immediately. If any FAIL, investigate before proceeding.

4. Run Smart Analysis (full tier only)

The smart analysis harness manages its own server (port 9877) — do NOT pass --port. It uses test-llm-comprehensive.txt which has an [all] baseline prompt and [@ label] template sections.

The AI judge is OPT-IN — default OFF. By default, run the smart suite WITHOUT an AI judge: it executes every prompt and records the model's raw outputs to the report for manual review, with no claude/codex scoring. Before running, ask the user (e.g. via AskUserQuestion) whether to enable the AI judge — it adds latency/cost and invokes an external CLI:

"Run the smart suite with an AI judge (claude) scoring each response, or without it (just record outputs for manual review)? Default: without."

Default — no AI judge (records outputs only; omit --smart entirely):

AFM_BIN=.build/release/afm ./Scripts/mlx-model-test.sh \
  --model MODEL \
  --prompts Scripts/test-llm-comprehensive.txt

Only if the user opts in — append --smart 1:claude (the --smart flag accepts a batch mode prefix and tool list):

AFM_BIN=.build/release/afm ./Scripts/mlx-model-test.sh \
  --model MODEL \
  --prompts Scripts/test-llm-comprehensive.txt \
  --smart 1:claude

Smart analysis options (only when the AI judge is enabled):

• --smart claude or --smart codex — batch mode 0 (one big swoop, may fail on large test suites)
• --smart 1:claude or --smart 1:codex — batch mode 1 (test-by-test, more reliable)
• --smart 1:claude,codex — run multiple AI judges
• --tests 1,5,10 — run only specific test numbers (1-indexed)

Note: The [all] prompt runs for every test variant. With high max_tokens (e.g., 32768 on code tests), thinking models may generate very long reasoning for the baseline prompt. Total run time for full suite: ~45-90 min depending on model speed.

5. Run GPU Shader Profile (full tier, or when investigating perf)

Generates an interactive HTML report with measured DRAM bandwidth, GPU utilization/power timelines, and per-kernel Metal shader names from xctrace Shader Timeline.

One-time setup (creates custom Instruments template with Shader Timeline enabled):

python3 Scripts/create-shader-template.py

Run the profile (no server needed — uses single-prompt mode):

python3 Scripts/gpu-profile-report.py MODEL [max_tokens] [prompt]
# Default: 4096 tokens, built-in GPU analysis prompt
# Example: python3 Scripts/gpu-profile-report.py mlx-community/Qwen3.5-35B-A3B-4bit

This does everything automatically:

1. Warms up mactop (bandwidth monitor, no sudo)
2. Runs inference with --gpu-profile --gpu-trace 15
3. Collects 300ms bandwidth/GPU/power samples via PTY during inference
4. Extracts shader kernel names from the xctrace trace
5. Generates /tmp/afm-gpu-profile.html and opens in browser

Or use individual flags on any AFM invocation:

afm mlx -m MODEL --gpu-profile -s "prompt"           # Zero-overhead stats
afm mlx -m MODEL --gpu-profile-bw -s "prompt"        # + mactop bandwidth (~5s)
afm mlx -m MODEL --gpu-trace 10 -s "prompt"          # xctrace shader trace

Live bandwidth monitor (run in separate terminal during server requests):

./Scripts/gpu-profile.sh bandwidth

What the report shows:

• Device info (chip, memory, architecture)
• Prefill/decode tok/s with exact timing
• Memory breakdown (model weights vs KV cache)
• DRAM bandwidth timeline chart (measured via mactop)
• GPU utilization & power timeline chart
• Per-kernel Metal shader names (from Shader Timeline)
• Exact command line for reproducibility

What to look for:

• GPU utilization <100% during decode → CPU-GPU pipeline bubbles
• Bandwidth utilization >80% → memory-bound, kernel optimization won't help
• Bandwidth utilization <20% with MoE model → normal (only active experts read)
• Key kernels: affine_qmv_fast (decode bottleneck), steel_gemm_fused (prefill), sdpa_vector (attention)

5b. API-Based GPU Profiling (per-request, no CLI flags needed)

Clients can request GPU profiling data via the X-AFM-Profile HTTP header. No server flags required — works on any running AFM server.

Two levels:

# Summary: GPU power, memory, bandwidth, tok/s
curl http://127.0.0.1:9999/v1/chat/completions \
  -H "X-AFM-Profile: true" \
  -d '{"model":"m","messages":[{"role":"user","content":"Hi"}]}'

# Extended: summary + 300ms time-series samples (for charts/dashboards)
curl http://127.0.0.1:9999/v1/chat/completions \
  -H "X-AFM-Profile: extended" \
  -d '{"model":"m","messages":[{"role":"user","content":"Hi"}]}'

Response fields (afm_profile):

• gpu_power_avg_w / gpu_power_peak_w — GPU power via native IOReport (no mactop)
• memory_weights_gib / memory_kv_gib / memory_peak_gib — memory breakdown in GiB
• prefill_tok_s / decode_tok_s — throughput
• est_bandwidth_gbs — DRAM bandwidth from IOReport power (calibrated at startup via MLX GPU stress)
• chip / theoretical_bw_gbs — hardware context
• gpu_samples — number of 300ms readings taken

Extended adds (afm_profile_extended):

• summary — same as afm_profile
• samples[] — per-300ms readings: {t, bw_gbs, gpu_pct, gpu_power_w, dram_power_w}

How it works internally:

• IOReport Energy Model + GPU Stats channels sampled every 300ms via DispatchSource timer
• DRAM bandwidth derived from DRAM power using chip-specific calibration constant
• Calibration runs once at startup: 1 GiB MLX GPU stress test (~2s, async, non-blocking)
• Per-request isolation: concurrent profiled requests are guarded (second request skips gracefully)
• Zero overhead when header not sent (one string lookup per request)
• Works for both streaming (SSE event before [DONE]) and non-streaming

What to look for:

• gpu_power_peak_w ~28W during decode on M3 Ultra (matches mactop)
• est_bandwidth_gbs ~170-180 GB/s for Qwen3.5-35B-A3B-4bit (21% of 800 GB/s theoretical)
• Short requests (<300ms): at least 1 sample (timer first-fires at 100ms)
• afm_profile absent from response when header not sent (no null pollution)

6. Run Promptfoo Agentic Evals (full tier, or when validating tool calling / structured output)

The promptfoo agentic eval suite tests AFM's tool-calling and structured-output across multiple server configurations and real-world agent framework schemas. It manages its own server lifecycle.

Prerequisites: promptfoo CLI must be installed (npm install -g promptfoo).

Run the full suite:

AFM_MODEL=MODEL \
AFM_BINARY=.build/arm64-apple-macosx/release/afm \
MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache \
./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh all

Run individual suites:

# Just structured output tests
./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh structured

# Just tool calling (all 3 parser profiles)
./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh toolcall

# Just grammar constraint validation (8 server phases)
./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh grammar-constraints

# Just one agent framework
./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh opencode

Available modes: all, structured, structured-stress, toolcall, toolcall-quality, grammar-constraints, agentic, frameworks, opencode, pi, openclaw, hermes, default, adaptive-xml, adaptive-xml-grammar

Suite Coverage (~137 test cases across 16 configs)

Server Profiles (managed automatically by the script)

Custom Provider & Judges

• providers/afm_provider.mjs — Custom promptfoo provider with two transports: api (OpenAI-compatible HTTP) and cli-guided-json (direct binary invocation). Supports extract modes: content, tool_calls, normalized_message, full_response. Captures responseHeaders for grammar header assertions.
• judges/assert-grammar-header.mjs — Validates X-Grammar-Constraints response header: expects "downgraded" when grammar not available, absent when grammar active.
• judges/classify-failures.mjs — Post-run AI-based failure classifier: categorizes each failure as afm_bug (server/protocol), model_quality (wrong tool/args), or harness_bug (false negative).

Environment Variables

Output

JSON reports per suite+profile in $AFM_PROMPTFOO_OUT_DIR:

• structured-MODEL_SLUG.json
• toolcall-{default,adaptive-xml,adaptive-xml-grammar}-MODEL_SLUG.json
• grammar-{schema,tools}-{no-grammar,grammar-enabled,adaptive-xml,concurrent,prefix-cache}-MODEL_SLUG.json
• {agentic,frameworks,opencode,pi,openclaw,hermes}-{default,adaptive-xml,adaptive-xml-grammar}-MODEL_SLUG.json

7. Review Reports

• Assertion report: test-reports/assertions-report-*.html
• Smart analysis: test-reports/smart-analysis-{tool}-*.md
• HTML report: test-reports/mlx-model-report-*.html
• GPU profile: /tmp/afm-gpu-profile.html (+ /tmp/afm-metal.trace for Instruments)
• JSONL data: test-reports/assertions-report-*.jsonl, test-reports/mlx-model-report-*.jsonl
• Promptfoo evals: $AFM_PROMPTFOO_OUT_DIR/{suite}-{profile}-MODEL_SLUG.json (default: /Volumes/edata/promptfoo/data/maclocal-api/current/)

7. Stop Server (if we started it)

kill %1  # or whatever the background job is

Interpreting Results

Assertion Test Failures

Smart Analysis False Positives

Known patterns where AI judges score incorrectly (see references/interpreting-scores.md):

• Stop sequences truncating output scored as "low quality" — truncation IS the expected behavior
• Empty content when stop fires on first visible token — correct behavior
• JSON mode not constraining thinking models — prompt injection, not grammar-constrained
• "Missing reasoning" when model doesn't support <think> — correct, not a bug
• Thinking model consuming entire max_tokens budget on reasoning with empty visible content — model behavior, not a server bug
• [all] baseline prompt scored low when it runs with a code/math test's high max_tokens and system prompt — irrelevant context for the baseline prompt

Promptfoo Eval Failures

Key rule: structured, toolcall, grammar-* (non-concurrent), frameworks suites should be 100% pass. Any failure there is a server bug. Everything else has model-quality variance.

Post-run failure classification (optional): Run judges/classify-failures.mjs on any result JSON to get AI-based afm_bug vs model_quality vs harness_bug classification.

When to Escalate

• SDPA regression: NaN or garbage in long-context tests → check MLX version, see MEMORY.md
• Tool call format mismatch: Unknown format → check ToolCallFormat.infer() and model's config.json
• Build failure: Vendor patch conflict → run Scripts/apply-mlx-patches.sh --check

Concurrency Benchmark

Full-harness concurrency sweep that starts the server, runs warmup, tests all concurrency levels, collects GPU metrics via mactop, saves JSON results, and generates a comparison chart.

Script

Scripts/benchmarks/benchmark_afm_vs_mlxlm.py

Usage

# AFM-only concurrency sweep (recommended for quick benchmarks)
python3 Scripts/benchmarks/benchmark_afm_vs_mlxlm.py --afm-only

# Full AFM vs mlx-lm comparison (both servers, fair A/B)
python3 Scripts/benchmarks/benchmark_afm_vs_mlxlm.py

# Re-generate graph from existing results
python3 Scripts/benchmarks/benchmark_afm_vs_mlxlm.py --graph
python3 Scripts/benchmarks/benchmark_afm_vs_mlxlm.py --graph Scripts/benchmark-results/FILE.json

What it does

1. Detects hardware (chip, memory)
2. Starts server(s) with --concurrent N
3. 60s GPU settle + multi-round warmup (JIT kernel compilation)
4. Sweeps concurrency levels: [1, 2, 4, 8, 12, 16, 20, 24, 32, 40, 50]
5. At each level: fires N simultaneous streaming 4096-token requests, measures aggregate tok/s, per-request tok/s, GPU power/temp/usage via mactop
6. Saves JSON to Scripts/benchmark-results/concurrency-benchmark-TIMESTAMP.json
7. Generates PNG chart to Scripts/benchmark-results/concurrency-benchmark-TIMESTAMP.png

Configuration (top of script)

Reference results (March 18, v0.9.7, M3 Ultra 512GB, --concurrent 28)

  B   Agg t/s   Per-req   Wall    GPU%   GPU W
  1     118.7     118.7   34.5s    94%   28.5W
  2     193.9      97.0   42.2s    93%   41.6W
  4     298.4      74.6   54.9s    97%   62.7W
  8     407.3      50.9   80.5s    96%   75.5W
 12     493.4      41.1   99.6s    98%   83.4W
 16     573.9      35.9  114.2s    99%   88.2W
 20     581.6      29.1  140.8s    98%   79.1W
 24     629.6      27.4  149.6s    99%   83.2W

Additional batch validation scripts

Key File Reference

Validation Checklist

Smoke Tier

• Server reachable, model loaded
• Basic completion returns content
• Stop sequences work (absent from output, correct finish_reason)
• Logprobs schema valid
• Think extraction works (if model supports it)
• Basic tool call works
• Error handling (empty messages, malformed JSON)

Standard Tier (adds)

• All smoke checks
• Streaming stop sequence parity
• Streaming logprobs
• Prompt cache: cached_tokens=0 first, >0 second
• Concurrent requests (2 and 3 simultaneous)
• Multi-tool calls
• Additional stop edge cases
• chat_template_kwargs: enable_thinking=false disables thinking (if model supports it)
• chat_template_kwargs: streaming parity
• chat_template_kwargs: default behavior unaffected
• OpenAI-compat evals: test-openai-compat-evals.py (non-stream, stream, logprobs, usage chunk)
• Guided JSON evals: test-guided-json-evals.py (API schema, streaming schema, SDK parse)

Full Tier (adds)

• All standard checks
• Performance: TTFT < 5s, tok/s > 1
• Long context (2K, 4K tokens) no crash/NaN
• Smart analysis: test-llm-comprehensive.txt (AI judge opt-in — default off, ask the user; enable with --smart 1:claude / --smart 1:codex)
• Streaming parity (assembled content matches non-streaming)
• Cache timing improvement visible
• Batch correctness: validate_responses.py at B={1,2,4,8}
• Batch mixed workload: validate_mixed_workload.py (short+long decode, GPU metrics)
• Batch prefix cache: validate_multiturn_prefix.py (multi-turn conversations under concurrency)
• GPU shader profile: gpu-profile-report.py (bandwidth, power, kernel names, HTML report)
• API profile: X-AFM-Profile: true returns afm_profile with GPU power + bandwidth
• API profile: X-AFM-Profile: extended returns afm_profile_extended with samples array
• API profile: no header → no afm_profile fields in response (no null pollution)
• API profile: streaming → profile SSE event before [DONE]
• API profile: concurrent profiled requests → second skips gracefully
• Promptfoo structured: 100% pass (json_schema + stress)
• Promptfoo toolcall: 100% pass (all 3 profiles)
• Promptfoo grammar-constraints (non-concurrent): 100% pass
• Promptfoo frameworks: 100% pass (all 3 profiles)
• Promptfoo opencode/pi/openclaw/hermes: >70% pass (model quality variance expected)
• Promptfoo grammar-header: downgrade/enforce headers correct