How do I install the Benchmark skill?

Run npx skillmds@latest add thinkfleetai/benchmark in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Benchmark safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Benchmark?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Benchmark free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Benchmark?

thinkfleetai (@thinkfleetai) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Benchmark by thinkfleetai — Agent Skill

Name: Benchmark
Author: thinkfleetai

Benchmark

Run MemMesh's competitive benchmark harness (LOCOMO / BEAM) to compare retrieval quality, tokens, latency, and cost against Mem0, Zep, full-context, and naive-RAG baselines. Use when the user wants proof MemMesh is better, is evaluating a migration, or asks "how does this compare to mem0".

thinkfleetai 4101aed 1.8 KB Updated Sep 17, 2026

File contents

benchmark

Put numbers on the comparison. MemMesh ships a real benchmark harness that runs the public LOCOMO dataset end-to-end against competing systems.

What it compares

Systems: thinkfleet (MemMesh) vs full_context vs naive_rag, and — with keys — Mem0 / Zep. Metrics: answer accuracy (rubric-scored), tokens consumed, latency, and cost per conversation.

Run it

The harness lives in the engine repo at crates/eval/competitive/:

cd crates/eval/competitive
python bench.py --systems thinkfleet,mem0,full_context --dataset locomo
# results land in results/

(Set the competitors' API keys via env for a head-to-head; without them you still get MemMesh vs full-context vs naive-RAG.)

Report honestly

MemMesh's positioning is calibration over raw accuracy — "80% means 80%" and honest abstention beat a slightly higher accuracy with overconfident wrong answers. So report the full picture:

accuracy and calibration error,
tokens / latency / cost (MemMesh's retrieval is far cheaper than full-context),
where MemMesh abstained vs. where a competitor answered confidently and wrong.

Don't cherry-pick a single accuracy number. If a competitor wins on one axis, say so, and show where MemMesh's calibration/cost advantage pays off.

Cost gating

Prove the win on the cheap tiers (LOCOMO, BEAM-100K) before spending on BEAM-1M/10M — a single 10M-token conversation is expensive. Escalate tiers only once the cheaper tier shows a clear, defensible lead.

Frequently asked questions