How do I install the Locomo Benchmark skill?

Run npx skillmds add majiayu000/locomo-benchmark in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Locomo Benchmark skill do?

Run LoCoMo benchmark for long-term conversational memory It is listed under Coding & Dev Tools on SkillMD.

Is Locomo Benchmark safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: makes network calls. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Locomo Benchmark?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Locomo Benchmark free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Locomo Benchmark?

majiayu000 (@majiayu000) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Locomo Benchmark by majiayu000 — Agent Skill

Name: Locomo Benchmark
Rating: 4.2 (5 reviews)
Author: majiayu000

File contents

LoCoMo Benchmark

Evaluate cc-soul's memory against the LoCoMo benchmark (ACL 2024) for long-term conversational memory.

Quick Start

Run the benchmark script:

# Test one conversation (default: conv-26)
python3 $PLUGIN_DIR/scripts/locomo-benchmark.py

# Test specific conversations
python3 $PLUGIN_DIR/scripts/locomo-benchmark.py conv-26 conv-30

# Full benchmark (all 10 conversations)
python3 $PLUGIN_DIR/scripts/locomo-benchmark.py --full

# Limit QA pairs per conversation
python3 $PLUGIN_DIR/scripts/locomo-benchmark.py --max-qa 20

Where $PLUGIN_DIR is /maps/projects/fernandezguerra/apps/repos/cc-soul (or installed plugin path).

What the Script Does

Downloads LoCoMo data from GitHub to /tmp/locomo/ (if not present)
Ingests conversations into cc-soul memory:
- Extracts session summaries as observations
- Creates triplets for speaker facts
- Tags with sample_id for retrieval
Evaluates QA pairs:
- Retrieves context using chitta recall --tag {sample_id}
- Calculates F1 score vs ground truth
Reports results by category

Cat	Name	Description
1	Multi-hop	Requires connecting multiple facts
2	Single-hop	Direct fact retrieval
3	Temporal	Date/time questions
4	Open-domain	General knowledge
5	Adversarial	Should answer "no information"

Baseline Scores (from paper)

Model	F1
Human ceiling	87.9%
AutoMem	90.5%
GPT-4	32.1%
GPT-3.5	23.7%
Mistral-7B	13.9%

Data

Repository: https://github.com/snap-research/locomo
Local cache: /tmp/locomo/data/locomo10.json
10 conversations, ~200 QA pairs each, ~35 sessions per conversation

Manual Execution

If you prefer to run manually:

# Ensure data exists
git clone https://github.com/snap-research/locomo /tmp/locomo

# Run benchmark
python3 /maps/projects/fernandezguerra/apps/repos/cc-soul/scripts/locomo-benchmark.py conv-26

Expected Output

=== LoCoMo Benchmark Results ===

Total QA Pairs: 50
Overall F1: XX.X%

By Category:
  Multi-hop (n=XX): XX.X%
  Single-hop (n=XX): XX.X%
  Temporal (n=XX): XX.X%
  Open-domain (n=XX): XX.X%
  Adversarial (n=XX): XX.X%

Per Conversation:
  conv-26: XX.X% (50 QA)

Comparison (from paper):
  Human ceiling: 87.9%
  GPT-4 baseline: 32.1%
  cc-soul: XX.X%

---name: locomo-benchmarkdescription: Run LoCoMo benchmark for long-term conversational memory--- # LoCoMo Benchmark Evaluate cc-soul's memory against the [LoCoMo benchmark](https://github.com/snap-research/locomo) (ACL 2024) for long-term conversational memory. ## Quick Start Run the benchmark script: ```bash # Test one conversation (default: conv-26) python3 $PLUGIN_DIR/scripts/locomo-benchmark.py # Test specific conversations python3 $PLUGIN_DIR/scripts/locomo-benchmark.py conv-26 conv-30 # Full benchmark (all 10 conversations) python3 $PLUGIN_DIR/scripts/locomo-benchmark.py --full # Limit QA pairs per conversation python3 $PLUGIN_DIR/scripts/locomo-benchmark.py --max-qa 20 ``` Where `$PLUGIN_DIR` is `/maps/projects/fernandezguerra/apps/repos/cc-soul` (or installed plugin path). ## What the Script Does 1. **Downloads** LoCoMo data from GitHub to `/tmp/locomo/` (if not present) 2. **Ingests** conversations into cc-soul memory: - Extracts session summaries as observations - Creates triplets for speaker facts - Tags with sample_id for retrieval 3. **Evaluates** QA pairs: - Retrieves context using `chitta recall --tag {sample_id}` - Calculates F1 score vs ground truth 4. **Reports** results by category ## Categories | Cat | Name | Description | |-----|------|-------------| | 1 | Multi-hop | Requires connecting multiple facts | | 2 | Single-hop | Direct fact retrieval | | 3 | Temporal | Date/time questions | | 4 | Open-domain | General knowledge | | 5 | Adversarial | Should answer "no information" | ## Baseline Scores (from paper) | Model | F1 | |-------|-----| | Human ceiling | 87.9% | | AutoMem | 90.5% | | GPT-4 | 32.1% | | GPT-3.5 | 23.7% | | Mistral-7B | 13.9% | ## Data - Repository: `https://github.com/snap-research/locomo` - Local cache: `/tmp/locomo/data/locomo10.json` - 10 conversations, ~200 QA pairs each, ~35 sessions per conversation ## Manual Execution If you prefer to run manually: ```bash # Ensure data exists git clone https://github.com/snap-research/locomo /tmp/locomo # Run benchmark python3 /maps/projects/fernandezguerra/apps/repos/cc-soul/scripts/locomo-benchmark.py conv-26 ``` ## Expected Output ``` === LoCoMo Benchmark Results === Total QA Pairs: 50 Overall F1: XX.X% By Category: Multi-hop (n=XX): XX.X% Single-hop (n=XX): XX.X% Temporal (n=XX): XX.X% Open-domain (n=XX): XX.X% Adversarial (n=XX): XX.X% Per Conversation: conv-26: XX.X% (50 QA) Comparison (from paper): Human ceiling: 87.9% GPT-4 baseline: 32.1% cc-soul: XX.X% ```

Frequently asked questions