← all publishers

tmuskal

@tmuskal source repo

Report this profile Claim this profile

• 15 skills
• 0 followers
• 8 hours ago last updated

1. ▌
   Setup 2 · tmuskal

   Set up the LongMemEval benchmarking environment - guides user through conda env, upstream clone, dataset download, and initializes config
   0
   installs
2. ▌
   Report 2 · tmuskal

   Generate a scorecard from a completed LongMemEval run - computes overall accuracy + per-question-type accuracy and writes scorecard.json
   0
   installs
3. ▌
   Browse Tests 2 · tmuskal

   List and preview LongMemEval items - shows dataset stats, per-question-type counts, and a truncated preview of a specific item by question_id
   0
   installs
4. ▌
   Compare Runs 2 · tmuskal

   Compare two LongMemEval runs - diff scorecards, harness_config, and per-question-type accuracy
   0
   installs
5. ▌
   Cross Harness 2 · tmuskal

   Cross-harness benchmarking for LongMemEval - emit instruction packs for Codex/Gemini/OpenCode and ingest their results into a comparable scorecard
   0
   installs
6. ▌
   Run Benchmark 2 · tmuskal

   Execute a LongMemEval benchmark run - drives per-item hypothesis generation and LLM-as-judge scoring with checkpointing, resumable, capped at 500 items by default
   0
   installs
7. ▌
   Benchmark Adder · tmuskal

   Given a benchmark repository URL (agentic envs like arc-agi, memory/eval benchmarks like longmemeval, QA/code/tool-use benchmarks, etc.), orchestrate the creation of a full Claude Code plugin that benchmarks the current harness setup against it. Wraps the babysitter:babysit skill with the benchmark-plugin-creator process.
   0
   installs
8. ▌
   Setup · tmuskal

   Set up the ARC-AGI benchmarking environment - installs dependencies, configures API access, and verifies the setup works
   0
   installs
9. ▌
   Report · tmuskal

   Generate and display comprehensive reports from completed ARC-AGI benchmark runs - shows scores, per-game breakdowns, and performance analysis
   0
   installs
10. ▌
    Judge · tmuskal

    LLM-as-judge shim for LongMemEval - wraps upstream get_anscheck_prompt and calls Anthropic (default) or OpenAI (fallback) with exponential backoff
    0
    installs
11. ▌
    Resume · tmuskal

    Detect an incomplete LongMemEval run and continue it from the last checkpoint
    0
    installs
12. ▌
    Browse Tests · tmuskal

    Explore available ARC-AGI environments - lists games, shows details with ASCII grid visualization, and displays historical scores
    0
    installs
13. ▌
    Compare Runs · tmuskal

    Compare two or more ARC-AGI benchmark runs - shows score deltas, config changes, and trends to track improvement or regression
    0
    installs
14. ▌
    Cross Harness · tmuskal

    Cross-harness benchmarking - generate instructions for Codex/Gemini/OpenCode, import results, and compare across harnesses
    0
    installs
15. ▌
    Run Benchmark · tmuskal bundle

    Execute benchmark runs against ARC-AGI games - plays games with Claude Code as the agent and records scores
    0
    installs