all publishers

tmuskal

@tmuskal source repo

15 published skills

  1. ▌
    Setup 2 · tmuskal
    Set up the LongMemEval benchmarking environment - guides user through conda env, upstream clone, dataset download, and initializes config
    3 repo stars
  2. ▌
    Report 2 · tmuskal
    Generate a scorecard from a completed LongMemEval run - computes overall accuracy + per-question-type accuracy and writes scorecard.json
    3 repo stars
  3. ▌
    Browse Tests 2 · tmuskal
    List and preview LongMemEval items - shows dataset stats, per-question-type counts, and a truncated preview of a specific item by question_id
    3 repo stars
  4. ▌
    Compare Runs 2 · tmuskal
    Compare two LongMemEval runs - diff scorecards, harness_config, and per-question-type accuracy
    3 repo stars
  5. ▌
    Cross Harness 2 · tmuskal
    Cross-harness benchmarking for LongMemEval - emit instruction packs for Codex/Gemini/OpenCode and ingest their results into a comparable scorecard
    3 repo stars
  6. ▌
    Run Benchmark 2 · tmuskal
    Execute a LongMemEval benchmark run - drives per-item hypothesis generation and LLM-as-judge scoring with checkpointing, resumable, capped at 500 items by default
    3 repo stars
  7. ▌
    Benchmark Adder · tmuskal
    Given a benchmark repository URL (agentic envs like arc-agi, memory/eval benchmarks like longmemeval, QA/code/tool-use benchmarks, etc.), orchestrate the creation of a full Claude Code plugin that benchmarks the current harness setup against it. Wraps the babysitter:babysit skill with the benchmark-plugin-creator process.
    3 repo stars
  8. ▌
    Setup · tmuskal
    Set up the ARC-AGI benchmarking environment - installs dependencies, configures API access, and verifies the setup works
    3 repo stars
  9. ▌
    Report · tmuskal
    Generate and display comprehensive reports from completed ARC-AGI benchmark runs - shows scores, per-game breakdowns, and performance analysis
    3 repo stars
  10. ▌
    Judge · tmuskal
    LLM-as-judge shim for LongMemEval - wraps upstream get_anscheck_prompt and calls Anthropic (default) or OpenAI (fallback) with exponential backoff
    3 repo stars
  11. ▌
    Resume · tmuskal
    Detect an incomplete LongMemEval run and continue it from the last checkpoint
    3 repo stars
  12. ▌
    Browse Tests · tmuskal
    Explore available ARC-AGI environments - lists games, shows details with ASCII grid visualization, and displays historical scores
    3 repo stars
  13. ▌
    Compare Runs · tmuskal
    Compare two or more ARC-AGI benchmark runs - shows score deltas, config changes, and trends to track improvement or regression
    3 repo stars
  14. ▌
    Cross Harness · tmuskal
    Cross-harness benchmarking - generate instructions for Codex/Gemini/OpenCode, import results, and compare across harnesses
    3 repo stars
  15. ▌
    Run Benchmark · tmuskal bundle
    Execute benchmark runs against ARC-AGI games - plays games with Claude Code as the agent and records scores
    3 repo stars