Omnimemeval Eval

OmniMemEval evaluation

Read this before the first run, after forgetting the workflow, or when resuming a run. The stable source of truth is evals/omnimemeval/benchmark.config.json; do not reconstruct its parameters on the command line.

Run

For an embedding-backed run, start the single local service in a separate terminal with the already prepared interpreter, then verify that /health reports "device":"cuda":

.benchmarks/bge-venv/Scripts/python.exe evals/omnimemeval/bge_server.py --device cuda

Do not use uv run --with for this service on Windows: it can resolve a second, CPU-only PyTorch environment. Omit --device only when automatic GPU-or-CPU selection is intentional; use --device cpu for an explicit CPU run.

# Inspect without model or embedding work.
npm run benchmark:omni -- longmemeval --dry-run

# Run one complete official suite.
npm run benchmark:omni -- longmemeval

Supported suite names are longmemeval, locomo, beam, personamem-v2, and halumem. The runner generates a unique version, loads the configured env file, establishes the NMG/UTF-8/venv environment, and delegates to the pinned official script.

Configure once

Edit the checked-in config when the experiment policy changes:

{
  "envFile": ".env.nmg-opencode",
  "commonArgs": ["--workers", "16", "--llm-workers", "16", "--top-k", "20"],
  "suites": {
    "longmemeval": [],
    "beam": ["--scale", "100k"]
  }
}

commonArgs apply to every suite. suites.<name> forwards only that suite's official options. Keep provider keys, model names, embedding endpoints, and QPP configuration in the env file. Do not put runner-owned --lib, --env, --version, or --replay flags in the config.

For a one-off canary, copy the config and select it explicitly:

npm run benchmark:omni -- beam --config evals/omnimemeval/canary.config.json

Do not add a CLI flag merely to avoid editing or copying the config.

Resume

Point at the exact existing result directory:

npm run benchmark:omni -- --resume `
  .benchmarks/official/OmniMemEval/results/lme/nmg-<version>

The runner infers the suite and version from experiment_config.sh, verifies that the run belongs to NMG, and rejects configured parameter drift. It removes destructive lifecycle flags before delegating; the official pipeline resumes its checkpoints. Do not infer a target from a similar directory name.

Boundaries

• Do not call the five official scripts directly for normal NMG runs.
• Do not create a suite-specific NMG wrapper; add official arguments to config.
• Keep official parsing, prompts, scoring, checkpoints, and reports upstream.
• Keep NMG evidence audits separate from the official run.
• The runner does not start or kill embedding services, Pi, daemons, or Python processes.
• Use the official result artifacts to diagnose a failure; do not tune product retrieval solely to repair orchestration.

Historical scores and incident narratives belong in docs/experiments/, not in this operating manual.