Run Magicore Evaluation

Run MagiCore Evaluation

Use this workflow to run the deterministic capability suite, retrieval-only scenarios, live model-judged scenarios, or the dedicated event-time comparison. Work from the repository root on Windows PowerShell.

Guardrails

• Read evaluation/README.md, evaluation/MagiCore.Evaluation/Scenarios.cs, and the selected dataset before interpreting a run.
• Never display, commit, or store credentials from evalconfig.local.yaml.
• Validate a custom dataset before spending model tokens.
• Start with the smallest scenario set that can falsify the hypothesis.
• Do not overwrite evaluation-latest.json or publish a timestamped report unless the user explicitly asks. Historical live reports are not regenerated by self-test runs.
• Treat skipped provider checks as unverified, not passed. Treat the bundled datasets as controlled synthetic evidence, not production validation.
• Keep model, embedding model, judge, dataset, topK, and scenario settings fixed when comparing runs.

Choose a Run

1. Inspect available scenarios

dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --list

Use an explicit --scenario list for focused work. The opt-in event-time scenario is listed but excluded from the default matrix to preserve comparability with published results.

2. Fast deterministic regression

Run public capability checks plus the default retrieval-only scenario without credentials:

dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --self-test

Require zero failed capabilities. Provider-dependent checks may be skipped with documented reasons.

3. Event-time retrieval comparison

Validate the dedicated cross-year dataset first:

dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --dataset .\evaluation\MagiCore.Evaluation\evaldataset.temporal.json --validate-dataset

Then compare the same data with and without temporal indexing and filtering:

dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --self-test --dataset .\evaluation\MagiCore.Evaluation\evaldataset.temporal.json --scenario baseline,event-time

The event-time scenario must:

• Parse each session date into MemoryAddOptions.ReferenceTime.
• Enable MemorySearchOptions.EnableTemporalSearch.
• Exclude undated memories after a confident explicit date or year match.
• Fail open for non-temporal questions.

Compare RetrievalHitRate and MeanRetrievedCount together. A positive deterministic result preserves evidence hits while lowering mean retrieved candidates. Inspect per-question RetrievedCount and RetrievedMemories in the JSON report, especially:

• atlas-2024-timeline and atlas-2025-timeline for year isolation.
• atlas-exact-date for day-level filtering.
• atlas-region for unchanged fail-open retrieval.

Do not claim answer-quality improvement from self-test mode because it does not generate or judge answers.

4. Focused live model run

Create local configuration only when it does not already exist:

Copy-Item .\evaluation\MagiCore.Evaluation\evalconfig.example.yaml .\evaluation\MagiCore.Evaluation\evalconfig.local.yaml

The user must place the API key directly in that ignored file. Never request or echo it through chat.

Run a focused scenario pair:

dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --scenario baseline,conflict-resolution

Run the model-judged event-time comparison:

dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --dataset .\evaluation\MagiCore.Evaluation\evaldataset.temporal.json --scenario baseline,event-time

Run the default longitudinal matrix only when broad evidence and its model cost are justified:

dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release

5. Custom dataset

Use the bundled JSON files as schemas. Validate before running:

dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --dataset .\path\to\dataset.json --validate-dataset
dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --dataset .\path\to\dataset.json --scenario baseline

For event-time evaluation, every session date must use yyyy-MM-dd. Questions should include explicit ISO dates or years when strict temporal filtering is expected, plus non-temporal controls to test fail-open behavior.

Interpret the Report

The command prints the Markdown report path and writes matching timestamped .md and .json files to the configured results directory.

Review in this order:

1. Capability failures. Stop and investigate any failure before comparing quality metrics.
2. Scenario errors and unexpected skips.
3. Accuracy and category accuracy for live model runs.
4. Retrieval hit rate and its numerator/denominator.
5. Mean retrieved candidates, especially for temporal selectivity.
6. Contradiction and adversarial regressions.
7. F1 and BLEU-1 as supporting lexical metrics, not substitutes for judgment.
8. Search latency, ingest time, memory count, and model/runtime settings.
9. Per-question retrieved memories and judge reasoning for every regression.

Wilson intervals describe question-sampling uncertainty only. They do not measure model, prompt, provider, or run-to-run variance. Repeat live runs before interpreting small differences.

Validate Code Changes

After changing the harness or memory behavior, run:

dotnet test .\tests\MagiCore.Tests\MagiCore.Tests.csproj --configuration Release --no-restore
dotnet test .\tests\MagiCore.NetStandard.Tests\MagiCore.NetStandard.Tests.csproj --configuration Release --no-restore
dotnet build .\MagiCore.slnx --configuration Release --no-restore

Report exact pass/fail/skip counts, selected scenarios, dataset, mode, models, key metric deltas, generated report path, and existing unrelated warnings. Remove incidental timestamped reports after validation unless the user asked to retain them.