# Run Magicore Evaluation

> Run, validate, compare, and interpret the MagiCore evaluation harness. Use for self-tests, live model evaluation, scenario comparisons, event-time retrieval evaluation, custom datasets, capability checks, benchmark reports, and regression analysis.

- Skill: `jihadkhawaja/run-magicore-evaluation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add jihadkhawaja/run-magicore-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jihadkhawaja/run-magicore-evaluation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: jihadkhawaja (https://skillmd.com/u/jihadkhawaja)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jihadkhawaja/run-magicore-evaluation

---


# Run MagiCore Evaluation

Use this workflow to run the deterministic capability suite, retrieval-only scenarios, live model-judged scenarios, or the dedicated event-time comparison. Work from the repository root on Windows PowerShell.

## Guardrails

- Read `evaluation/README.md`, `evaluation/MagiCore.Evaluation/Scenarios.cs`, and the selected dataset before interpreting a run.
- Never display, commit, or store credentials from `evalconfig.local.yaml`.
- Validate a custom dataset before spending model tokens.
- Start with the smallest scenario set that can falsify the hypothesis.
- Do not overwrite `evaluation-latest.json` or publish a timestamped report unless the user explicitly asks. Historical live reports are not regenerated by self-test runs.
- Treat skipped provider checks as unverified, not passed. Treat the bundled datasets as controlled synthetic evidence, not production validation.
- Keep model, embedding model, judge, dataset, `topK`, and scenario settings fixed when comparing runs.

## Choose a Run

### 1. Inspect available scenarios

```powershell
dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --list
```

Use an explicit `--scenario` list for focused work. The opt-in `event-time` scenario is listed but excluded from the default matrix to preserve comparability with published results.

### 2. Fast deterministic regression

Run public capability checks plus the default retrieval-only scenario without credentials:

```powershell
dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --self-test
```

Require zero failed capabilities. Provider-dependent checks may be skipped with documented reasons.

### 3. Event-time retrieval comparison

Validate the dedicated cross-year dataset first:

```powershell
dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --dataset .\evaluation\MagiCore.Evaluation\evaldataset.temporal.json --validate-dataset
```

Then compare the same data with and without temporal indexing and filtering:

```powershell
dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --self-test --dataset .\evaluation\MagiCore.Evaluation\evaldataset.temporal.json --scenario baseline,event-time
```

The event-time scenario must:

- Parse each session date into `MemoryAddOptions.ReferenceTime`.
- Enable `MemorySearchOptions.EnableTemporalSearch`.
- Exclude undated memories after a confident explicit date or year match.
- Fail open for non-temporal questions.

Compare `RetrievalHitRate` and `MeanRetrievedCount` together. A positive deterministic result preserves evidence hits while lowering mean retrieved candidates. Inspect per-question `RetrievedCount` and `RetrievedMemories` in the JSON report, especially:

- `atlas-2024-timeline` and `atlas-2025-timeline` for year isolation.
- `atlas-exact-date` for day-level filtering.
- `atlas-region` for unchanged fail-open retrieval.

Do not claim answer-quality improvement from self-test mode because it does not generate or judge answers.

### 4. Focused live model run

Create local configuration only when it does not already exist:

```powershell
Copy-Item .\evaluation\MagiCore.Evaluation\evalconfig.example.yaml .\evaluation\MagiCore.Evaluation\evalconfig.local.yaml
```

The user must place the API key directly in that ignored file. Never request or echo it through chat.

Run a focused scenario pair:

```powershell
dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --scenario baseline,conflict-resolution
```

Run the model-judged event-time comparison:

```powershell
dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --dataset .\evaluation\MagiCore.Evaluation\evaldataset.temporal.json --scenario baseline,event-time
```

Run the default longitudinal matrix only when broad evidence and its model cost are justified:

```powershell
dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release
```

### 5. Custom dataset

Use the bundled JSON files as schemas. Validate before running:

```powershell
dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --dataset .\path\to\dataset.json --validate-dataset
dotnet run --project .\evaluation\MagiCore.Evaluation\MagiCore.Evaluation.csproj --configuration Release -- --dataset .\path\to\dataset.json --scenario baseline
```

For event-time evaluation, every session date must use `yyyy-MM-dd`. Questions should include explicit ISO dates or years when strict temporal filtering is expected, plus non-temporal controls to test fail-open behavior.

## Interpret the Report

The command prints the Markdown report path and writes matching timestamped `.md` and `.json` files to the configured results directory.

Review in this order:

1. Capability failures. Stop and investigate any failure before comparing quality metrics.
2. Scenario errors and unexpected skips.
3. Accuracy and category accuracy for live model runs.
4. Retrieval hit rate and its numerator/denominator.
5. Mean retrieved candidates, especially for temporal selectivity.
6. Contradiction and adversarial regressions.
7. F1 and BLEU-1 as supporting lexical metrics, not substitutes for judgment.
8. Search latency, ingest time, memory count, and model/runtime settings.
9. Per-question retrieved memories and judge reasoning for every regression.

Wilson intervals describe question-sampling uncertainty only. They do not measure model, prompt, provider, or run-to-run variance. Repeat live runs before interpreting small differences.

## Validate Code Changes

After changing the harness or memory behavior, run:

```powershell
dotnet test .\tests\MagiCore.Tests\MagiCore.Tests.csproj --configuration Release --no-restore
dotnet test .\tests\MagiCore.NetStandard.Tests\MagiCore.NetStandard.Tests.csproj --configuration Release --no-restore
dotnet build .\MagiCore.slnx --configuration Release --no-restore
```

Report exact pass/fail/skip counts, selected scenarios, dataset, mode, models, key metric deltas, generated report path, and existing unrelated warnings. Remove incidental timestamped reports after validation unless the user asked to retain them.