# Maple Eval

> This evaluation probes the fidelity and predictive accuracy of local explanations generated by MAPLE. It measures how well a local linear model approximates the target model's predictions in the neighborhood of a test point, while also benchmarking overall regression accuracy against standard baselines. Use when the user wants to benchmark on UCI datasets, or asks about evaluating this task. Reports causal metric.

- Skill: `qhjqhj00/maple-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/maple-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/maple-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/maple-eval

---


# maple-eval

> Model Agnostic Supervised Local Explanations — Plumb et al. (2018) (arXiv:1807.02910, 2018)

## What this evaluates

This evaluation probes the fidelity and predictive accuracy of local explanations generated by MAPLE. It measures how well a local linear model approximates the target model's predictions in the neighborhood of a test point, while also benchmarking overall regression accuracy against standard baselines.

## Datasets

- **UCI datasets** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/GDPlumb/MAPLE

## Metrics

- `causal metric` **(primary)** — range: other
  - Computes the RMSE between the local model's predictions and the target model's predictions at perturbed points sampled from a Gaussian neighborhood $p_x = \mathcal{N}(x, \sigma I)$. Uses squared $l_2$ loss averaged over 5 samples per test point.

## Input / output format

**Input**: Standardized feature vector x from the test set.

**Output**: Local linear model parameters (coefficients and intercept) used to predict at perturbed points x'.

## Scoring recipe

```python
def compute_causal_metric_rmse(local_model, target_model, test_set, sigma=0.1, n_samples=5):
    sq_losses = []
    for x in test_set:
        for _ in range(n_samples):
            x_prime = np.random.normal(x, sigma)
            pred_local = local_model.predict(x_prime)
            pred_target = target_model.predict(x_prime)
            sq_losses.append((pred_local - pred_target) ** 2)
    return np.sqrt(np.mean(sq_losses))
```

## Common pitfalls

- The neighborhood scale sigma must be chosen carefully; sigma=0.1 is used because data is normalized, but larger values like 0.25 change performance rankings in high dimensions.
- The causal metric evaluates fidelity to the target model's predictions, not ground truth labels, so a low score means the explanation faithfully reproduces the model, not necessarily that it is correct.
- All features and the response must be standardized to mean zero and variance one before running the evaluation.

## Evidence (verbatim from paper)

> We use our proposed causal metric defined in (1) as our evaluation metric, defining  $p_x$  as  $\mathcal{N}(x,\sigma I)$ , using the squared  $l_2$  loss, and approximating the expectation by taking  $x$  from the testing set and drawing five  $x'$  per testing point.

## Citation

```bibtex
@misc{plumb2018maple,
  title={Model Agnostic Supervised Local Explanations},
  author={Plumb et al. (2018)},
  year={2018},
  note={arXiv:1807.02910}
}
```

- arXiv: 1807.02910