Maple Eval

maple-eval

Model Agnostic Supervised Local Explanations — Plumb et al. (2018) (arXiv:1807.02910, 2018)

What this evaluates

This evaluation probes the fidelity and predictive accuracy of local explanations generated by MAPLE. It measures how well a local linear model approximates the target model's predictions in the neighborhood of a test point, while also benchmarking overall regression accuracy against standard baselines.

Datasets

• UCI datasets — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/GDPlumb/MAPLE

Metrics

• causal metric (primary) — range: other
  • Computes the RMSE between the local model's predictions and the target model's predictions at perturbed points sampled from a Gaussian neighborhood $p_x = \mathcal{N}(x, \sigma I)$. Uses squared $l_2$ loss averaged over 5 samples per test point.

Input / output format

Input: Standardized feature vector x from the test set.

Output: Local linear model parameters (coefficients and intercept) used to predict at perturbed points x'.

Scoring recipe

def compute_causal_metric_rmse(local_model, target_model, test_set, sigma=0.1, n_samples=5):
    sq_losses = []
    for x in test_set:
        for _ in range(n_samples):
            x_prime = np.random.normal(x, sigma)
            pred_local = local_model.predict(x_prime)
            pred_target = target_model.predict(x_prime)
            sq_losses.append((pred_local - pred_target) ** 2)
    return np.sqrt(np.mean(sq_losses))

Common pitfalls

• The neighborhood scale sigma must be chosen carefully; sigma=0.1 is used because data is normalized, but larger values like 0.25 change performance rankings in high dimensions.
• The causal metric evaluates fidelity to the target model's predictions, not ground truth labels, so a low score means the explanation faithfully reproduces the model, not necessarily that it is correct.
• All features and the response must be standardized to mean zero and variance one before running the evaluation.

Evidence (verbatim from paper)

We use our proposed causal metric defined in (1) as our evaluation metric, defining $p_x$ as $\mathcal{N}(x,\sigma I)$ , using the squared $l_2$ loss, and approximating the expectation by taking $x$ from the testing set and drawing five $x'$ per testing point.

Citation

@misc{plumb2018maple,
  title={Model Agnostic Supervised Local Explanations},
  author={Plumb et al. (2018)},
  year={2018},
  note={arXiv:1807.02910}
}

• arXiv: 1807.02910