Srdsd Feynman Eval

srdsd-feynman-eval

Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery — Matsubara et al. (2022) (arXiv:2206.10540, 2022)

What this evaluates

Evaluates symbolic regression methods on their ability to recover known physical laws from tabular data, testing both predictive accuracy and structural interpretability while probing robustness against irrelevant dummy variables.

Datasets

• SRSD-Feynman — total 240; splits: test (-1)

Metrics

• R^2 > 0.999 (primary) — range: percent
  • Binary accuracy metric where a prediction is considered correct if the coefficient of determination (R²) on the held-out test set exceeds 0.999.
• Solution Rate — range: percent
  • Binary metric reporting the percentage of independent runs that exactly recover the true symbolic expression.
• NED — range: [0, 1]
  • Normalized Edit Distance measuring the structural divergence between the parse trees of the predicted and true equations, normalized to [0, 1] where 0 indicates identical structure.

Input / output format

Input: Tabular dataset containing feature columns (including potential dummy variables) and a target column, along with the true underlying physical law.

Output: A symbolic mathematical expression (equation) representing the predicted relationship between features and the target.

Scoring recipe

def evaluate(pred_expr, gold_expr, test_X, test_y):
    r2 = 1 - (sum((test_y - pred_y)**2) / sum((test_y - mean(test_y))**2))
    r2_acc = 1.0 if r2 > 0.999 else 0.0
    sol_rate = 1.0 if str(pred_expr) == str(gold_expr) else 0.0
    pred_tree = parse_to_tree(pred_expr)
    gold_tree = parse_to_tree(gold_expr)
    ned = normalized_edit_distance(pred_tree, gold_tree)
    return {'R^2 > 0.999': r2_acc, 'Solution Rate': sol_rate, 'NED': ned}

Common pitfalls

• High R² scores can be achieved by models that incorrectly include dummy/irrelevant variables, masking poor structural recovery.
• Solution rate is a strict binary metric that becomes sparse (0%) on harder datasets, failing to capture how close a prediction is to the true formula.
• NED requires parsing symbolic expressions into trees; naive string comparison will yield incorrect distances.

Evidence (verbatim from paper)

Table 4: Baseline results for SRSD-Feynman from various perspectives: 1) accuracy (R^2 > 0.999) (La Cava et al., 2021), 2) solution rate (La Cava et al., 2021), and 3) NED (normalized edit distance).

Citation

@misc{matsubara2022rethinking,
  title={Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery},
  author={Matsubara et al. (2022)},
  year={2022},
  note={arXiv:2206.10540}
}

• arXiv: 2206.10540