# Grables Eval

> Evaluates whether tabular models can capture extension-sensitive inter-row dependencies (e.g., global counts, overlaps) compared to graph-based message-passing models, and tests if hybrid approaches combining tabular features with graph-derived representations improve performance. Use when the user wants to benchmark on Synthetic transactions dataset, Retail transaction dataset, relbench-trial, or asks about evaluating this task. Reports ROC-AUC.

- Skill: `qhjqhj00/grables-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/grables-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/grables-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/grables-eval

---


# grables-eval

> Grables: Tabular Learning Beyond Independent Rows — Tamara Cucumides, Floris Geerts (2026) (arXiv:2602.03945, 2026)

## What this evaluates

Evaluates whether tabular models can capture extension-sensitive inter-row dependencies (e.g., global counts, overlaps) compared to graph-based message-passing models, and tests if hybrid approaches combining tabular features with graph-derived representations improve performance.

## Datasets

- **Synthetic transactions dataset** — total ?; splits: train (-1), val (-1), test (-1)
- **Retail transaction dataset** — total ?; splits: train (-1), val (-1), test (-1)
- **relbench-trial** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `ROC-AUC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve; measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall, reported to handle class imbalance in logical tasks.

## Input / output format

**Input**: Tabular rows with categorical and numerical features, or graph structures (incidence/relational entity graphs) constructed from the tables.

**Output**: Predicted probabilities or class labels for the target variable.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred_proba):
    auc = roc_auc_score(y_true, y_pred_proba)
    y_pred = (y_pred_proba >= 0.5).astype(int)
    f1 = f1_score(y_true, y_pred, average='macro')
    return {'roc_auc': auc, 'f1': f1}
```

## Common pitfalls

- Tasks must be generated independently within each train/val/test split to prevent information leakage across splits.
- Tabular models may exploit row-local shortcuts rather than learning true inter-row dependencies; stress-test perturbations are required to verify genuine relational reasoning.
- Class imbalance is common; relying solely on accuracy or AUC without F1 can mask poor minority-class performance.

## Evidence (verbatim from paper)

> Across all experiments, performance is measured using the area under the receiver operating characteristic curve (ROC–AUC). Because the logical tasks naturally induce class imbalance, we additionally report F1 scores where appropriate.

## Citation

```bibtex
@misc{cucumides2026grables,
  title={Grables: Tabular Learning Beyond Independent Rows},
  author={Tamara Cucumides, Floris Geerts (2026)},
  year={2026},
  note={arXiv:2602.03945}
}
```

- arXiv: 2602.03945