Vera AI Structured Generating

Structured/Tabular Data --- Full Analysis & Manuscript Generation

Table of Contents

• Scope Boundary
• Workflow
• Additional Inputs
• Output Structure
• Key References (read before generation)
• Reporting Standards
• Method Status
• Leakage Guards (MANDATORY — must appear verbatim in generated code)
• Task-Type Branching (Classification vs Regression)
• Configuration Defaults
• Why These Defaults
• Minimal Smoke Test
• Cross-Skill Interface

Open-source skill. Read reference/specs/output-variation-protocol.md before every generation --- apply all variation layers for natural, diverse output.

Scope Boundary

Use this skill when:

• vera-ai-structured-reviewing has already established the target type and baseline.
• The goal is supervised tabular prediction with a richer model battery and cross-method comparison.

Do not use this skill when:

• The task is survival analysis, time series, repeated measures, causal estimation, or multimodal modeling where text/images dominate.
• The dataset cannot support anything beyond exploratory modeling.
• A method is desired that is not explicitly listed in the status table below.

Workflow

Continues from where vera-ai-structured-reviewing stopped (PART 0-2 done).

Additional Inputs

Collect if not already provided:

• Target discipline (for reporting conventions)
• Target journal or style (JMLR, NeurIPS, ICML, domain journal, etc.)
• Research question / hypothesis
• Task type: classification or regression
• Subgroup variable for stratification

Output Structure

output/
├── methods.md
├── results.md
├── tables/             <- Markdown + CSV per table
├── figures/            <- PNGs, 300 DPI
├── references.bib
└── code.py             <- Style-varied

Key References (read before generation)

Reporting Standards

Same as vera-ai-structured-reviewing, plus:

• Classification: report F1 (weighted) and AUC (macro) with bootstrapped 95% CIs
• Regression: report RMSE, R2, and MAE with bootstrapped 95% CIs
• Deep learning: report training epochs, best epoch, learning rate, batch size
• TabNet: report n_steps, n_a, n_d, relaxation factor, sparsity coefficient
• Feature importance: unified 0-100 scale across ML and DL models
  • LogReg: |coefficients| rescaled
  • SVM: permutation importance
  • RF: Gini importance
  • XGBoost/LightGBM/CatBoost: gain-based importance
  • MLP: permutation importance
  • TabNet: built-in attention masks
• Model comparison: frame as convergent findings, not horse race
• Stacking ensemble: report base learners, meta-learner, CV strategy
• Tree-based with small N: frame as "exploratory"; never claim generalizability

Method Status

Leakage Guards (MANDATORY — must appear verbatim in generated code)

Two leakage paths are common in tabular pipelines and MUST be blocked explicitly. Generated code MUST use sklearn.pipeline.Pipeline (or ColumnTransformer + Pipeline) so that any fit is confined to the training fold:

1. Target encoding leakage. High-cardinality categorical features encoded by target mean MUST be fit on train only, never on the full dataset. Inside cross-validation, target encoding is fit inside each fold, not once globally. Use category_encoders.TargetEncoder inside a Pipeline, or K-fold out-of-fold target encoding (sklearn.preprocessing.TargetEncoder with cv argument). NEVER compute df['target_enc'] = df.groupby('cat')['y'].transform('mean') before splitting — that leaks the test target into train features.

2. Stacking meta-learner leakage. The meta-learner MUST be trained on out-of-fold (OOF) predictions from the base learners, never on in-fold predictions. Use sklearn.ensemble.StackingClassifier / StackingRegressor with cv=5 (default generates OOF) — this is correct by construction. When writing stacking manually: (a) split train into K folds, (b) for each fold, fit base learners on K−1 folds and predict on the held-out fold, (c) concatenate the K held-out prediction vectors into the OOF matrix used as meta-learner features, (d) refit base learners on full train for inference. NEVER train the meta-learner on predictions from models that saw those same rows during fitting.

Also standard: fit imputer, scaler, and encoder on train-fold only (all via Pipeline); never refit on test or on combined train+test.

Task-Type Branching (Classification vs Regression)

This skill supports both classification and regression. The task_type key in config/default.json (or detected from the testing-skill output) determines which branch of the workflow runs:

Classifier-specific reporting (calibration, threshold selection, confusion matrix) is skipped for regression; regression-specific reporting (residual diagnostics, heteroscedasticity) is skipped for classification. Each workflow file (04–08) MUST check task_type before emitting code.

Configuration Defaults

Pipeline constants live in config/default.json. Read it before generation. Key knobs in addition to the testing-skill config:

• task_type — "classification" or "regression" (see section above)
• ml_models.classification.*, ml_models.regression.* — per-task hyperparameter grids
• deep_models.{MLP, TabNet}.* — DL hyperparameters
• stacking.{base_learners, meta_learner, cv_folds} — ensemble recipe
• feature_importance.scale (0, 100) — unified scale across model families

To override: create config/local.json.

Why These Defaults

• cv=5 for stacking is the default because it is the smallest fold count that usually gives stable out-of-fold meta-features without turning baseline runs into a compute sink.
• CatBoost is kept as a first-class method because native categorical handling removes a large amount of brittle feature engineering from the open-source path.
• Ridge is the default linear regression baseline because it is numerically better behaved than unconstrained OLS once the feature set gets moderately wide after encoding.

Minimal Smoke Test

• Smoke-test prompt: "Continue from vera-ai-structured-reviewing on load_breast_cancer() or load_diabetes(). Fit the full additional tabular model battery, produce the unified artifacts, and keep all preprocessing inside train-fold-only pipelines."
• Expected pass condition: at least one linear model, one tree ensemble, one neural model, and one ensemble method complete without violating the leakage guards.
• Expected artifacts: methods.md, results.md, tables/, figures/, references.bib, and one consolidated code.py.

Cross-Skill Interface

Method Unit Contract:
├── code_python      -> .py script (style-varied)
├── methods_md       -> methods.md (varied structure)
├── results_md       -> results.md (varied phrasing)
├── tables/          -> Markdown + CSV
├── figures/         -> PNGs 300 DPI (varied layout)
├── references_bib   -> .bib with cited references
└── comparison       -> cross-method narrative (in results.md)