Gru D Eval

gru-d-eval

Recurrent Neural Networks for Multivariate Time Series with Missing Values — Che et al. (2016) (arXiv:1606.01865, 2016)

What this evaluates

This evaluation probes a model's ability to handle multivariate time series with missing values by jointly learning temporal dependencies and informative missing patterns. It tests classification performance on clinical and synthetic datasets, measuring how well the model exploits masking and time-interval information for early prediction and multi-task diagnosis.

Datasets

• Gesture — total 378; splits: train (-1), val (-1), test (-1)
• PhysioNet Challenge 2012 — total 8000; splits: train (-1), val (-1), test (-1)
• MIMIC-III — total 19714; splits: train (-1), val (-1), test (-1)

Metrics

• AUC score (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve. It measures the probability that a classifier ranks a randomly chosen positive instance higher than a randomly chosen negative one, averaged across all classification thresholds.

Input / output format

Input: Multivariate time series sequences of measurements, accompanied by binary masking vectors (indicating observed vs. missing entries) and time-interval vectors (encoding gaps between observations). Inputs are normalized to zero mean and unit variance. Non-RNN baselines use fixed-length, regularly sampled sequences with forward/backward imputation.

Output: Classification probabilities or hard labels via a soft-max regressor applied to the final hidden state. Supports binary classification (e.g., mortality) and multi-task classification (e.g., 4 PhysioNet tasks or 20 MIMIC-III ICD-9 codes).

Scoring recipe

auc_scores = []
for fold in range(5):
    train, val, test = split_into_folds(dataset, k=5)
    model = train_model(train, val)  # early stopping on val
    preds = model.predict(test)
    auc = compute_auc(y_true=test.labels, y_score=preds)
    auc_scores.append(auc)
final_score = np.mean(auc_scores)
report_mean_std(final_score, auc_scores)

Common pitfalls

• Non-RNN baselines require fixed-length inputs and forward/backward imputation, which can distort temporal dynamics compared to RNNs that handle variable lengths natively.
• Ignoring informative missingness (masking and time intervals) leads to suboptimal performance, especially when missingness patterns correlate with the target label.
• Early stopping is applied on a validation set, but the exact validation split strategy beyond 5-fold CV is not detailed, potentially affecting reproducibility.

Evidence (verbatim from paper)

We report the results from 5-fold cross validation in terms of area under the ROC curve (AUC score).

Citation

@misc{che2016recurrent,
  title={Recurrent Neural Networks for Multivariate Time Series with Missing Values},
  author={Che et al. (2016)},
  year={2016},
  note={arXiv:1606.01865}
}

• arXiv: 1606.01865