Ilsep Regression Eval

ilsep-regression-eval

Genetic Architect: Discovering Genomic Structure with Learned Neural Architectures — Deming et al. (2016) (arXiv:1605.07156, 2016)

What this evaluates

Probes a model's ability to predict gene expression levels from promoter sequences across different cellular contexts, testing its capacity to capture long-range regulatory dependencies and quantitative biological signals.

Datasets

• ILSEP — total 14116; splits: train (-1), val (-1), test (-1)

Metrics

• Pearson r (primary) — range: [-1, 1]
  • Pearson correlation coefficient measuring the linear correlation between predicted and actual gene expression values.

Input / output format

Input: One-hot encoded array of shape (4, 2000) representing a promoter DNA sequence.

Output: Floating point vector of shape (243,) representing gene expression levels across different lineages/cell types.

Scoring recipe

def compute_pearson_r(predictions, labels):
    # predictions: shape (N, 243) or (N,)
    # labels: shape (N, 243) or (N,)
    return np.corrcoef(predictions.flatten(), labels.flatten())[0, 1]

Common pitfalls

• Evaluation uses 10-fold cross-validation, so the reported metric is an average across folds, not a single train/test split.
• The task supports both single-task regression (one model per lineage) and multi-task regression (243 outputs per model); metrics are averaged across lineages in both cases.

Evidence (verbatim from paper)

The input and target data for the TFBS classification task consists of 108 datasets with an average of ~31,000 sequences of 101 characters per dataset. In a head-to-head comparison on each dataset, AttentionNet outperforms DeepMotif in 67.6% of cases and the mean AUC across datasets for AttentionNet is 0.933, improving over both DeepMotif (0.927) and DeepBind (0.904) (Table 1). The input dataset for the ILSEP task is 14,116 one-hot encoded (4,2000) input promoter sequences and corresponding (243,) floating point gene expression outputs ranging between 2.60 and 13.95 (see appendix for details). We split the dataset using 10-fold cross validation to obtain predictions for all promoter gene expression pairs. The PromoterNet architecture demonstrates a marked gain in performance over DeepBind and DeepMotif architectures adapted to the ILSEP regression task, achieving an average Pearson r correlation value of 0.587 between out-of-sample predictions and target expression values across lineages, compared to 0.506 and 0.441 for DeepBind (Alipanahi et al., 2015) and DeepMotif (Lanchantin et al., 2016) respectively (Figure 3A).

Citation

@misc{deming2016genetic,
  title={Genetic Architect: Discovering Genomic Structure with Learned Neural Architectures},
  author={Deming et al. (2016)},
  year={2016},
  note={arXiv:1605.07156}
}

• arXiv: 1605.07156