qhjqhj00 tfbs-classification-eval
Genetic Architect: Discovering Genomic Structure with Learned Neural Architectures — Deming et al. (2016) (arXiv:1605.07156, 2016)
What this evaluates
Evaluates a model's ability to classify short DNA sequences as transcription factor binding sites or not, capturing its capacity to learn regulatory sequence patterns from genomic data.
Datasets
- TFBS classification — total ?; splits: train (-1), test (-1)
Metrics
AUC(primary) — range: [0, 1]- Area under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
Input / output format
Input: One-hot encoded array of length 101 representing a DNA sequence (A, C, G, T).
Output: Binary classification label (1 for binding site, 0 for non-binding).
Scoring recipe
def compute_auc(predictions, labels):
# predictions: probabilities or scores for positive class
# labels: ground truth binary labels
fpr, tpr, _ = roc_curve(labels, predictions)
return auc(fpr, tpr)
Common pitfalls
- Datasets are paired (one positive, one negative per dataset), requiring careful handling during training and evaluation.
- AUC is computed per dataset and then averaged across all 108 datasets, rather than pooling all predictions globally before computing a single AUC.
Evidence (verbatim from paper)
The input and target data for the TFBS classification task consists of 108 datasets with an average of ~31,000 sequences of 101 characters per dataset. In a head-to-head comparison on each dataset, AttentionNet outperforms DeepMotif in 67.6% of cases and the mean AUC across datasets for AttentionNet is 0.933, improving over both DeepMotif (0.927) and DeepBind (0.904) (Table 1). The input dataset for the ILSEP task is 14,116 one-hot encoded (4,2000) input promoter sequences and corresponding (243,) floating point gene expression outputs ranging between 2.60 and 13.95 (see appendix for details). We split the dataset using 10-fold cross validation to obtain predictions for all promoter gene expression pairs. The PromoterNet architecture demonstrates a marked gain in performance over DeepBind and DeepMotif architectures adapted to the ILSEP regression task, achieving an average Pearson r correlation value of 0.587 between out-of-sample predictions and target expression values across lineages, compared to 0.506 and 0.441 for DeepBind (Alipanahi et al., 2015) and DeepMotif (Lanchantin et al., 2016) respectively (Figure 3A).
Citation
@misc{deming2016genetic,
title={Genetic Architect: Discovering Genomic Structure with Learned Neural Architectures},
author={Deming et al. (2016)},
year={2016},
note={arXiv:1605.07156}
}
- arXiv: 1605.07156