Granger Causal Inference Eval

granger-causal-inference-eval

Granger causal inference on DAGs identifies genomic loci regulating transcription — Singh et al. (2022) (arXiv:2210.10168, 2022)

What this evaluates

Probes a model's ability to identify causal genomic regulatory relationships (ATAC-seq peaks to RNA-seq genes) using temporal causal inference on single-cell multimodal data. It evaluates how well predicted peak-gene associations align with independent biological proxies like eQTLs and chromatin interactions, testing robustness to high-dimensional sparsity and partial temporal orderings.

Datasets

• sci-CAR, SNARE-seq, SHARE-seq — total ?; splits: test (-1)

Metrics

• AUPRC (primary) — range: [0, 1]
  • Area under the precision-recall curve computed over ranked peak-gene pair scores against binary labels derived from eQTL p-values or chromatin interaction overlaps.
• AUROC — range: [0, 1]
  • Area under the receiver operating characteristic curve computed over the same ranked scores and binary labels.

Input / output format

Input: Per-cell log-transformed ATAC-seq and RNA-seq count profiles, pseudotime estimates, and a kNN graph structure. Candidate peak-gene pairs are defined as ATAC-seq peaks within 1 Mb of a gene's transcription start site.

Output: Continuous association scores ranking the likelihood of a causal regulatory relationship for each candidate peak-gene pair.

Scoring recipe

def compute_metrics(scores, labels):
    # scores: float array of association scores for peak-gene pairs
    # labels: binary array (1 for true eQTL/interaction, 0 for false)
    precision, recall, _ = precision_recall_curve(labels, scores)
    auprc = auc(recall, precision)
    fpr, tpr, _ = roc_curve(labels, scores)
    auroc = auc(fpr, tpr)
    return auprc, auroc

Common pitfalls

• Ground truth is unavailable at genome scale, so evaluation relies on indirect proxies (eQTLs and Hi-C/5C) which have their own errors and sparsity.
• Mouse eQTL data is limited, so human eQTLs are mapped to the mouse genome, potentially introducing cross-species regulatory mismatches.
• Methods are compared relatively against each other rather than against an absolute performance threshold due to proxy dataset noise.

Evidence (verbatim from paper)

To assess the effectiveness of GrID-Net in predicting eQTLs, we labeled peak–gene pairs associated with eQTLs as true (eQTL p<10^-10) or false (eQTL p>0.9), discarding pairs that were not in either category. GrID-Net and the alternative methods were evaluated on their accuracy in predicting the true eQTLs based on the ranking of association scores generated by each method for each peak–gene pair. Across all datasets, GrID-Net outperformed the other methods in predicting eQTLs, achieving the highest AUPRC (area under precision-recall curve) and AUROC (area under receiver operating characteristic) in all cases, indicating that peak–gene pairs prioritized by GrID-Net more closely align with evidence from population genetics (Table 1).

Citation

@misc{singh2022granger,
  title={Granger causal inference on DAGs identifies genomic loci regulating transcription},
  author={Singh et al. (2022)},
  year={2022},
  note={arXiv:2210.10168}
}

• arXiv: 2210.10168