Deepecmp Eval

deepecmp-eval

DeepECMP: Predicting Extracellular Matrix Proteins using Deep Learning — Ghafoor et al. (2021) (arXiv:2110.03689, 2021)

What this evaluates

Binary classification of protein sequences to determine if they are extracellular matrix (ECM) proteins. It evaluates the model's ability to handle class imbalance and generalize across different species and feature extraction methods.

Datasets

• benchmark dataset — total ?; splits: test (-1)
• independent dataset — total ?; splits: test (-1)
• ECMPride dataset — total ?; splits: test (-1)

Metrics

• balanced accuracy (primary) — range: percent
  • Calculated as the average of sensitivity (recall for the positive class) and specificity (recall for the negative class): (Sensitivity + Specificity) / 2. Used as the headline metric to rank performance across methods.
• accuracy — range: percent
  • Overall proportion of correct predictions: (TP + TN) / (TP + TN + FP + FN).
• sensitivity — range: percent
  • True positive rate: TP / (TP + FN).
• specificity — range: percent
  • True negative rate: TN / (TN + FP).

Input / output format

Input: Protein sequence string (provided as FASTA files or via UniProtKB API), internally converted to a 100-dimensional vector using ProtVec.

Output: Binary classification label (ECM protein vs non-ECM protein). The ensemble outputs predictions from 11 feed-forward neural networks, and the majority vote determines the final label.

Scoring recipe

def balanced_accuracy(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    specificity = tn / (tn + fp) if (tn + fp) > 0 else 0.0
    return (sensitivity + specificity) / 2 * 100

Common pitfalls

• Using raw accuracy on highly imbalanced datasets will overestimate performance; balanced accuracy is required for fair comparison.
• Performance varies significantly across the benchmark, independent, and ECMPride datasets due to differences in species coverage and feature extraction (e.g., PSSM vs ProtVec).
• The ensemble voting mechanism (majority vote across 11 FNNs) must be implemented exactly as described; averaging probabilities or using a single model yields different results.

Evidence (verbatim from paper)

DeepECMP reached $83.6%$ balanced accuracy on the benchmark dataset (Table 2), $71.0%$ balanced accuracy on the independent dataset (Table 3) and $77.5%$ balanced accuracy on the ECMPride dataset (Table 4). By using balanced accuracy as the metric to rank performance, on the benchmark dataset DeepECMP ranked 4/7, on the independent dataset 3/7 and 3/3 on the ECMPride dataset.

Citation

@misc{ghafoor2021deepecmp,
  title={DeepECMP: Predicting Extracellular Matrix Proteins using Deep Learning},
  author={Ghafoor et al. (2021)},
  year={2021},
  note={arXiv:2110.03689}
}

• arXiv: 2110.03689