Cohen's Kappa Agreement Analysis for Categorical Variation Signs
Summary
Compute Cohen's kappa coefficient to quantify concordance between two categorical classifiers (e.g., RAS and RPS directional changes) across multiple pairwise comparisons, accounting for chance agreement. Essential for validating whether two independent computational predictions of reaction flux direction agree better than random.
When to use
When you have two independent predictions of categorical outcomes (up/down/no-change variation signs) across multiple sample pairs and need to measure agreement beyond what would be expected by chance. Specifically, use this when comparing RAS directional variations derived from gene expression against RPS directional variations derived from substrate concentrations, to discriminate transcriptional from metabolic regulation.
When NOT to use
- Input variation signs are continuous (not categorical): use Pearson or Spearman correlation instead of kappa
- More than two raters/classifiers need agreement assessment: use Fleiss' kappa or Krippendorff's alpha instead
- Reactions lack complete substrate metabolomics coverage: these are filtered out before kappa computation per the protocol
Inputs
- RAS variation signs (up/down/no-change) per reaction and cell-line pair from t-test on normalized transcriptomics scores
- RPS variation signs (up/down/no-change) per reaction and cell-line pair from Mann–Whitney U test on sampled flux distributions
- FFD (flux distribution) variation signs per reaction and cell-line pair
- List of reactions with complete substrate metabolomics coverage (81 reactions in INTEGRATE study)
- 10 pairwise cell-line comparisons
Outputs
- Cohen's kappa coefficient table with values for each reaction and pairwise comparison (RPSvsRAS, RPSvsFFD, RASvsFFD columns)
- Scatter plot (RPSvsRAS kappa vs RPSvsFFD kappa, colored by RASvsFFD score, labeled for kappa ≥ 0.2)
- Heatmap of Cohen's kappa values for all 81 fully-covered reactions, ordered by RPSvsFFD score
- Validation statistics comparing empirical agreement distribution to random baseline
How to apply
For each reaction and each of the 10 pairwise cell-line comparisons, extract the sign of directional change (up=+1, down=−1, no-change=0) from both RAS (computed via t-test on normalized transcriptomics scores) and RPS (computed via Mann–Whitney U test on flux distribution samples). Compute Cohen's kappa as κ = (p_o − p_e) / (1 − p_e), where p_o is the empirical proportion of agreement and p_e is the expected proportion under random assignment. Generate scatter plots with kappa values on axes (e.g., RPSvsRAS kappa vs RPSvsFFD kappa), label points for reactions with fair or better concordance (kappa ≥ 0.2), and produce heatmaps ordered by concordance score. Validate that empirical agreement probabilities exceed those from randomized datasets and that results match published figures (e.g., Fig 4A–B).
Related tools
- scipy.stats or statsmodels Cohen's kappa implementation (Direct computation of κ from contingency tables of variation signs)
- COBRApy optGpSampler (Generates sampled flux distributions for FFD variation sign computation via Mann–Whitney U test) — https://github.com/opencobra/cobrapy
- Pandas/NumPy (Data wrangling and sign extraction from RAS and RPS vectors; contingency table construction)
- Matplotlib/Seaborn (Scatter plot and heatmap visualization of Cohen's kappa concordance scores)
Examples
python pipeline/concordanceAnalysis.py --valLog 1.2 --weight linear --resultsMetabolomicFile resultsMetabolomic --metabolic_model ENGRO2_irrev.xml --lcellLines ['MCF102A','SKBR3','MCF7','MDAMB231','MDAMB361']
Evaluation signals
- Kappa values for all 81 fully-covered reactions are produced with no missing entries for reactions with complete GPR associations
- Kappa ≥ 0.2 reactions (fair agreement or better) are correctly labeled and visible in scatter and heatmap plots, matching published Fig 4A–B
- Empirical agreement proportions (p_o) exceed expected random agreement (p_e) for majority of reactions, confirming non-chance concordance
- Heatmap is properly ordered by RPSvsFFD kappa scores in descending order
- Scatter plot axis labels (RPSvsRAS, RPSvsFFD) and color scale (RASvsFFD) are internally consistent and match paper notation
Limitations
- Cohen's kappa requires both classifiers to yield valid categorical outcomes; reactions without GPR associations produce missing RAS values and are excluded from concordance analysis
- Kappa is undefined or poorly-behaved when one category is extremely rare (e.g., all reactions classified as 'no-change'); the protocol filters to the 81 reactions with full metabolomics coverage to mitigate this
- Kappa depends on the choice of sign threshold (fold-change ≥20%, p<0.05); different thresholds for RAS and RPS tests can inflate disagreement unrelated to biological regulation
- With 10 pairwise comparisons, multiple-testing correction (Benjamini–Hochberg FDR) must be applied to empirical p-values derived from randomized baseline sampling to set significance at FDR<5%
Evidence
- [other] compute the Cohen's kappa coefficient quantifying agreement between RAS variation sign and RPS variation sign, accounting for chance agreement: "For each reaction and each pairwise comparison, compute the Cohen's kappa coefficient quantifying agreement between RAS variation sign and RPS variation sign, accounting for chance agreement."
- [other] kappa values reported in a heatmap, with reactions ranked according to RPSvsFFD concordance scores and only those with scores greater than 0.2 displayed: "Cohen's kappa values reported in a heatmap, with reactions ranked according to RPSvsFFD concordance scores and only those with scores greater than 0.2 displayed."
- [other] empirical probability of agreement between RAS and RPS variations exceeds that expected from two independent random datasets: "Validation: verify that the produced kappa values and agreement distributions match those displayed in the published Fig 4A–B and that the empirical probability of agreement between RAS and RPS"
- [other] Filter to retain only the 81 reactions for which all substrate abundances were quantified: "Filter to retain only the 81 reactions for which all substrate abundances were quantified in the LC-MS metabolomics dataset."
- [other] Missing RPSvsRAS values occur when a reaction is not associated with a GPR: "Missing RPSvsRAS values occur when a reaction is not associated with a GPR"
1---2name: cohen-kappa-inter-rater-reliability3description: Use when when you have two independent predictions of categorical outcomes (up/down/no-change variation signs) across multiple sample pairs and need to measure agreement beyond what would be expected by chance.4license: CC-BY-4.05---67# Cohen's Kappa Agreement Analysis for Categorical Variation Signs89## Summary1011Compute Cohen's kappa coefficient to quantify concordance between two categorical classifiers (e.g., RAS and RPS directional changes) across multiple pairwise comparisons, accounting for chance agreement. Essential for validating whether two independent computational predictions of reaction flux direction agree better than random.1213## When to use1415When you have two independent predictions of categorical outcomes (up/down/no-change variation signs) across multiple sample pairs and need to measure agreement beyond what would be expected by chance. Specifically, use this when comparing RAS directional variations derived from gene expression against RPS directional variations derived from substrate concentrations, to discriminate transcriptional from metabolic regulation.1617## When NOT to use1819- Input variation signs are continuous (not categorical): use Pearson or Spearman correlation instead of kappa20- More than two raters/classifiers need agreement assessment: use Fleiss' kappa or Krippendorff's alpha instead21- Reactions lack complete substrate metabolomics coverage: these are filtered out before kappa computation per the protocol2223## Inputs2425- RAS variation signs (up/down/no-change) per reaction and cell-line pair from t-test on normalized transcriptomics scores26- RPS variation signs (up/down/no-change) per reaction and cell-line pair from Mann–Whitney U test on sampled flux distributions27- FFD (flux distribution) variation signs per reaction and cell-line pair28- List of reactions with complete substrate metabolomics coverage (81 reactions in INTEGRATE study)29- 10 pairwise cell-line comparisons3031## Outputs3233- Cohen's kappa coefficient table with values for each reaction and pairwise comparison (RPSvsRAS, RPSvsFFD, RASvsFFD columns)34- Scatter plot (RPSvsRAS kappa vs RPSvsFFD kappa, colored by RASvsFFD score, labeled for kappa ≥ 0.2)35- Heatmap of Cohen's kappa values for all 81 fully-covered reactions, ordered by RPSvsFFD score36- Validation statistics comparing empirical agreement distribution to random baseline3738## How to apply3940For each reaction and each of the 10 pairwise cell-line comparisons, extract the sign of directional change (up=+1, down=−1, no-change=0) from both RAS (computed via t-test on normalized transcriptomics scores) and RPS (computed via Mann–Whitney U test on flux distribution samples). Compute Cohen's kappa as κ = (p_o − p_e) / (1 − p_e), where p_o is the empirical proportion of agreement and p_e is the expected proportion under random assignment. Generate scatter plots with kappa values on axes (e.g., RPSvsRAS kappa vs RPSvsFFD kappa), label points for reactions with fair or better concordance (kappa ≥ 0.2), and produce heatmaps ordered by concordance score. Validate that empirical agreement probabilities exceed those from randomized datasets and that results match published figures (e.g., Fig 4A–B).4142## Related tools4344- **scipy.stats or statsmodels Cohen's kappa implementation** (Direct computation of κ from contingency tables of variation signs)45- **COBRApy optGpSampler** (Generates sampled flux distributions for FFD variation sign computation via Mann–Whitney U test) — https://github.com/opencobra/cobrapy46- **Pandas/NumPy** (Data wrangling and sign extraction from RAS and RPS vectors; contingency table construction)47- **Matplotlib/Seaborn** (Scatter plot and heatmap visualization of Cohen's kappa concordance scores)4849## Examples5051```52python pipeline/concordanceAnalysis.py --valLog 1.2 --weight linear --resultsMetabolomicFile resultsMetabolomic --metabolic_model ENGRO2_irrev.xml --lcellLines ['MCF102A','SKBR3','MCF7','MDAMB231','MDAMB361']53```5455## Evaluation signals5657- Kappa values for all 81 fully-covered reactions are produced with no missing entries for reactions with complete GPR associations58- Kappa ≥ 0.2 reactions (fair agreement or better) are correctly labeled and visible in scatter and heatmap plots, matching published Fig 4A–B59- Empirical agreement proportions (p_o) exceed expected random agreement (p_e) for majority of reactions, confirming non-chance concordance60- Heatmap is properly ordered by RPSvsFFD kappa scores in descending order61- Scatter plot axis labels (RPSvsRAS, RPSvsFFD) and color scale (RASvsFFD) are internally consistent and match paper notation6263## Limitations6465- Cohen's kappa requires both classifiers to yield valid categorical outcomes; reactions without GPR associations produce missing RAS values and are excluded from concordance analysis66- Kappa is undefined or poorly-behaved when one category is extremely rare (e.g., all reactions classified as 'no-change'); the protocol filters to the 81 reactions with full metabolomics coverage to mitigate this67- Kappa depends on the choice of sign threshold (fold-change ≥20%, p<0.05); different thresholds for RAS and RPS tests can inflate disagreement unrelated to biological regulation68- With 10 pairwise comparisons, multiple-testing correction (Benjamini–Hochberg FDR) must be applied to empirical p-values derived from randomized baseline sampling to set significance at FDR<5%6970## Evidence7172- [other] compute the Cohen's kappa coefficient quantifying agreement between RAS variation sign and RPS variation sign, accounting for chance agreement: "For each reaction and each pairwise comparison, compute the Cohen's kappa coefficient quantifying agreement between RAS variation sign and RPS variation sign, accounting for chance agreement."73- [other] kappa values reported in a heatmap, with reactions ranked according to RPSvsFFD concordance scores and only those with scores greater than 0.2 displayed: "Cohen's kappa values reported in a heatmap, with reactions ranked according to RPSvsFFD concordance scores and only those with scores greater than 0.2 displayed."74- [other] empirical probability of agreement between RAS and RPS variations exceeds that expected from two independent random datasets: "Validation: verify that the produced kappa values and agreement distributions match those displayed in the published Fig 4A–B and that the empirical probability of agreement between RAS and RPS"75- [other] Filter to retain only the 81 reactions for which all substrate abundances were quantified: "Filter to retain only the 81 reactions for which all substrate abundances were quantified in the LC-MS metabolomics dataset."76- [other] Missing RPSvsRAS values occur when a reaction is not associated with a GPR: "Missing RPSvsRAS values occur when a reaction is not associated with a GPR"
HolobiomicsLab 