mapping-agreement-cross-tool-validation
Summary
Quantify per-read and per-transcript agreement between two transcript-quantification tools (e.g., salmon versions or implementations) to assess consistency and expose algorithmic differences. This skill validates that a reimplementation or alternative tool produces equivalent quantification outputs within acceptable tolerance.
When to use
When you have built a new tool implementation or major version and need to verify it produces equivalent results to a reference implementation on the same input data and index. Specifically: when two versions claim to use the same algorithm (selective alignment, chaining, MEM extraction) but differ in language/rewrite, and you need to quantify the magnitude of discrepancy and trace which algorithmic steps account for residual differences.
When NOT to use
- Input indices are not byte-identical or were built with different parameters (k-mer length, decoy handling); index differences will confound tool-implementation differences.
- Tools are being run with different quantification parameters (e.g., one with bias correction, one without; different library-type inference); compare only under matched parameter sets.
- You are comparing tools on different datasets or read lengths; per-read and per-transcript agreement are dataset-dependent and cannot be generalized across experiments.
Inputs
- byte-identical transcriptome index (same FASTA, same k-mer length, deterministic non-ACGT replacement)
- paired-end RNA-seq reads in FASTQ or gzip format
- quant.sf output files from both tools
- unmapped_names.txt files (per-read mapping status) from both tools
Outputs
- Pearson correlation coefficient for NumReads per transcript
- Pearson correlation coefficient for TPM per transcript
- per-read mapping agreement percentage (% reads both tools agree are mapped or unmapped)
- total assigned NumReads difference as percentage
- breakdown of mapping discrepancies by category (fully unmapped, partially mapped)
How to apply
Build a byte-identical reference index and run both tools on the same sequencing dataset (e.g., ERR188044, 36.35M paired-end reads) using identical parameters (e.g., selective-alignment mode, -l A auto-detect library type, no bias correction). Extract per-transcript quantification columns (NumReads, TPM) from both tools' quant.sf outputs and compute Pearson correlations. For per-read mapping agreement, parse unmapped_names.txt files from both runs and count reads on which both tools agree in mapping status (mapped vs. fully unmapped, coded 'u'). Report total assigned reads difference as a percentage. When residual disagreement exists (~0.15% per-read mapping disagreement), identify the source by comparing parameter defaults (e.g., orphanChainSubThresh, postMergeChainSubThresh chain-pruning thresholds) and trace whether the difference is algorithmic or parametric. Strong agreement (Pearson > 0.998, per-read agreement > 99.8%) indicates the implementations are functionally equivalent despite language differences.
Related tools
Examples
salmon index -t GRCh38_cDNA.fa -i idx_salmon2 -k 31 && salmon index -t GRCh38_cDNA.fa -i idx_salmon112 -k 31 && salmon quant -i idx_salmon2 -l A -1 ERR188044_R1.fastq.gz -2 ERR188044_R2.fastq.gz -o quant_rust --no-bias-seq-bias && salmon quant -i idx_salmon112 -l A -1 ERR188044_R1.fastq.gz -2 ERR188044_R2.fastq.gz -o quant_cpp112 --no-bias-seq-bias && paste <(cut -f1,4,5 quant_rust/quant.sf) <(cut -f4,5 quant_cpp112/quant.sf) | Rscript -e 'x <- read.table(stdin()); cor(x[,2],x[,4]); cor(x[,3],x[,5])'
Evaluation signals
- Per-transcript Pearson correlation for NumReads ≥ 0.9985 indicates high quantitative agreement across thousands of transcripts.
- Per-transcript Pearson correlation for TPM ≥ 0.9989 (higher due to normalization reducing noise) indicates abundance ranking is consistent.
- Per-read mapping agreement ≥ 99.8% (i.e., ≤0.2% reads where tools disagree on mapped vs. unmapped status) indicates low residual algorithmic difference.
- Total assigned NumReads difference < 1% indicates tools assign nearly identical total read mass across the transcriptome.
- When disagreement exists, mapping-gap analysis (e.g., '62,812 reads Rust maps but C++ leaves unmapped') can be traced to specific parameter defaults (chain-pruning thresholds account for ~80% of the difference) rather than core algorithmic divergence.
Limitations
- Per-read and per-transcript agreement are sensitive to minor parameter differences (orphanChainSubThresh, postMergeChainSubThresh); identical algorithm with different defaults will show measurable disagreement (up to ~0.2% per-read in this case).
- Correlation metrics alone (Pearson > 0.998) do not reveal systematic bias or directional differences; residual disagreement must be traced by inspecting specific reads and chains to distinguish algorithmic bugs from intentional parameter tuning.
- Index byte-identity is required; any mismatch in k-mer extraction, decoy handling, or bloom-filter configuration will confound the tool comparison and make it impossible to isolate implementation effects.
- This skill validates quantification consistency but does not assess downstream inference (EM, VBEM) convergence or posterior variance; abundance point estimates may agree while credible intervals diverge.
Evidence
- [methods] Per-read mapping agreement between C++ 1.12.0 and Rust is 99.83% on byte-identical index: "per-read mapping agreement | 99.83%"
- [methods] On byte-identical index, salmon 2.0 (Rust) and salmon 1.12.0 (C++) achieve per-transcript Pearson correlations of 0.99854 for NumReads and 0.99897 for TPM: "On a byte-identical index, salmon 2.0 (Rust) and salmon 1.12.0 (C++) achieve per-transcript Pearson correlations of 0.99854 for NumReads and 0.99897 for TPM, with per-read mapping agreement at 99.83%."
- [methods]
80% of mapping gap is attributable to chain-pruning-threshold defaults, not algorithmic divergence: "80% (~49k): a chain-sub-optimality default difference (describable, deferred). C++ prunes low-coverage chains/orphans before alignment using orphanChainSubThresh = 0.95 and postMergeChainSubThresh ="
- [methods] Workflow specifies byte-identical index construction and paired parameter control: "Build a byte-identical GRCh38 cDNA index using both salmon 2.0 and C++ salmon 1.12.0 from the same cleaned FASTA with deterministic non-ACGT replacement (193,759 transcripts). 2. Run salmon 2.0 quant"
- [readme] salmon 2.0 is a Rust rewrite and must be validated against C++ reference: "This is salmon 2.0 — a from-scratch Rust rewrite of salmon. It keeps the same workflow (
salmon index → salmon quant → quant.sf) and the same output formats downstream tools read, but it is a"
1---2name: mapping-agreement-cross-tool-validation3description: Use when when you have built a new tool implementation or major version and need to verify it produces equivalent results to a reference implementation on the same input data and index.4license: CC-BY-4.05---67# mapping-agreement-cross-tool-validation89## Summary1011Quantify per-read and per-transcript agreement between two transcript-quantification tools (e.g., salmon versions or implementations) to assess consistency and expose algorithmic differences. This skill validates that a reimplementation or alternative tool produces equivalent quantification outputs within acceptable tolerance.1213## When to use1415When you have built a new tool implementation or major version and need to verify it produces equivalent results to a reference implementation on the same input data and index. Specifically: when two versions claim to use the same algorithm (selective alignment, chaining, MEM extraction) but differ in language/rewrite, and you need to quantify the magnitude of discrepancy and trace which algorithmic steps account for residual differences.1617## When NOT to use1819- Input indices are not byte-identical or were built with different parameters (k-mer length, decoy handling); index differences will confound tool-implementation differences.20- Tools are being run with different quantification parameters (e.g., one with bias correction, one without; different library-type inference); compare only under matched parameter sets.21- You are comparing tools on different datasets or read lengths; per-read and per-transcript agreement are dataset-dependent and cannot be generalized across experiments.2223## Inputs2425- byte-identical transcriptome index (same FASTA, same k-mer length, deterministic non-ACGT replacement)26- paired-end RNA-seq reads in FASTQ or gzip format27- quant.sf output files from both tools28- unmapped_names.txt files (per-read mapping status) from both tools2930## Outputs3132- Pearson correlation coefficient for NumReads per transcript33- Pearson correlation coefficient for TPM per transcript34- per-read mapping agreement percentage (% reads both tools agree are mapped or unmapped)35- total assigned NumReads difference as percentage36- breakdown of mapping discrepancies by category (fully unmapped, partially mapped)3738## How to apply3940Build a byte-identical reference index and run both tools on the same sequencing dataset (e.g., ERR188044, 36.35M paired-end reads) using identical parameters (e.g., selective-alignment mode, `-l A` auto-detect library type, no bias correction). Extract per-transcript quantification columns (NumReads, TPM) from both tools' quant.sf outputs and compute Pearson correlations. For per-read mapping agreement, parse unmapped_names.txt files from both runs and count reads on which both tools agree in mapping status (mapped vs. fully unmapped, coded 'u'). Report total assigned reads difference as a percentage. When residual disagreement exists (~0.15% per-read mapping disagreement), identify the source by comparing parameter defaults (e.g., orphanChainSubThresh, postMergeChainSubThresh chain-pruning thresholds) and trace whether the difference is algorithmic or parametric. Strong agreement (Pearson > 0.998, per-read agreement > 99.8%) indicates the implementations are functionally equivalent despite language differences.4142## Related tools4344- **salmon 2.0 (Rust)** (primary implementation to validate (generates quant.sf and unmapped_names.txt)) — https://github.com/COMBINE-lab/salmon45- **salmon 1.12.0 (C++)** (reference implementation for cross-tool comparison) — https://github.com/COMBINE-lab/salmon/tree/cpp46- **piscem-rs** (underlying mapper in Rust port; exposes orientation behavior in k-mer lookup) — https://github.com/COMBINE-lab/salmon4748## Examples4950```51salmon index -t GRCh38_cDNA.fa -i idx_salmon2 -k 31 && salmon index -t GRCh38_cDNA.fa -i idx_salmon112 -k 31 && salmon quant -i idx_salmon2 -l A -1 ERR188044_R1.fastq.gz -2 ERR188044_R2.fastq.gz -o quant_rust --no-bias-seq-bias && salmon quant -i idx_salmon112 -l A -1 ERR188044_R1.fastq.gz -2 ERR188044_R2.fastq.gz -o quant_cpp112 --no-bias-seq-bias && paste <(cut -f1,4,5 quant_rust/quant.sf) <(cut -f4,5 quant_cpp112/quant.sf) | Rscript -e 'x <- read.table(stdin()); cor(x[,2],x[,4]); cor(x[,3],x[,5])'52```5354## Evaluation signals5556- Per-transcript Pearson correlation for NumReads ≥ 0.9985 indicates high quantitative agreement across thousands of transcripts.57- Per-transcript Pearson correlation for TPM ≥ 0.9989 (higher due to normalization reducing noise) indicates abundance ranking is consistent.58- Per-read mapping agreement ≥ 99.8% (i.e., ≤0.2% reads where tools disagree on mapped vs. unmapped status) indicates low residual algorithmic difference.59- Total assigned NumReads difference < 1% indicates tools assign nearly identical total read mass across the transcriptome.60- When disagreement exists, mapping-gap analysis (e.g., '62,812 reads Rust maps but C++ leaves unmapped') can be traced to specific parameter defaults (chain-pruning thresholds account for ~80% of the difference) rather than core algorithmic divergence.6162## Limitations6364- Per-read and per-transcript agreement are sensitive to minor parameter differences (orphanChainSubThresh, postMergeChainSubThresh); identical algorithm with different defaults will show measurable disagreement (up to ~0.2% per-read in this case).65- Correlation metrics alone (Pearson > 0.998) do not reveal systematic bias or directional differences; residual disagreement must be traced by inspecting specific reads and chains to distinguish algorithmic bugs from intentional parameter tuning.66- Index byte-identity is required; any mismatch in k-mer extraction, decoy handling, or bloom-filter configuration will confound the tool comparison and make it impossible to isolate implementation effects.67- This skill validates quantification consistency but does not assess downstream inference (EM, VBEM) convergence or posterior variance; abundance point estimates may agree while credible intervals diverge.6869## Evidence7071- [methods] Per-read mapping agreement between C++ 1.12.0 and Rust is 99.83% on byte-identical index: "per-read mapping agreement | 99.83%"72- [methods] On byte-identical index, salmon 2.0 (Rust) and salmon 1.12.0 (C++) achieve per-transcript Pearson correlations of 0.99854 for NumReads and 0.99897 for TPM: "On a byte-identical index, salmon 2.0 (Rust) and salmon 1.12.0 (C++) achieve per-transcript Pearson correlations of 0.99854 for NumReads and 0.99897 for TPM, with per-read mapping agreement at 99.83%."73- [methods] ~80% of mapping gap is attributable to chain-pruning-threshold defaults, not algorithmic divergence: "~80% (~49k): a chain-sub-optimality default difference (describable, deferred). C++ prunes low-coverage chains/orphans before alignment using orphanChainSubThresh = 0.95 and postMergeChainSubThresh ="74- [methods] Workflow specifies byte-identical index construction and paired parameter control: "Build a byte-identical GRCh38 cDNA index using both salmon 2.0 and C++ salmon 1.12.0 from the same cleaned FASTA with deterministic non-ACGT replacement (193,759 transcripts). 2. Run salmon 2.0 quant"75- [readme] salmon 2.0 is a Rust rewrite and must be validated against C++ reference: "This is salmon 2.0 — a from-scratch Rust rewrite of salmon. It keeps the same workflow (`salmon index` → `salmon quant` → `quant.sf`) and the same output formats downstream tools read, but it is a"
HolobiomicsLab