index-construction-determinism
Summary
Verify that transcript indices built from the same source FASTA with identical parameters produce byte-identical outputs across different tool versions or implementations, ensuring reproducible quantification results. This skill surfaces hidden tool bugs (e.g., k-mer orientation errors) that would otherwise manifest only as small divergences in read-mapping agreement.
When to use
When comparing quantification results between two versions of a tool (e.g., salmon 2.0 Rust rewrite vs. C++ salmon 1.12.0), or when validating a new tool implementation against a reference version. Apply this skill early in validation pipelines to establish a reproducibility baseline before running downstream inference—if indices are not byte-identical, downstream disagreement cannot be attributed solely to algorithmic differences in quantification inference.
When NOT to use
- When comparing tools designed for different data types (e.g., salmon for bulk RNA-seq vs. piscem for single-cell), as index format and parameterization differ by design.
- When the input FASTA has not been consistently cleaned (e.g., non-ACGT characters are replaced non-deterministically), as index divergence will not be interpretable.
- When comparing versions with intentionally different default chain-pruning thresholds or other algorithmic settings, unless those settings are first harmonized; raw disagreement alone does not reveal bugs.
Inputs
- Transcriptome FASTA (cleaned, with deterministic non-ACGT replacement)
- Index parameters (k-mer length, thread count, optional decoy sequences)
- Paired-end or single-end RNA-seq reads (FASTQ)
Outputs
- Byte-identical (or non-identical) index files from both tool versions
- quant.sf quantification tables (NumReads and TPM columns) from both tools
- Per-transcript Pearson correlation coefficients (NumReads and TPM)
- unmapped_names.txt files from both tools
- Per-read mapping agreement percentage
- Total assigned NumReads difference as a percentage
How to apply
Build the same transcriptome index twice using both tool versions with identical parameters (same cleaned FASTA with deterministic non-ACGT replacement, same k-mer length, same threads, same decoys if any). Compare the resulting index files byte-for-byte using checksums or binary diff tools. If indices match, run both tools on the same reads with the same selective-alignment mode parameters and library-type detection settings (e.g., -l A auto-detect). Extract per-transcript NumReads and TPM columns from the quant.sf outputs and compute Pearson correlations; extract unmapped read names and compute the percentage of per-read mapping agreement (focusing on fully-unmapped reads coded 'u'). Correlations ≥0.999 and per-read agreement ≥99.8% on a byte-identical index indicate the tools map and quantify equivalently; larger discrepancies signal algorithmic divergences or bugs. If indices are not byte-identical but quantification correlations remain high, investigate whether the index format changed intentionally or whether a pufferfish/lookup-table bug is present.
Related tools
- salmon 2.0 (Rust rewrite of salmon for index construction and quantification; primary tool being validated) — https://github.com/COMBINE-lab/salmon
- salmon 1.12.0 (C++ reference implementation for byte-identity and quantification agreement comparison) — https://github.com/COMBINE-lab/salmon/tree/cpp
- pufferfish (K-mer lookup engine underlying index construction; bugs (e.g., k-mer-orientation in SSHash streaming lookup) are revealed by index-level validation)
Examples
salmon index -t transcripts.fa -i salmon_index_v2 -k 31 -p 16; salmon index -t transcripts.fa -i salmon_index_cpp -k 31 -p 16; diff <(md5sum salmon_index_v2/*) <(md5sum salmon_index_cpp/*); salmon quant -i salmon_index_v2 -l A -1 reads_1.fq.gz -2 reads_2.fq.gz -o quant_v2; salmon quant -i salmon_index_cpp -l A -1 reads_1.fq.gz -2 reads_2.fq.gz -o quant_cpp; paste quant_v2/quant.sf quant_cpp/quant.sf | awk '{print $1, $5, $10}' | Rscript -e 'data <- read.table("/dev/stdin", header=F); print(cor(data$V2, data$V3, method="pearson"))'
Evaluation signals
- Byte-identity of index files (checksums match or binary diff is empty) when built from the same FASTA with identical parameters
- Per-transcript Pearson correlation ≥0.999 for both NumReads and TPM between the two tools on a byte-identical index
- Per-read mapping agreement ≥99.8% (percentage of reads where both tools agree on mapped/unmapped status)
- Total assigned NumReads difference <1% when quantifying the same dataset with the same selective-alignment mode and library-type parameters
- Reproducibility check: re-running the same commands on the same inputs yields identical quantification outputs (deterministic randomness in seeding or tie-breaking)
Limitations
- Chain-pruning threshold differences (e.g., orphanChainSubThresh and postMergeChainSubThresh) between versions can account for ~80% of per-read mapping disagreement even on byte-identical indices; harmonize these parameters before concluding that disagreement indicates a bug.
- Index format changes (e.g., salmon 2.0 Rust uses a new format) mean byte-identity is not achievable across major versions; in such cases, correlations and per-read agreement become the primary validation metrics rather than checksums.
- Very short reads (<50 bp) or highly repetitive transcriptomes may show larger discrepancies; the validation was demonstrated on 76 bp paired-end reads from a human reference (GRCh38).
- Downstream inference divergence (VBEM vs. EM, bias-correction toggling) will mask index-level agreement, so validation must be performed with the same inference settings across tools.
Evidence
- [other] Build a byte-identical GRCh38 cDNA index using both salmon 2.0 and C++ salmon 1.12.0 from the same cleaned FASTA with deterministic non-ACGT replacement: "Build a byte-identical GRCh38 cDNA index using both salmon 2.0 and C++ salmon 1.12.0 from the same cleaned FASTA with deterministic non-ACGT replacement (193,759 transcripts)."
- [other] On a byte-identical index, salmon 2.0 (Rust) and salmon 1.12.0 (C++) achieve per-transcript Pearson correlations of 0.99854 for NumReads and 0.99897 for TPM, with per-read mapping agreement at 99.83%: "On a byte-identical index, salmon 2.0 (Rust) and salmon 1.12.0 (C++) achieve per-transcript Pearson correlations of 0.99854 for NumReads and 0.99897 for TPM, with per-read mapping agreement at 99.83%."
- [other] After fixing pufferfish (commit 5dce7f4, salmon pin bumped), C++ salmon maps the same reads: 83.48% → 85.55%, matching the Rust port to within 1 read: "After fixing pufferfish (commit 5dce7f4, salmon pin bumped), C++ salmon maps the same reads: 83.48% → 85.55%, matching the Rust port to within 1 read"
- [other] On real short reads the Rust port mapped ~2% more reads than C++ salmon. That extra ~2% was not a Rust 'sensitivity choice' — it exposed a k-mer-orientation bug in pufferfish's SSHash streaming lookup: "On real short reads the Rust port mapped ~2% more reads than C++ salmon. That extra ~2% was not a Rust 'sensitivity choice' — it exposed a k-mer-orientation bug in pufferfish's SSHash streaming lookup"
- [other]
80% (49k): a chain-sub-optimality default difference (describable, deferred). C++ prunes low-coverage chains/orphans before alignment using orphanChainSubThresh = 0.95 and postMergeChainSubThresh =: "80% (49k): a chain-sub-optimality default difference (describable, deferred). C++ prunes low-coverage chains/orphans before alignment using orphanChainSubThresh = 0.95 and postMergeChainSubThresh"
- [intro] salmon 2.0 maintains the same workflow (salmon index → salmon quant → quant.sf) and output formats as the previous version: "It keeps the same workflow (
salmon index → salmon quant → quant.sf) and the same output formats downstream tools read"
- [readme] the index format changed, so you must rebuild your index: "the index format changed, so you must rebuild your index"
1---2name: index-construction-determinism3description: Use when when comparing quantification results between two versions of a tool (e.g., salmon 2.0 Rust rewrite vs. C++ salmon 1.12.0), or when validating a new tool implementation against a reference version.4license: CC-BY-4.05---67# index-construction-determinism89## Summary1011Verify that transcript indices built from the same source FASTA with identical parameters produce byte-identical outputs across different tool versions or implementations, ensuring reproducible quantification results. This skill surfaces hidden tool bugs (e.g., k-mer orientation errors) that would otherwise manifest only as small divergences in read-mapping agreement.1213## When to use1415When comparing quantification results between two versions of a tool (e.g., salmon 2.0 Rust rewrite vs. C++ salmon 1.12.0), or when validating a new tool implementation against a reference version. Apply this skill early in validation pipelines to establish a reproducibility baseline before running downstream inference—if indices are not byte-identical, downstream disagreement cannot be attributed solely to algorithmic differences in quantification inference.1617## When NOT to use1819- When comparing tools designed for different data types (e.g., salmon for bulk RNA-seq vs. piscem for single-cell), as index format and parameterization differ by design.20- When the input FASTA has not been consistently cleaned (e.g., non-ACGT characters are replaced non-deterministically), as index divergence will not be interpretable.21- When comparing versions with intentionally different default chain-pruning thresholds or other algorithmic settings, unless those settings are first harmonized; raw disagreement alone does not reveal bugs.2223## Inputs2425- Transcriptome FASTA (cleaned, with deterministic non-ACGT replacement)26- Index parameters (k-mer length, thread count, optional decoy sequences)27- Paired-end or single-end RNA-seq reads (FASTQ)2829## Outputs3031- Byte-identical (or non-identical) index files from both tool versions32- quant.sf quantification tables (NumReads and TPM columns) from both tools33- Per-transcript Pearson correlation coefficients (NumReads and TPM)34- unmapped_names.txt files from both tools35- Per-read mapping agreement percentage36- Total assigned NumReads difference as a percentage3738## How to apply3940Build the same transcriptome index twice using both tool versions with identical parameters (same cleaned FASTA with deterministic non-ACGT replacement, same k-mer length, same threads, same decoys if any). Compare the resulting index files byte-for-byte using checksums or binary diff tools. If indices match, run both tools on the same reads with the same selective-alignment mode parameters and library-type detection settings (e.g., `-l A` auto-detect). Extract per-transcript NumReads and TPM columns from the quant.sf outputs and compute Pearson correlations; extract unmapped read names and compute the percentage of per-read mapping agreement (focusing on fully-unmapped reads coded 'u'). Correlations ≥0.999 and per-read agreement ≥99.8% on a byte-identical index indicate the tools map and quantify equivalently; larger discrepancies signal algorithmic divergences or bugs. If indices are not byte-identical but quantification correlations remain high, investigate whether the index format changed intentionally or whether a pufferfish/lookup-table bug is present.4142## Related tools4344- **salmon 2.0** (Rust rewrite of salmon for index construction and quantification; primary tool being validated) — https://github.com/COMBINE-lab/salmon45- **salmon 1.12.0** (C++ reference implementation for byte-identity and quantification agreement comparison) — https://github.com/COMBINE-lab/salmon/tree/cpp46- **pufferfish** (K-mer lookup engine underlying index construction; bugs (e.g., k-mer-orientation in SSHash streaming lookup) are revealed by index-level validation)4748## Examples4950```51salmon index -t transcripts.fa -i salmon_index_v2 -k 31 -p 16; salmon index -t transcripts.fa -i salmon_index_cpp -k 31 -p 16; diff <(md5sum salmon_index_v2/*) <(md5sum salmon_index_cpp/*); salmon quant -i salmon_index_v2 -l A -1 reads_1.fq.gz -2 reads_2.fq.gz -o quant_v2; salmon quant -i salmon_index_cpp -l A -1 reads_1.fq.gz -2 reads_2.fq.gz -o quant_cpp; paste quant_v2/quant.sf quant_cpp/quant.sf | awk '{print $1, $5, $10}' | Rscript -e 'data <- read.table("/dev/stdin", header=F); print(cor(data$V2, data$V3, method="pearson"))'52```5354## Evaluation signals5556- Byte-identity of index files (checksums match or binary diff is empty) when built from the same FASTA with identical parameters57- Per-transcript Pearson correlation ≥0.999 for both NumReads and TPM between the two tools on a byte-identical index58- Per-read mapping agreement ≥99.8% (percentage of reads where both tools agree on mapped/unmapped status)59- Total assigned NumReads difference <1% when quantifying the same dataset with the same selective-alignment mode and library-type parameters60- Reproducibility check: re-running the same commands on the same inputs yields identical quantification outputs (deterministic randomness in seeding or tie-breaking)6162## Limitations6364- Chain-pruning threshold differences (e.g., orphanChainSubThresh and postMergeChainSubThresh) between versions can account for ~80% of per-read mapping disagreement even on byte-identical indices; harmonize these parameters before concluding that disagreement indicates a bug.65- Index format changes (e.g., salmon 2.0 Rust uses a new format) mean byte-identity is not achievable across major versions; in such cases, correlations and per-read agreement become the primary validation metrics rather than checksums.66- Very short reads (<50 bp) or highly repetitive transcriptomes may show larger discrepancies; the validation was demonstrated on 76 bp paired-end reads from a human reference (GRCh38).67- Downstream inference divergence (VBEM vs. EM, bias-correction toggling) will mask index-level agreement, so validation must be performed with the same inference settings across tools.6869## Evidence7071- [other] Build a byte-identical GRCh38 cDNA index using both salmon 2.0 and C++ salmon 1.12.0 from the same cleaned FASTA with deterministic non-ACGT replacement: "Build a byte-identical GRCh38 cDNA index using both salmon 2.0 and C++ salmon 1.12.0 from the same cleaned FASTA with deterministic non-ACGT replacement (193,759 transcripts)."72- [other] On a byte-identical index, salmon 2.0 (Rust) and salmon 1.12.0 (C++) achieve per-transcript Pearson correlations of 0.99854 for NumReads and 0.99897 for TPM, with per-read mapping agreement at 99.83%: "On a byte-identical index, salmon 2.0 (Rust) and salmon 1.12.0 (C++) achieve per-transcript Pearson correlations of 0.99854 for NumReads and 0.99897 for TPM, with per-read mapping agreement at 99.83%."73- [other] After fixing pufferfish (commit 5dce7f4, salmon pin bumped), C++ salmon maps the same reads: 83.48% → 85.55%, matching the Rust port to within 1 read: "After fixing pufferfish (commit 5dce7f4, salmon pin bumped), C++ salmon maps the same reads: 83.48% → 85.55%, matching the Rust port to within 1 read"74- [other] On real short reads the Rust port mapped ~2% more reads than C++ salmon. That extra ~2% was not a Rust 'sensitivity choice' — it exposed a k-mer-orientation bug in pufferfish's SSHash streaming lookup: "On real short reads the Rust port mapped ~2% more reads than C++ salmon. That extra ~2% was not a Rust 'sensitivity choice' — it exposed a k-mer-orientation bug in pufferfish's SSHash streaming lookup"75- [other] ~80% (~49k): a chain-sub-optimality default difference (describable, deferred). C++ prunes low-coverage chains/orphans before alignment using orphanChainSubThresh = 0.95 and postMergeChainSubThresh =: "~80% (~49k): a chain-sub-optimality default difference (describable, deferred). C++ prunes low-coverage chains/orphans before alignment using orphanChainSubThresh = 0.95 and postMergeChainSubThresh"76- [intro] salmon 2.0 maintains the same workflow (salmon index → salmon quant → quant.sf) and output formats as the previous version: "It keeps the same workflow (`salmon index` → `salmon quant` → `quant.sf`) and the same output formats downstream tools read"77- [readme] the index format changed, so you must rebuild your index: "the index format changed, so you must rebuild your index"
HolobiomicsLab