# Dataset Integrity Assessment

> Use when when you have downloaded a released version of a structured dataset (e.g., LOTUS from Zenodo) and need to confirm it matches the documented headline statistics before downstream analysis, or when auditing data integrity after ingestion into a processing pipeline.

- Skill: `holobiomicslab/dataset-integrity-assessment` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/dataset-integrity-assessment`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/dataset-integrity-assessment/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/dataset-integrity-assessment

---


# dataset-integrity-assessment

## Summary

Systematically verify the completeness and accuracy of a curated structure-organism dataset by reproducing reported aggregate statistics (unique pairs, structures, organisms, sources) and comparing observed counts against published benchmarks using absolute and relative difference metrics.

## When to use

When you have downloaded a released version of a structured dataset (e.g., LOTUS from Zenodo) and need to confirm it matches the documented headline statistics before downstream analysis, or when auditing data integrity after ingestion into a processing pipeline.

## When NOT to use

- When comparing counts across different dataset versions or snapshots (use change-tracking instead of direct reproduction).
- When the input files are raw (non-curated) database exports prior to standardization and integration.
- When only a sample or subset of the dataset is available (e.g., test mode); request full dataset release or document the subset scope.

## Inputs

- 3D structure-organism pair TSV/CSV file with structure ID and organism ID columns
- 2D structure-organism pair TSV/CSV file with structure ID and organism ID columns
- Dataset metadata or source attribution file listing database provenance

## Outputs

- Verification table with observed counts, reported counts, absolute differences, and relative differences for all aggregate statistics
- Boolean integrity report indicating pass/fail for each count category

## How to apply

Load the structure-organism pair tables (3D and 2D TSV/CSV files) into R or Python. Group by structure identifier and organism identifier columns to enumerate unique referenced (raw) structure-organism pairs separately for 3D and 2D subsets. Deduplicate the structure ID field across all records to count unique curated structures per dimension. Count unique organism identifiers across all structure-organism records regardless of dimension. Enumerate source database identifiers in the metadata or source attribution columns to verify the documented database count. Compile all counts into a verification table with columns for observed value, reported value, absolute difference, and relative difference (%). Documented targets for LOTUS are: 588,694 (3D) and 484,174 (2D) unique referenced structure-organism pairs; 231,330 (3D) and 153,956 (2D) unique curated structures; 42,166 unique organisms; and 31 initial source databases.

## Related tools

- **R** (Counting and deduplication of structure, organism, and source identifiers; verification table generation) — https://github.com/lotusnprod/lotus-processor
- **Python 3** (Data loading, grouping, and statistical comparison of observed versus reported counts) — https://github.com/lotusnprod/lotus-processor

## Examples

```
# R example: Count unique pairs in 3D structure-organism file
lotus_3d <- read.csv('lotus_3d_pairs.tsv', sep='\t')
unique_pairs_3d <- nrow(unique(lotus_3d[, c('structure_id', 'organism_id')]))
unique_structures_3d <- length(unique(lotus_3d$structure_id))
cat('3D pairs:', unique_pairs_3d, '| Expected: 588694 | Diff:', 588694 - unique_pairs_3d)
```

## Evaluation signals

- Observed count of 3D unique referenced structure-organism pairs matches or is within <1% relative difference of 588,694.
- Observed count of 2D unique referenced structure-organism pairs matches or is within <1% relative difference of 484,174.
- Observed unique curated structures (3D and 2D combined or separately) align with reported counts (231,330 for 3D, 153,956 for 2D) within <1% relative difference.
- Observed unique organism count matches or is within <1% relative difference of 42,166.
- Enumerated source database identifiers total 31 or document any discrepancy with explanation (e.g., database mergers, removals).

## Limitations

- This skill reproduces published aggregate statistics only; it does not validate the semantic correctness or taxonomic accuracy of individual organism assignments or structure records.
- Relative difference tolerance thresholds (e.g., <1%) are not explicitly stated in the source material and should be set according to the dataset's version and release notes.
- Source database count verification depends on the presence and format of provenance metadata in the dataset; missing or inconsistently annotated source fields will prevent accurate enumeration.
- The skill assumes TSV/CSV input format; other serializations (JSON, RDF, SQL dumps) require prior format conversion.

## Evidence

- [other] 588,694 (3D) and 484,174 (2D) unique referenced structure-organism pairs, 231,330 (3D) and 153,956 (2D) unique curated structures, 42,166 unique organisms, and 31 source databases: "The EnrichedIndex reports LOTUS contains 588,694 (3D) and 484,174 (2D) unique referenced structure-organism pairs, 231,330 (3D) and 153,956 (2D) unique curated structures, 42,166 unique organisms,"
- [other] Count unique referenced (raw) structure-organism pairs by grouping on structure identifier and organism identifier columns: "Load the 3D and 2D structure-organism pair files using R or Python and count unique referenced (raw) structure-organism pairs by grouping on structure identifier and organism identifier columns."
- [other] Count unique curated structure identifiers and enumerate source database identifiers: "Count unique curated structure identifiers across the 3D and 2D datasets separately by deduplicating the structure ID field. Count unique organism identifiers across all structure-organism records to"
- [other] Compile counts into verification table with absolute and relative difference columns: "Compile all counts into a verification table and compare against the reported values with absolute and relative difference columns."
- [readme] Download the LOTUS dataset deposit from Zenodo: "The data used to support the findings of this study have been deposited on Zenodo. A snapshot of the repository at the time of publication is also available under the same link."