# Dataset Storage Interface Architecture

> Use when you have NMR dataset metadata (cache-file flag, total point count, block layout configuration) and need to select an appropriate storage backend that balances memory efficiency, access patterns, and platform-specific constraints.

- Skill: `holobiomicslab/dataset-storage-interface-architecture-2` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/dataset-storage-interface-architecture-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/dataset-storage-interface-architecture-2/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/dataset-storage-interface-architecture-2

---


# dataset-storage-interface-architecture

> **License: restricted** — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->
## Summary

A conditional selection strategy for choosing among multiple storage backend implementations (SubMatrixFile, BigMappedMatrixFile, MappedSubMatrixFile, MappedMatrixFile) based on dataset metadata, memory constraints, and block layout configuration. This skill optimizes memory-mapped file access and cache performance for large NMR datasets.

## When to use

Apply this skill when you have NMR dataset metadata (cache-file flag, total point count, block layout configuration) and need to select an appropriate storage backend that balances memory efficiency, access patterns, and platform-specific constraints. Use it specifically when dataset size exceeds available heap memory or when multi-block (tiled) layouts are present.

## When NOT to use

- Dataset size is small enough to fit entirely in heap memory and single-block layout applies — use direct in-memory storage instead
- Storage backend is externally mandated by downstream API contract or immutable configuration

## Inputs

- Dataset metadata object (containing cache-file flag, total point count, block layout configuration)
- System platform identifier (Windows vs. other)
- Memory availability constraints

## Outputs

- Selected DatasetStorageInterface implementation class (SubMatrixFile, BigMappedMatrixFile, MappedSubMatrixFile, or MappedMatrixFile)
- Selection justification and performance rationale

## How to apply

First, parse the dataset metadata to extract the cache-file flag, total number of points, and block layout configuration. Then apply conditional logic in sequence: (1) if cache file is enabled in a Windows processing context, select SubMatrixFile; (2) if total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile to enable memory-mapped access; (3) if layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access; (4) otherwise, fallback to MappedMatrixFile for single-block layouts. Document the selection criteria and rationale, especially the performance trade-offs between MappedSubMatrixFile and MappedMatrixFile for borderline single-block cases.

## Related tools

- **NMRFx** (NMR data processing and visualization platform in which Dataset.createDataFile() storage-backend selection logic is embedded) — https://github.com/nanalysis/nmrfx

## Evaluation signals

- Verify that selected storage class matches the decision tree: SubMatrixFile only when cache flag + Windows, BigMappedMatrixFile only when byte count > Integer.MAX_VALUE / 2, MappedSubMatrixFile only when multiple blocks are present, MappedMatrixFile as fallback
- Confirm that memory-mapped implementations (BigMappedMatrixFile, MappedSubMatrixFile, MappedMatrixFile) are chosen for datasets exceeding heap limits
- Check that single-block layouts never select MappedSubMatrixFile (confirm block count == 1 before fallback decision)
- Validate that platform-specific path (SubMatrixFile for Windows cache mode) is taken only when both conditions hold
- Review generated documentation comments address stated performance trade-offs, especially rationale for MappedSubMatrixFile vs. MappedMatrixFile boundary

## Limitations

- Selection logic depends on accurate metadata parsing; malformed or incomplete cache-file flag, point count, or block layout will trigger fallback (MappedMatrixFile) regardless of actual dataset size
- Integer.MAX_VALUE / 2 threshold is architecture-dependent; 32-bit systems may require different thresholds than 64-bit platforms
- Windows-specific SubMatrixFile path assumes cache file is pre-allocated and accessible; failure to stage cache file will degrade performance
- No dynamic re-selection once DatasetStorageInterface is instantiated; suboptimal choice (e.g., MappedMatrixFile for a multi-block layout) persists for the dataset's lifetime

## Evidence

- [other] Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration.: "Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration."
- [other] Implement conditional logic to select SubMatrixFile if cache file is enabled (Windows processing context).: "Implement conditional logic to select SubMatrixFile if cache file is enabled (Windows processing context)."
- [other] If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets.: "If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets."
- [other] If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access.: "If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access."
- [other] Fallback to MappedMatrixFile for single-block layouts.: "Fallback to MappedMatrixFile for single-block layouts."