dataset-storage-interface-architecture
Summary
A conditional selection strategy for choosing among multiple storage backend implementations (SubMatrixFile, BigMappedMatrixFile, MappedSubMatrixFile, MappedMatrixFile) based on dataset metadata, memory constraints, and block layout configuration. This skill optimizes memory-mapped file access and cache performance for large NMR datasets.
When to use
Apply this skill when you have NMR dataset metadata (cache-file flag, total point count, block layout configuration) and need to select an appropriate storage backend that balances memory efficiency, access patterns, and platform-specific constraints. Use it specifically when dataset size exceeds available heap memory or when multi-block (tiled) layouts are present.
When NOT to use
- Dataset size is small enough to fit entirely in heap memory and single-block layout applies — use direct in-memory storage instead
- Storage backend is externally mandated by downstream API contract or immutable configuration
Inputs
- Dataset metadata object (containing cache-file flag, total point count, block layout configuration)
- System platform identifier (Windows vs. other)
- Memory availability constraints
Outputs
- Selected DatasetStorageInterface implementation class (SubMatrixFile, BigMappedMatrixFile, MappedSubMatrixFile, or MappedMatrixFile)
- Selection justification and performance rationale
How to apply
First, parse the dataset metadata to extract the cache-file flag, total number of points, and block layout configuration. Then apply conditional logic in sequence: (1) if cache file is enabled in a Windows processing context, select SubMatrixFile; (2) if total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile to enable memory-mapped access; (3) if layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access; (4) otherwise, fallback to MappedMatrixFile for single-block layouts. Document the selection criteria and rationale, especially the performance trade-offs between MappedSubMatrixFile and MappedMatrixFile for borderline single-block cases.
Related tools
Evaluation signals
- Verify that selected storage class matches the decision tree: SubMatrixFile only when cache flag + Windows, BigMappedMatrixFile only when byte count > Integer.MAX_VALUE / 2, MappedSubMatrixFile only when multiple blocks are present, MappedMatrixFile as fallback
- Confirm that memory-mapped implementations (BigMappedMatrixFile, MappedSubMatrixFile, MappedMatrixFile) are chosen for datasets exceeding heap limits
- Check that single-block layouts never select MappedSubMatrixFile (confirm block count == 1 before fallback decision)
- Validate that platform-specific path (SubMatrixFile for Windows cache mode) is taken only when both conditions hold
- Review generated documentation comments address stated performance trade-offs, especially rationale for MappedSubMatrixFile vs. MappedMatrixFile boundary
Limitations
- Selection logic depends on accurate metadata parsing; malformed or incomplete cache-file flag, point count, or block layout will trigger fallback (MappedMatrixFile) regardless of actual dataset size
- Integer.MAX_VALUE / 2 threshold is architecture-dependent; 32-bit systems may require different thresholds than 64-bit platforms
- Windows-specific SubMatrixFile path assumes cache file is pre-allocated and accessible; failure to stage cache file will degrade performance
- No dynamic re-selection once DatasetStorageInterface is instantiated; suboptimal choice (e.g., MappedMatrixFile for a multi-block layout) persists for the dataset's lifetime
Evidence
- [other] Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration.: "Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration."
- [other] Implement conditional logic to select SubMatrixFile if cache file is enabled (Windows processing context).: "Implement conditional logic to select SubMatrixFile if cache file is enabled (Windows processing context)."
- [other] If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets.: "If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets."
- [other] If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access.: "If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access."
- [other] Fallback to MappedMatrixFile for single-block layouts.: "Fallback to MappedMatrixFile for single-block layouts."
1---2name: dataset-storage-interface-architecture3description: Use when you have NMR dataset metadata (cache-file flag, total point count, block layout configuration) and need to select an appropriate storage backend that balances memory efficiency, access patterns, and platform-specific constraints.4license: CC-BY-4.05---67# dataset-storage-interface-architecture89## Summary1011A conditional selection strategy for choosing among multiple storage backend implementations (SubMatrixFile, BigMappedMatrixFile, MappedSubMatrixFile, MappedMatrixFile) based on dataset metadata, memory constraints, and block layout configuration. This skill optimizes memory-mapped file access and cache performance for large NMR datasets.1213## When to use1415Apply this skill when you have NMR dataset metadata (cache-file flag, total point count, block layout configuration) and need to select an appropriate storage backend that balances memory efficiency, access patterns, and platform-specific constraints. Use it specifically when dataset size exceeds available heap memory or when multi-block (tiled) layouts are present.1617## When NOT to use1819- Dataset size is small enough to fit entirely in heap memory and single-block layout applies — use direct in-memory storage instead20- Storage backend is externally mandated by downstream API contract or immutable configuration2122## Inputs2324- Dataset metadata object (containing cache-file flag, total point count, block layout configuration)25- System platform identifier (Windows vs. other)26- Memory availability constraints2728## Outputs2930- Selected DatasetStorageInterface implementation class (SubMatrixFile, BigMappedMatrixFile, MappedSubMatrixFile, or MappedMatrixFile)31- Selection justification and performance rationale3233## How to apply3435First, parse the dataset metadata to extract the cache-file flag, total number of points, and block layout configuration. Then apply conditional logic in sequence: (1) if cache file is enabled in a Windows processing context, select SubMatrixFile; (2) if total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile to enable memory-mapped access; (3) if layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access; (4) otherwise, fallback to MappedMatrixFile for single-block layouts. Document the selection criteria and rationale, especially the performance trade-offs between MappedSubMatrixFile and MappedMatrixFile for borderline single-block cases.3637## Related tools3839- **NMRFx** (NMR data processing and visualization platform in which Dataset.createDataFile() storage-backend selection logic is embedded) — https://github.com/nanalysis/nmrfx4041## Evaluation signals4243- Verify that selected storage class matches the decision tree: SubMatrixFile only when cache flag + Windows, BigMappedMatrixFile only when byte count > Integer.MAX_VALUE / 2, MappedSubMatrixFile only when multiple blocks are present, MappedMatrixFile as fallback44- Confirm that memory-mapped implementations (BigMappedMatrixFile, MappedSubMatrixFile, MappedMatrixFile) are chosen for datasets exceeding heap limits45- Check that single-block layouts never select MappedSubMatrixFile (confirm block count == 1 before fallback decision)46- Validate that platform-specific path (SubMatrixFile for Windows cache mode) is taken only when both conditions hold47- Review generated documentation comments address stated performance trade-offs, especially rationale for MappedSubMatrixFile vs. MappedMatrixFile boundary4849## Limitations5051- Selection logic depends on accurate metadata parsing; malformed or incomplete cache-file flag, point count, or block layout will trigger fallback (MappedMatrixFile) regardless of actual dataset size52- Integer.MAX_VALUE / 2 threshold is architecture-dependent; 32-bit systems may require different thresholds than 64-bit platforms53- Windows-specific SubMatrixFile path assumes cache file is pre-allocated and accessible; failure to stage cache file will degrade performance54- No dynamic re-selection once DatasetStorageInterface is instantiated; suboptimal choice (e.g., MappedMatrixFile for a multi-block layout) persists for the dataset's lifetime5556## Evidence5758- [other] Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration.: "Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration."59- [other] Implement conditional logic to select SubMatrixFile if cache file is enabled (Windows processing context).: "Implement conditional logic to select SubMatrixFile if cache file is enabled (Windows processing context)."60- [other] If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets.: "If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets."61- [other] If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access.: "If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access."62- [other] Fallback to MappedMatrixFile for single-block layouts.: "Fallback to MappedMatrixFile for single-block layouts."
HolobiomicsLab 