block-layout-analysis-for-io-optimization
Summary
Analyzes dataset block layout configuration to select the optimal memory-mapped storage backend (MappedMatrixFile, MappedSubMatrixFile, BigMappedMatrixFile, or SubMatrixFile) for efficient I/O and memory access patterns. This skill balances memory constraints, file size, and tiling strategy to minimize latency and memory pressure during large dataset operations.
When to use
When preparing NMR datasets for processing in NMRFx and the Dataset.createDataFile() method must choose among competing storage backends. Specifically, apply this skill when you have parsed dataset metadata that includes: cache-file flag status, total number of data points, and block layout configuration (single-block vs. multi-block/sub-matrix). The decision is necessary before committing data to disk, especially for datasets where memory-mapped access would improve performance.
When NOT to use
- Dataset is already in-memory or use-once (streaming); block-layout analysis is only valuable for datasets requiring repeated random access or where memory pressure is a constraint.
- Underlying storage backend is already fixed by upstream processing rules or API contracts; this skill assumes selection flexibility at the createDataFile() point.
- Block layout metadata is absent, incomplete, or unreliable; the cascading logic depends on accurate cache-file flags and size estimates.
Inputs
- Dataset metadata object (cache-file flag, total point count)
- Block layout configuration (single-block indicator or multi-block/sub-matrix structure)
- Dataset size in bytes
- Processing context (Windows or other OS)
Outputs
- Selected DatasetStorageInterface implementation class (SubMatrixFile, BigMappedMatrixFile, MappedSubMatrixFile, or MappedMatrixFile)
- Instantiated file handle or mapped region for downstream I/O operations
- Decision log documenting selection rationale and performance trade-offs
How to apply
Parse the dataset metadata to extract three key parameters: (1) cache-file flag (Windows context indicator), (2) total dataset size in bytes relative to Integer.MAX_VALUE / 2 threshold, and (3) block layout structure. Apply the selection logic in cascading order: First, if cache-file flag is enabled in a Windows processing context, select SubMatrixFile for compatibility. Second, if the dataset size in bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile to enable 64-bit memory-mapped access and avoid overflow. Third, if the layout configuration defines multiple blocks (sub-matrix case), select MappedSubMatrixFile to leverage efficient tiled access patterns. Finally, for single-block layouts, fall back to MappedMatrixFile. Document the selection decision and add inline comments addressing performance trade-offs, especially for single-block cases where MappedSubMatrixFile may offer marginal tiling benefits despite added complexity.
Related tools
- NMRFx (Runtime environment for dataset creation and storage backend instantiation; provides Dataset class and DatasetStorageInterface implementations) — https://github.com/nanalysis/nmrfx
Evaluation signals
- Verify selection logic is deterministic: given identical metadata inputs, the same backend class is chosen every time.
- Confirm size threshold comparisons are correct: datasets with byte count ≤ Integer.MAX_VALUE / 2 do not route to BigMappedMatrixFile; those exceeding the threshold do.
- Validate block-count detection: single-block layouts select MappedMatrixFile; multi-block layouts select MappedSubMatrixFile (or higher-precedence backends if cache or size constraints apply).
- Audit decision log for completeness: each storage backend selection includes rationale and, where applicable, performance trade-off commentary.
- Test Windows vs. non-Windows branching: cache-file flag in Windows context consistently triggers SubMatrixFile selection before size/block checks.
Limitations
- Integer.MAX_VALUE / 2 threshold is a hard boundary; datasets near this limit may exhibit marginal performance differences depending on actual access patterns, not predicted by this rule alone.
- Single-block vs. multi-block classification depends on accurate block layout metadata; corrupted or missing layout configuration will cause misclassification.
- MappedSubMatrixFile vs. MappedMatrixFile trade-off for single-block datasets is noted as requiring documentation but is not resolved by this skill; practitioner judgment needed for performance-critical scenarios.
- Windows-specific cache-file handling assumes OS-level memory management differs; no validation of actual OS or file system type is performed.
- No dynamic re-selection: once a backend is instantiated, changes to dataset characteristics (e.g., data is added post-creation) are not automatically re-evaluated.
Evidence
- [other] Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration.: "Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration."
- [other] Select SubMatrixFile if cache file is enabled (Windows processing context).: "Select SubMatrixFile if cache file is enabled (Windows processing context)."
- [other] If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets.: "If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets."
- [other] If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access.: "If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access."
- [other] Document selection criteria and add comments addressing the performance trade-offs between MappedSubMatrixFile and MappedMatrixFile for single-block cases.: "Document selection criteria and add comments addressing the performance trade-offs between MappedSubMatrixFile and MappedMatrixFile for single-block cases."
1---2name: block-layout-analysis-for-io-optimization3description: Use when when preparing NMR datasets for processing in NMRFx and the Dataset.createDataFile() method must choose among competing storage backends.4license: CC-BY-4.05---67# block-layout-analysis-for-io-optimization89## Summary1011Analyzes dataset block layout configuration to select the optimal memory-mapped storage backend (MappedMatrixFile, MappedSubMatrixFile, BigMappedMatrixFile, or SubMatrixFile) for efficient I/O and memory access patterns. This skill balances memory constraints, file size, and tiling strategy to minimize latency and memory pressure during large dataset operations.1213## When to use1415When preparing NMR datasets for processing in NMRFx and the Dataset.createDataFile() method must choose among competing storage backends. Specifically, apply this skill when you have parsed dataset metadata that includes: cache-file flag status, total number of data points, and block layout configuration (single-block vs. multi-block/sub-matrix). The decision is necessary before committing data to disk, especially for datasets where memory-mapped access would improve performance.1617## When NOT to use1819- Dataset is already in-memory or use-once (streaming); block-layout analysis is only valuable for datasets requiring repeated random access or where memory pressure is a constraint.20- Underlying storage backend is already fixed by upstream processing rules or API contracts; this skill assumes selection flexibility at the createDataFile() point.21- Block layout metadata is absent, incomplete, or unreliable; the cascading logic depends on accurate cache-file flags and size estimates.2223## Inputs2425- Dataset metadata object (cache-file flag, total point count)26- Block layout configuration (single-block indicator or multi-block/sub-matrix structure)27- Dataset size in bytes28- Processing context (Windows or other OS)2930## Outputs3132- Selected DatasetStorageInterface implementation class (SubMatrixFile, BigMappedMatrixFile, MappedSubMatrixFile, or MappedMatrixFile)33- Instantiated file handle or mapped region for downstream I/O operations34- Decision log documenting selection rationale and performance trade-offs3536## How to apply3738Parse the dataset metadata to extract three key parameters: (1) cache-file flag (Windows context indicator), (2) total dataset size in bytes relative to Integer.MAX_VALUE / 2 threshold, and (3) block layout structure. Apply the selection logic in cascading order: First, if cache-file flag is enabled in a Windows processing context, select SubMatrixFile for compatibility. Second, if the dataset size in bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile to enable 64-bit memory-mapped access and avoid overflow. Third, if the layout configuration defines multiple blocks (sub-matrix case), select MappedSubMatrixFile to leverage efficient tiled access patterns. Finally, for single-block layouts, fall back to MappedMatrixFile. Document the selection decision and add inline comments addressing performance trade-offs, especially for single-block cases where MappedSubMatrixFile may offer marginal tiling benefits despite added complexity.3940## Related tools4142- **NMRFx** (Runtime environment for dataset creation and storage backend instantiation; provides Dataset class and DatasetStorageInterface implementations) — https://github.com/nanalysis/nmrfx4344## Evaluation signals4546- Verify selection logic is deterministic: given identical metadata inputs, the same backend class is chosen every time.47- Confirm size threshold comparisons are correct: datasets with byte count ≤ Integer.MAX_VALUE / 2 do not route to BigMappedMatrixFile; those exceeding the threshold do.48- Validate block-count detection: single-block layouts select MappedMatrixFile; multi-block layouts select MappedSubMatrixFile (or higher-precedence backends if cache or size constraints apply).49- Audit decision log for completeness: each storage backend selection includes rationale and, where applicable, performance trade-off commentary.50- Test Windows vs. non-Windows branching: cache-file flag in Windows context consistently triggers SubMatrixFile selection before size/block checks.5152## Limitations5354- Integer.MAX_VALUE / 2 threshold is a hard boundary; datasets near this limit may exhibit marginal performance differences depending on actual access patterns, not predicted by this rule alone.55- Single-block vs. multi-block classification depends on accurate block layout metadata; corrupted or missing layout configuration will cause misclassification.56- MappedSubMatrixFile vs. MappedMatrixFile trade-off for single-block datasets is noted as requiring documentation but is not resolved by this skill; practitioner judgment needed for performance-critical scenarios.57- Windows-specific cache-file handling assumes OS-level memory management differs; no validation of actual OS or file system type is performed.58- No dynamic re-selection: once a backend is instantiated, changes to dataset characteristics (e.g., data is added post-creation) are not automatically re-evaluated.5960## Evidence6162- [other] Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration.: "Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration."63- [other] Select SubMatrixFile if cache file is enabled (Windows processing context).: "Select SubMatrixFile if cache file is enabled (Windows processing context)."64- [other] If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets.: "If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets."65- [other] If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access.: "If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access."66- [other] Document selection criteria and add comments addressing the performance trade-offs between MappedSubMatrixFile and MappedMatrixFile for single-block cases.: "Document selection criteria and add comments addressing the performance trade-offs between MappedSubMatrixFile and MappedMatrixFile for single-block cases."
HolobiomicsLab