Kosh

Kosh Skill

Use this skill when an agent needs to store results, attach files, query them later, or build lightweight data-processing hooks around files and datasets.

Common Patterns

• Create or open a store with kosh.connect(...) or kosh.utils.create_new_db(...).
• Create a dataset with store.create(...), then store metadata and attach files with dataset.associate(uri, mime_type, metadata=...).
• Create an ensemble with store.create_ensemble(...) when a group of datasets should be searched together.
• Query later with store.find(...) for datasets, store.find_ensembles(...) for ensembles, dataset.find(...) for associated sources, ensemble.find_datasets(...) for member datasets, plus dataset.list_features() and dataset.describe_feature(...).
• Read results with dataset.get(feature, transformers=[...]) or dataset.get_execution_graph(feature, transformers=[...]).

Storing Results

• Use datasets to hold metadata about a run, a job, or a generated artifact.
• Associate output files directly to the dataset so they can be rediscovered by feature name or metadata.
• Use ensembles to collect related runs and add ensemble-level metadata for the group.
• Keep a given attribute in one place: put dataset-specific values on the dataset and group-level values on the ensemble.
• Use ensemble_tags for labels that belong to an ensemble membership, not the dataset itself. These tags are searchable with ensemble.find_datasets(ensemble_tags=...) and can be inspected with dataset.list_ensemble_tags(...).
• Example: ensemble_tags={"even_or_odd": "even", "data_type": "test data"}.

Workflow Tracking

• Use the workflow CLI (kosh_workflow) when you want to create or update datasets from a step plus parameter set.
• Use kosh.parameter_store.StepRequest and kosh.parameter_store.apply_step(...) from Python when scripting workflow updates.
• Treat workflow_step as the default step field unless a different field name is configured with --step-field.
• Use --ensemble to scope workflow operations to one ensemble, --param key=value or implicit --key=value filters to match init parameters, and --upsert-init when the init step should reuse an existing dataset instead of creating a duplicate.
• Use --meta for dataset metadata updates and --associate for files that belong on the dataset; use --emeta/--ensemble-meta and --associate-ensemble for ensemble-level updates.

Parsing Output Files

• Add a custom loader when Kosh needs to parse a file format that is not already supported or when the file should expose custom features.
• Implement list_features() and extract() on a KoshLoader.
• A loader can expose multiple features and return data in different formats, so use it when the raw file needs parsing or format-specific access.
• Register the loader with store.add_loader(MyLoader) before querying associated files that depend on it.
• For loaders that support direct indexing, implement __getitem__ so slices can be served efficiently.

Feature Aliases

• Use alias_feature when different sources expose the same data under different names, such as uppercase CSV headers versus lowercase HDF5 datasets.
• Map aliases both ways when needed, for example {"col_a": "COL_A"} on one dataset and {"COL_A": "col_a"} on the other.
• Aliases let dataset["col_a"] or dataset["COL_A"] resolve to the same underlying feature when the exact name is not present.

Transformers and Operators

• Use KoshTransformer or @kosh.numpy_transformer / @kosh.typed_transformer to reshape, switch formats, or post-process one input stream.
• Use KoshOperator or @kosh.numpy_operator / @kosh.typed_operator to combine multiple inputs, including data loaded from different sources or formats.
• Keep transforms small and composable; use them to normalize or convert results after loading, not to replace the loader’s parsing job.

Slicing and Propagation

• Kosh uses __getitem__ for indexing.
• Implement __getitem_propagate__(self, key, input_index) in transformers or operators when a slice should be pushed back toward the loader.
• If propagation is not possible, return None and let Kosh apply the slice after the upstream step runs.

Notebook References

• Store creation and metadata: examples/Example_00_Open_Store_And_Add_Datasets.ipynb, Example_03_Working_with_Datasets.ipynb
• Reading data and feature access: examples/Example_02_Read_Data.ipynb, Example_column_based_text_files.ipynb
• Ensembles and workflow tracking: examples/Example_Ensembles.ipynb, Example_Simulation_Workflow.ipynb, Example_Workflow_Manager.ipynb
• Custom loaders: examples/Example_Custom_Loader.ipynb, Example_MNIST.ipynb, Example_Sidre.ipynb
• Transformers and operators: examples/Example_05a_Transformers.ipynb, Example_05b_Transformers-SKL.ipynb, Example_06_Operators.ipynb
• Advanced slicing and propagation: examples/Example_Advanced_Data_Slicing.ipynb
• Data movement and interoperability: examples/Example_07_Transferring_Datasets.ipynb, Example_Moving_Datasets.ipynb, Kosh_and_Sina_Interoperability.ipynb

Practical Notes

• Prefer attaching metadata to the dataset or ensemble instead of inventing a separate bookkeeping layer.
• Use existing file-backed loaders when possible; add a custom loader only when the built-ins do not expose the data cleanly.
• Keep notebook-style examples aligned with the existing Example_*.ipynb naming pattern.