Datasets
A data processing toolkit for ingesting, transforming, querying, and managing dataset entries from the command line. All operations are logged with timestamps and stored locally.
Commands
Data Operations
Each data command works in two modes: run without arguments to view recent entries, or pass input to record a new entry.
| Command |
Description |
datasets ingest <input> |
Ingest data — record a new ingest entry or view recent ones |
datasets transform <input> |
Transform data — record a transformation or view recent ones |
datasets query <input> |
Query data — record a query or view recent ones |
datasets filter <input> |
Filter data — record a filter operation or view recent ones |
datasets aggregate <input> |
Aggregate data — record an aggregation or view recent ones |
datasets visualize <input> |
Visualize data — record a visualization or view recent ones |
datasets export <input> |
Export data — record an export entry or view recent ones |
datasets sample <input> |
Sample data — record a sample or view recent ones |
datasets schema <input> |
Schema management — record a schema entry or view recent ones |
datasets validate <input> |
Validate data — record a validation or view recent ones |
datasets pipeline <input> |
Pipeline management — record a pipeline step or view recent ones |
datasets profile <input> |
Profile data — record a profile or view recent ones |
Utility Commands
| Command |
Description |
datasets stats |
Show summary statistics — entry counts per category, total entries, disk usage |
datasets export <fmt> |
Export all data to a file (formats: json, csv, txt) |
datasets search <term> |
Search all log files for a term (case-insensitive) |
datasets recent |
Show last 20 entries from activity history |
datasets status |
Health check — version, data directory, entry count, disk usage, last activity |
datasets help |
Show available commands |
datasets version |
Show version (v2.0.0) |
Data Storage
All data is stored locally at ~/.local/share/datasets/:
- Each data command writes to its own log file (e.g.,
ingest.log, transform.log)
- Entries are stored as
timestamp|value pairs (pipe-delimited)
- All actions are tracked in
history.log with timestamps
- Export generates files in the data directory (
export.json, export.csv, or export.txt)
Requirements
- Bash (with
set -euo pipefail)
- Standard Unix utilities:
date, wc, du, grep, tail, cat, sed
- No external dependencies or API keys required
When to Use
- To log and track data processing operations (ingest, transform, query, etc.)
- To maintain a searchable history of data pipeline activities
- To export accumulated records in JSON, CSV, or plain text format
- As part of larger automation or data-pipeline workflows
- When you need a lightweight, local-only dataset operation tracker
Examples
# Record a new ingest entry
datasets ingest "loaded training_data.csv 10000 rows"
# View recent transform entries
datasets transform
# Record a query
datasets query "filter by date > 2026-01-01"
# Search across all logs
datasets search "training"
# Export everything as JSON
datasets export json
# Check overall statistics
datasets stats
# View recent activity
datasets recent
# Health check
datasets status
Powered by BytesAgain | bytesagain.com | hello@bytesagain.com
💬 Feedback & Feature Requests: https://bytesagain.com/feedback
1---2name: datasets3description: Browse and load ready-to-use AI/ML datasets with fast manipulation. Use when searching datasets, loading training data, transforming formats.4license: Apache-2.05---6
7# Datasets
8
9A data processing toolkit for ingesting, transforming, querying, and managing dataset entries from the command line. All operations are logged with timestamps and stored locally.
10
11## Commands
12
13### Data Operations
14
15Each data command works in two modes: run without arguments to view recent entries, or pass input to record a new entry.
16
17| Command | Description |
18|---------|-------------|
19| `datasets ingest <input>` | Ingest data — record a new ingest entry or view recent ones |
20| `datasets transform <input>` | Transform data — record a transformation or view recent ones |
21| `datasets query <input>` | Query data — record a query or view recent ones |
22| `datasets filter <input>` | Filter data — record a filter operation or view recent ones |
23| `datasets aggregate <input>` | Aggregate data — record an aggregation or view recent ones |
24| `datasets visualize <input>` | Visualize data — record a visualization or view recent ones |
25| `datasets export <input>` | Export data — record an export entry or view recent ones |
26| `datasets sample <input>` | Sample data — record a sample or view recent ones |
27| `datasets schema <input>` | Schema management — record a schema entry or view recent ones |
28| `datasets validate <input>` | Validate data — record a validation or view recent ones |
29| `datasets pipeline <input>` | Pipeline management — record a pipeline step or view recent ones |
30| `datasets profile <input>` | Profile data — record a profile or view recent ones |
31
32### Utility Commands
33
34| Command | Description |
35|---------|-------------|
36| `datasets stats` | Show summary statistics — entry counts per category, total entries, disk usage |
37| `datasets export <fmt>` | Export all data to a file (formats: `json`, `csv`, `txt`) |
38| `datasets search <term>` | Search all log files for a term (case-insensitive) |
39| `datasets recent` | Show last 20 entries from activity history |
40| `datasets status` | Health check — version, data directory, entry count, disk usage, last activity |
41| `datasets help` | Show available commands |
42| `datasets version` | Show version (v2.0.0) |
43
44## Data Storage
45
46All data is stored locally at `~/.local/share/datasets/`:
47
48- Each data command writes to its own log file (e.g., `ingest.log`, `transform.log`)
49- Entries are stored as `timestamp|value` pairs (pipe-delimited)
50- All actions are tracked in `history.log` with timestamps
51- Export generates files in the data directory (`export.json`, `export.csv`, or `export.txt`)
52
53## Requirements
54
55- Bash (with `set -euo pipefail`)
56- Standard Unix utilities: `date`, `wc`, `du`, `grep`, `tail`, `cat`, `sed`
57- No external dependencies or API keys required
58
59## When to Use
60
61- To log and track data processing operations (ingest, transform, query, etc.)
62- To maintain a searchable history of data pipeline activities
63- To export accumulated records in JSON, CSV, or plain text format
64- As part of larger automation or data-pipeline workflows
65- When you need a lightweight, local-only dataset operation tracker
66
67## Examples
68
69```bash
70# Record a new ingest entry
71datasets ingest "loaded training_data.csv 10000 rows"
72
73# View recent transform entries
74datasets transform
75
76# Record a query
77datasets query "filter by date > 2026-01-01"
78
79# Search across all logs
80datasets search "training"
81
82# Export everything as JSON
83datasets export json
84
85# Check overall statistics
86datasets stats
87
88# View recent activity
89datasets recent
90
91# Health check
92datasets status
93```
94
95---
96Powered by BytesAgain | bytesagain.com | hello@bytesagain.com
97💬 Feedback & Feature Requests: https://bytesagain.com/feedback
bytesagain