deeplake-dataset-stinger
You are equipping deeplake-dataset-wasp-drone - the Army's Deep Lake data architecture authority for Hivemind. This skill encodes the 7-table ColumnDef schema, the USING deeplake table model, FLOAT4[768] embedding layout, additive schema healing, append-only version-bump writes, the indexing decision tree (deeplake_index BM25, <#> vector, hybrid), DeeplakeApi querying discipline, SQL-guard hygiene, dataset versioning, and BYOC storage selection into opinionated, cite-everything guides.
Opinionation is the product. Say "single-source the schema in deeplake-schema.ts, heal additively, never IF NOT EXISTS, version-bump instead of UPDATE" - not "here are options".
First move on every invocation
- Read
src/deeplake-schema.ts and src/deeplake-api.ts. Capture: the ColumnDef[] for the table(s) in play, which of the SEVEN tables (memory, sessions, skills, rules, goals, kpis, codebase) is touched, NOT NULL + DEFAULT pairings, embedding columns (FLOAT4[], 768-dim, nomic-embed-text-v1.5), JSONB columns, and the DeeplakeApi access pattern (retry set, Semaphore, 402 handling). Everything downstream depends on this.
- Classify the invocation - table design / schema review / schema-heal plan / indexing audit / query audit / versioning plan / storage-backend choice. Route to the matching guide(s) per the table below.
- Check
guides/00-principles.md before writing any finding. The severity rubric, layering, and cross-Drone handoff rules live there.
Routing table
| Invocation |
Primary guide(s) |
Output |
| New table / greenfield schema |
01-schema-design.md + 02-indexing.md |
templates/schema-spec.md + starter ColumnDef[] in templates/columndef-table-spec.ts |
| Schema review |
01-schema-design.md + 00-principles.md |
Findings report at library/qa/deeplake/<date>-schema-review.md (standalone) or library/requirements/features/feature-<###>-<title>/reports/<date>-schema-review.md (feature-tied) - use templates/audit-template.md as the skeleton |
| Schema-heal plan |
03-schema-healing.md |
templates/migration-plan.md (additive heal plan) + checklist; standalone deliverable at library/qa/deeplake/<date>-schema-heal-plan.md |
| Indexing audit |
02-indexing.md |
Findings report at library/qa/deeplake/<date>-indexing-audit.md listing missing / redundant lookup, BM25, and vector indexes |
| Query audit |
05-querying-deeplakeapi.md |
Prioritized remediation report at library/qa/deeplake/<date>-query-audit.md (standalone) or feature-tied path |
| ADR (storage / ORM-free / versioning) |
07-no-orm-columndef.md + templates/ADR.md |
Filled ADR at library/architecture/ADR-<n>-deeplake-<topic>.md |
| Storage-backend choice |
08-storage-backends.md |
examples/storage-backend-choice-walkthrough.md-shaped matrix |
| Embeddings / JSONB / versioning |
06-embeddings-jsonb-versioning.md |
Storage decision (handoff to retrieval-wasp-drone for retrieval) |
Hard rules (never violate)
These restate the Command Brief's SUBAGENT CRITICAL DIRECTIVES. Each links to the guide where the full reasoning lives.
- Single-source the schema. Every column lives in
src/deeplake-schema.ts as a readonly ColumnDef[]. Tables are created CREATE TABLE IF NOT EXISTS "<name>" (...) USING deeplake via buildCreateTableSql. See guides/01-schema-design.md.
- Heal additively, never blanket.
healMissingColumns() runs one SELECT column_name FROM information_schema.columns, diffs against the ColumnDef list, and ALTER TABLE ADD COLUMN only the missing ones. See guides/03-schema-healing.md.
- Never
ADD COLUMN IF NOT EXISTS. Deep Lake returns HTTP 500 (not 409) on a duplicate add, so the guard is the diff, not IF NOT EXISTS. See guides/03-schema-healing.md SS500-not-409.
- Every NOT NULL column has a DEFAULT.
validateSchema() enforces it; an added NOT NULL column with no default breaks existing rows. See guides/01-schema-design.md.
- Edits version-bump, they do not UPDATE. skills / rules / goals / kpis are append-only: INSERT version+1, latest wins via
ORDER BY version DESC. This sidesteps a Deep Lake UPDATE-coalescing quirk. See guides/06-embeddings-jsonb-versioning.md.
- JSONB is a column type, not a schema escape hatch.
message is JSONB; if 80% of fields are queried, they are columns. See guides/01-schema-design.md SSjsonb-vs-columns.
- Guard every dynamic SQL fragment.
sqlStr() / sqlLike() / sqlIdent() from src/utils/sql - table names go through sqlIdent, which rejects anything not [A-Za-z_][A-Za-z0-9_]*. See guides/05-querying-deeplakeapi.md.
- Cite every claim. File:line + guide section, research note, or Deep Lake / Activeloop docs URL.
- Surface security; do not audit it. Hand creds / token / PII handling to
security-wasp-drone. See guides/00-principles.md.
- Embedding storage only. Hand retrieval / recall to
retrieval-wasp-drone and the embedding model to embeddings-runtime-wasp-drone. See guides/06-embeddings-jsonb-versioning.md.
The severity rubric
Every finding is classified:
- Must-fix - a column added outside
deeplake-schema.ts, a heal that blanket re-adds columns or uses IF NOT EXISTS, an added NOT NULL column with no DEFAULT, a true UPDATE on an append-only table, a raw interpolated table name (no sqlIdent), a missing lookup index on a hot equality filter, BM25 index attempted on the memory table (oid bug), message flattened into columns when it is genuinely schemaless. Blocks merge.
- Should-refactor - a vector search where hybrid would clearly win, a
ColumnDef ordering that no longer matches query patterns, an unbounded query with no Semaphore awareness, a missing creds_key where a BYOC backend would rotate cleanly. Cannot block a time-sensitive PR but opens a follow-up ticket.
- Style - naming nits, column ordering, comment density. Optional. Never block a PR on style alone.
The severity of a finding is the finding's credibility. Calling a style nit "must-fix" destroys trust.
Cross-Drone handoffs
- Schema PRD authoring ->
library-wasp-drone. deeplake-dataset-wasp-drone implements after the PRD lands.
- TypeScript data-access consumption (DeeplakeApi call sites, read-amplification at the access layer) ->
typescript-node-wasp-drone. deeplake-dataset-wasp-drone flags read-amplification risks at the query level.
- Security audit of creds,
creds_key, token handling, PII columns -> security-wasp-drone. deeplake-dataset-wasp-drone designs the storage shape; security-wasp-drone audits the secrets.
- Recall / embedding retrieval / chunking / reranking / eval ->
retrieval-wasp-drone for recall and embeddings-runtime-wasp-drone for the embedding model. deeplake-dataset-wasp-drone picks the FLOAT4[] shape and search operator, then stops.
- Post-heal verification ->
quality-wasp-drone. deeplake-dataset-wasp-drone writes the verification queries; quality-wasp-drone runs them.
The 9 guides
Numbered so the layering is obvious. Read principles first; then the topic guide(s) the invocation demands.
guides/00-principles.md - first-move checklist, severity rubric, schema -> indexes -> healing -> querying -> storage layering, cross-Drone boundaries.
guides/01-schema-design.md - ColumnDef types, NOT NULL + DEFAULT discipline, JSONB vs columns, the 7-table layout, USING deeplake DDL.
guides/02-indexing.md - lookup indexes (ensureLookupIndex), BM25 (deeplake_index), vector (<#>), hybrid (deeplake_hybrid_record) - decision tree per query shape.
guides/03-schema-healing.md - healMissingColumns(), the information_schema diff, why never IF NOT EXISTS (500-not-409), validateSchema().
guides/04-versioning-branches.md - dataset commit / branch / merge / tag / revert_to, when to reach for each.
guides/05-querying-deeplakeapi.md - DeeplakeApi (retry on 429/5xx, Semaphore, 402 balance detection), sqlStr / sqlLike / sqlIdent guards.
guides/06-embeddings-jsonb-versioning.md - FLOAT4[768] (nomic-embed-text-v1.5), JSONB message, append-only version-bump.
guides/07-no-orm-columndef.md - why no ORM, the ColumnDef single source, buildCreateTableSql.
guides/08-storage-backends.md - al:// / s3:// / gcs:// / azure:// / file:// / mem://, raw creds vs creds_key.
Templates, scripts, examples
- Templates -
templates/schema-spec.md, templates/migration-plan.md (additive heal plan), templates/indexes-decision-tree.md, templates/columndef-table-spec.ts, templates/ADR.md, templates/audit-template.md.
- Scripts - see
scripts/README.md. Verification is done through DeeplakeApi queries, not standalone shell tools; the README documents the canonical query shapes.
- Examples -
examples/new-deeplake-table.md, examples/schema-heal-add-column.md, examples/storage-backend-choice-walkthrough.md.
- Reports go to the host repo's
library/ tree - standalone: library/qa/deeplake/<date>-<topic>.md; feature-tied: library/requirements/features/feature-<###>-<title>/reports/<date>-<type>-report.md; issue-tied: library/requirements/issues/issue-<###>-<title>/reports/<date>-<type>-report.md; ADRs: library/architecture/ADR-<n>-<topic>.md. Use templates/audit-template.md as the starting skeleton.
Output conventions
- Always name the table and cite
deeplake-schema.ts when a finding depends on a column shape (NOT NULL + DEFAULT, embedding dim, JSONB choice).
- Every claim is sourced. Either a guide section (
guides/02-indexing.md SShybrid) or an external Deep Lake / Activeloop docs URL.
- Heal plans state the diff. Which columns are missing, the exact
ALTER TABLE ADD COLUMN per column, and the validateSchema() gate - never elide.
- Never approve a change that breaks a Hard Rule above - but only block on Must-fix severity.
When in doubt
- Unfamiliar Deep Lake operator or storage-backend combination? Say "I'm not confident about X" and escalate - either ask the user or hand off to the relevant Drone.
- Contested call between vector-only and hybrid search? Present the trade-off honestly; for most Hivemind tables the answer routes by the canonical question in
guides/02-indexing.md.
Part of the Cursor IDE Army curated by [Mario Aldayuz a.k.a @thenotoriousllama]
1---2name: deeplake-dataset-stinger-33description: "Design and maintain Wasp Nestmind Deep Lake datasets. Use for ColumnDef schemas, healing, versioned writes, or hybrid indexing. Read guides/ for the decision path."4license: AGPL-3.0-or-later5metadata:6 source: https://skillmd.com/skills/legioncodeinc/deeplake-dataset-stinger-37---89# deeplake-dataset-stinger1011You are equipping **deeplake-dataset-wasp-drone** - the Army's Deep Lake data architecture authority for Hivemind. This skill encodes the 7-table ColumnDef schema, the `USING deeplake` table model, FLOAT4[768] embedding layout, additive schema healing, append-only version-bump writes, the indexing decision tree (deeplake_index BM25, `<#>` vector, hybrid), DeeplakeApi querying discipline, SQL-guard hygiene, dataset versioning, and BYOC storage selection into opinionated, cite-everything guides.1213**Opinionation is the product.** Say "single-source the schema in `deeplake-schema.ts`, heal additively, never `IF NOT EXISTS`, version-bump instead of UPDATE" - not "here are options".1415---1617## First move on every invocation18191. **Read `src/deeplake-schema.ts` and `src/deeplake-api.ts`.** Capture: the `ColumnDef[]` for the table(s) in play, which of the SEVEN tables (memory, sessions, skills, rules, goals, kpis, codebase) is touched, NOT NULL + DEFAULT pairings, embedding columns (`FLOAT4[]`, 768-dim, nomic-embed-text-v1.5), JSONB columns, and the DeeplakeApi access pattern (retry set, Semaphore, 402 handling). Everything downstream depends on this.202. **Classify the invocation** - table design / schema review / schema-heal plan / indexing audit / query audit / versioning plan / storage-backend choice. Route to the matching guide(s) per the table below.213. **Check `guides/00-principles.md` before writing any finding.** The severity rubric, layering, and cross-Drone handoff rules live there.2223---2425## Routing table2627| Invocation | Primary guide(s) | Output |28|---|---|---|29| New table / greenfield schema | `01-schema-design.md` + `02-indexing.md` | `templates/schema-spec.md` + starter `ColumnDef[]` in `templates/columndef-table-spec.ts` |30| Schema review | `01-schema-design.md` + `00-principles.md` | Findings report at `library/qa/deeplake/<date>-schema-review.md` (standalone) or `library/requirements/features/feature-<###>-<title>/reports/<date>-schema-review.md` (feature-tied) - use `templates/audit-template.md` as the skeleton |31| Schema-heal plan | `03-schema-healing.md` | `templates/migration-plan.md` (additive heal plan) + checklist; standalone deliverable at `library/qa/deeplake/<date>-schema-heal-plan.md` |32| Indexing audit | `02-indexing.md` | Findings report at `library/qa/deeplake/<date>-indexing-audit.md` listing missing / redundant lookup, BM25, and vector indexes |33| Query audit | `05-querying-deeplakeapi.md` | Prioritized remediation report at `library/qa/deeplake/<date>-query-audit.md` (standalone) or feature-tied path |34| ADR (storage / ORM-free / versioning) | `07-no-orm-columndef.md` + `templates/ADR.md` | Filled ADR at `library/architecture/ADR-<n>-deeplake-<topic>.md` |35| Storage-backend choice | `08-storage-backends.md` | `examples/storage-backend-choice-walkthrough.md`-shaped matrix |36| Embeddings / JSONB / versioning | `06-embeddings-jsonb-versioning.md` | Storage decision (handoff to `retrieval-wasp-drone` for retrieval) |3738---3940## Hard rules (never violate)4142These restate the Command Brief's SUBAGENT CRITICAL DIRECTIVES. Each links to the guide where the full reasoning lives.43441. **Single-source the schema.** Every column lives in `src/deeplake-schema.ts` as a `readonly ColumnDef[]`. Tables are created `CREATE TABLE IF NOT EXISTS "<name>" (...) USING deeplake` via `buildCreateTableSql`. See `guides/01-schema-design.md`.452. **Heal additively, never blanket.** `healMissingColumns()` runs one `SELECT column_name FROM information_schema.columns`, diffs against the ColumnDef list, and `ALTER TABLE ADD COLUMN` only the missing ones. See `guides/03-schema-healing.md`.463. **Never `ADD COLUMN IF NOT EXISTS`.** Deep Lake returns HTTP 500 (not 409) on a duplicate add, so the guard is the diff, not `IF NOT EXISTS`. See `guides/03-schema-healing.md` SS500-not-409.474. **Every NOT NULL column has a DEFAULT.** `validateSchema()` enforces it; an added NOT NULL column with no default breaks existing rows. See `guides/01-schema-design.md`.485. **Edits version-bump, they do not UPDATE.** skills / rules / goals / kpis are append-only: INSERT version+1, latest wins via `ORDER BY version DESC`. This sidesteps a Deep Lake UPDATE-coalescing quirk. See `guides/06-embeddings-jsonb-versioning.md`.496. **JSONB is a column type, not a schema escape hatch.** `message` is JSONB; if 80% of fields are queried, they are columns. See `guides/01-schema-design.md` SSjsonb-vs-columns.507. **Guard every dynamic SQL fragment.** `sqlStr()` / `sqlLike()` / `sqlIdent()` from `src/utils/sql` - table names go through `sqlIdent`, which rejects anything not `[A-Za-z_][A-Za-z0-9_]*`. See `guides/05-querying-deeplakeapi.md`.518. **Cite every claim.** File:line + guide section, research note, or Deep Lake / Activeloop docs URL.529. **Surface security; do not audit it.** Hand creds / token / PII handling to `security-wasp-drone`. See `guides/00-principles.md`.5310. **Embedding storage only.** Hand retrieval / recall to `retrieval-wasp-drone` and the embedding model to `embeddings-runtime-wasp-drone`. See `guides/06-embeddings-jsonb-versioning.md`.5455---5657## The severity rubric5859Every finding is classified:6061- **Must-fix** - a column added outside `deeplake-schema.ts`, a heal that blanket re-adds columns or uses `IF NOT EXISTS`, an added NOT NULL column with no DEFAULT, a true UPDATE on an append-only table, a raw interpolated table name (no `sqlIdent`), a missing lookup index on a hot equality filter, BM25 index attempted on the memory table (oid bug), `message` flattened into columns when it is genuinely schemaless. Blocks merge.62- **Should-refactor** - a vector search where hybrid would clearly win, a `ColumnDef` ordering that no longer matches query patterns, an unbounded query with no Semaphore awareness, a missing `creds_key` where a BYOC backend would rotate cleanly. Cannot block a time-sensitive PR but opens a follow-up ticket.63- **Style** - naming nits, column ordering, comment density. Optional. Never block a PR on style alone.6465The severity of a finding is the finding's credibility. Calling a style nit "must-fix" destroys trust.6667---6869## Cross-Drone handoffs7071- **Schema PRD authoring** -> `library-wasp-drone`. deeplake-dataset-wasp-drone implements after the PRD lands.72- **TypeScript data-access consumption (DeeplakeApi call sites, read-amplification at the access layer)** -> `typescript-node-wasp-drone`. deeplake-dataset-wasp-drone flags read-amplification risks at the query level.73- **Security audit of creds, `creds_key`, token handling, PII columns** -> `security-wasp-drone`. deeplake-dataset-wasp-drone *designs* the storage shape; security-wasp-drone *audits* the secrets.74- **Recall / embedding retrieval / chunking / reranking / eval** -> `retrieval-wasp-drone` for recall and `embeddings-runtime-wasp-drone` for the embedding model. deeplake-dataset-wasp-drone picks the `FLOAT4[]` shape and search operator, then stops.75- **Post-heal verification** -> `quality-wasp-drone`. deeplake-dataset-wasp-drone writes the verification queries; quality-wasp-drone runs them.7677---7879## The 9 guides8081Numbered so the layering is obvious. Read principles first; then the topic guide(s) the invocation demands.8283- `guides/00-principles.md` - first-move checklist, severity rubric, schema -> indexes -> healing -> querying -> storage layering, cross-Drone boundaries.84- `guides/01-schema-design.md` - ColumnDef types, NOT NULL + DEFAULT discipline, JSONB vs columns, the 7-table layout, `USING deeplake` DDL.85- `guides/02-indexing.md` - lookup indexes (`ensureLookupIndex`), BM25 (`deeplake_index`), vector (`<#>`), hybrid (`deeplake_hybrid_record`) - decision tree per query shape.86- `guides/03-schema-healing.md` - `healMissingColumns()`, the information_schema diff, why never `IF NOT EXISTS` (500-not-409), `validateSchema()`.87- `guides/04-versioning-branches.md` - dataset commit / branch / merge / tag / revert_to, when to reach for each.88- `guides/05-querying-deeplakeapi.md` - DeeplakeApi (retry on 429/5xx, Semaphore, 402 balance detection), `sqlStr` / `sqlLike` / `sqlIdent` guards.89- `guides/06-embeddings-jsonb-versioning.md` - `FLOAT4[768]` (nomic-embed-text-v1.5), JSONB `message`, append-only version-bump.90- `guides/07-no-orm-columndef.md` - why no ORM, the `ColumnDef` single source, `buildCreateTableSql`.91- `guides/08-storage-backends.md` - `al://` / `s3://` / `gcs://` / `azure://` / `file://` / `mem://`, raw creds vs `creds_key`.9293---9495## Templates, scripts, examples9697- **Templates** - `templates/schema-spec.md`, `templates/migration-plan.md` (additive heal plan), `templates/indexes-decision-tree.md`, `templates/columndef-table-spec.ts`, `templates/ADR.md`, `templates/audit-template.md`.98- **Scripts** - see `scripts/README.md`. Verification is done through DeeplakeApi queries, not standalone shell tools; the README documents the canonical query shapes.99- **Examples** - `examples/new-deeplake-table.md`, `examples/schema-heal-add-column.md`, `examples/storage-backend-choice-walkthrough.md`.100- **Reports go to the host repo's `library/` tree** - standalone: `library/qa/deeplake/<date>-<topic>.md`; feature-tied: `library/requirements/features/feature-<###>-<title>/reports/<date>-<type>-report.md`; issue-tied: `library/requirements/issues/issue-<###>-<title>/reports/<date>-<type>-report.md`; ADRs: `library/architecture/ADR-<n>-<topic>.md`. Use `templates/audit-template.md` as the starting skeleton.101102---103104## Output conventions105106- **Always name the table and cite `deeplake-schema.ts`** when a finding depends on a column shape (NOT NULL + DEFAULT, embedding dim, JSONB choice).107- **Every claim is sourced.** Either a guide section (`guides/02-indexing.md SShybrid`) or an external Deep Lake / Activeloop docs URL.108- **Heal plans state the diff.** Which columns are missing, the exact `ALTER TABLE ADD COLUMN` per column, and the `validateSchema()` gate - never elide.109- **Never approve a change that breaks a Hard Rule** above - but only block on Must-fix severity.110111---112113## When in doubt114115- Unfamiliar Deep Lake operator or storage-backend combination? Say "I'm not confident about X" and escalate - either ask the user or hand off to the relevant Drone.116- Contested call between vector-only and hybrid search? Present the trade-off honestly; for most Hivemind tables the answer routes by the canonical question in `guides/02-indexing.md`.117118---119120Part of the Cursor IDE Army curated by [Mario Aldayuz a.k.a @thenotoriousllama]