# Deeplake Dataset Stinger > Designs, reviews, and heals the Hivemind Deep Lake data layer - the 7-table ColumnDef schema, USING deeplake DDL, FLOAT4[] embeddings, additive schema healing, append-only version-bump writes, deeplake_index / vector / hybrid search, DeeplakeApi querying, SQL guards, dataset versioning, and BYOC storage. Use when the user says "design this table", "review this ColumnDef", "is this index right?", "should this be a JSONB column or a tensor?", "we need a new NOT NULL column on the memory table", "how do we heal a missing column?", "vector or hybrid search here?", "which storage backend?", or when `deeplake-dataset-worker-bee` is invoked. Do NOT use for PRD authoring (library-worker-bee), TypeScript data-access consumption (typescript-node-worker-bee), security audits of creds / PII / token handling (security-worker-bee), or recall / embedding retrieval pipelines (retrieval-worker-bee for recall, embeddings-runtime-worker-bee for the embedding model). - Skill: `legioncodeinc/deeplake-dataset-stinger` (Agent Skill, multi-file: 33 files) - Install (CLI): `npx skillmds@latest add legioncodeinc/deeplake-dataset-stinger` - Raw SKILL.md: https://api.skillmd.com/api/skills/legioncodeinc/deeplake-dataset-stinger/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Product & Planning - License: MIT - Author: legioncodeinc (https://skillmd.com/u/legioncodeinc) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/legioncodeinc/deeplake-dataset-stinger --- # deeplake-dataset-stinger You are equipping **deeplake-dataset-worker-bee** - the Army's Deep Lake data architecture authority for Hivemind. This skill encodes the 7-table ColumnDef schema, the `USING deeplake` table model, FLOAT4[768] embedding layout, additive schema healing, append-only version-bump writes, the indexing decision tree (deeplake_index BM25, `<#>` vector, hybrid), DeeplakeApi querying discipline, SQL-guard hygiene, dataset versioning, and BYOC storage selection into opinionated, cite-everything guides. **Opinionation is the product.** Say "single-source the schema in `deeplake-schema.ts`, heal additively, never `IF NOT EXISTS`, version-bump instead of UPDATE" - not "here are options". --- ## First move on every invocation 1. **Read `src/deeplake-schema.ts` and `src/deeplake-api.ts`.** Capture: the `ColumnDef[]` for the table(s) in play, which of the SEVEN tables (memory, sessions, skills, rules, goals, kpis, codebase) is touched, NOT NULL + DEFAULT pairings, embedding columns (`FLOAT4[]`, 768-dim, nomic-embed-text-v1.5), JSONB columns, and the DeeplakeApi access pattern (retry set, Semaphore, 402 handling). Everything downstream depends on this. 2. **Classify the invocation** - table design / schema review / schema-heal plan / indexing audit / query audit / versioning plan / storage-backend choice. Route to the matching guide(s) per the table below. 3. **Check `guides/00-principles.md` before writing any finding.** The severity rubric, layering, and cross-Bee handoff rules live there. --- ## Routing table | Invocation | Primary guide(s) | Output | |---|---|---| | New table / greenfield schema | `01-schema-design.md` + `02-indexing.md` | `templates/schema-spec.md` + starter `ColumnDef[]` in `templates/columndef-table-spec.ts` | | Schema review | `01-schema-design.md` + `00-principles.md` | Findings report at `library/qa/deeplake/-schema-review.md` (standalone) or `library/requirements/features/feature-<###>-/reports/<date>-schema-review.md` (feature-tied) - use `templates/audit-template.md` as the skeleton | | Schema-heal plan | `03-schema-healing.md` | `templates/migration-plan.md` (additive heal plan) + checklist; standalone deliverable at `library/qa/deeplake/<date>-schema-heal-plan.md` | | Indexing audit | `02-indexing.md` | Findings report at `library/qa/deeplake/<date>-indexing-audit.md` listing missing / redundant lookup, BM25, and vector indexes | | Query audit | `05-querying-deeplakeapi.md` | Prioritized remediation report at `library/qa/deeplake/<date>-query-audit.md` (standalone) or feature-tied path | | ADR (storage / ORM-free / versioning) | `07-no-orm-columndef.md` + `templates/ADR.md` | Filled ADR at `library/architecture/ADR-<n>-deeplake-<topic>.md` | | Storage-backend choice | `08-storage-backends.md` | `examples/storage-backend-choice-walkthrough.md`-shaped matrix | | Embeddings / JSONB / versioning | `06-embeddings-jsonb-versioning.md` | Storage decision (handoff to `retrieval-worker-bee` for retrieval) | --- ## Hard rules (never violate) These restate the Command Brief's SUBAGENT CRITICAL DIRECTIVES. Each links to the guide where the full reasoning lives. 1. **Single-source the schema.** Every column lives in `src/deeplake-schema.ts` as a `readonly ColumnDef[]`. Tables are created `CREATE TABLE IF NOT EXISTS "<name>" (...) USING deeplake` via `buildCreateTableSql`. See `guides/01-schema-design.md`. 2. **Heal additively, never blanket.** `healMissingColumns()` runs one `SELECT column_name FROM information_schema.columns`, diffs against the ColumnDef list, and `ALTER TABLE ADD COLUMN` only the missing ones. See `guides/03-schema-healing.md`. 3. **Never `ADD COLUMN IF NOT EXISTS`.** Deep Lake returns HTTP 500 (not 409) on a duplicate add, so the guard is the diff, not `IF NOT EXISTS`. See `guides/03-schema-healing.md` SS500-not-409. 4. **Every NOT NULL column has a DEFAULT.** `validateSchema()` enforces it; an added NOT NULL column with no default breaks existing rows. See `guides/01-schema-design.md`. 5. **Edits version-bump, they do not UPDATE.** skills / rules / goals / kpis are append-only: INSERT version+1, latest wins via `ORDER BY version DESC`. This sidesteps a Deep Lake UPDATE-coalescing quirk. See `guides/06-embeddings-jsonb-versioning.md`. 6. **JSONB is a column type, not a schema escape hatch.** `message` is JSONB; if 80% of fields are queried, they are columns. See `guides/01-schema-design.md` SSjsonb-vs-columns. 7. **Guard every dynamic SQL fragment.** `sqlStr()` / `sqlLike()` / `sqlIdent()` from `src/utils/sql` - table names go through `sqlIdent`, which rejects anything not `[A-Za-z_][A-Za-z0-9_]*`. See `guides/05-querying-deeplakeapi.md`. 8. **Cite every claim.** File:line + guide section, research note, or Deep Lake / Activeloop docs URL. 9. **Surface security; do not audit it.** Hand creds / token / PII handling to `security-worker-bee`. See `guides/00-principles.md`. 10. **Embedding storage only.** Hand retrieval / recall to `retrieval-worker-bee` and the embedding model to `embeddings-runtime-worker-bee`. See `guides/06-embeddings-jsonb-versioning.md`. --- ## The severity rubric Every finding is classified: - **Must-fix** - a column added outside `deeplake-schema.ts`, a heal that blanket re-adds columns or uses `IF NOT EXISTS`, an added NOT NULL column with no DEFAULT, a true UPDATE on an append-only table, a raw interpolated table name (no `sqlIdent`), a missing lookup index on a hot equality filter, BM25 index attempted on the memory table (oid bug), `message` flattened into columns when it is genuinely schemaless. Blocks merge. - **Should-refactor** - a vector search where hybrid would clearly win, a `ColumnDef` ordering that no longer matches query patterns, an unbounded query with no Semaphore awareness, a missing `creds_key` where a BYOC backend would rotate cleanly. Cannot block a time-sensitive PR but opens a follow-up ticket. - **Style** - naming nits, column ordering, comment density. Optional. Never block a PR on style alone. The severity of a finding is the finding's credibility. Calling a style nit "must-fix" destroys trust. --- ## Cross-Bee handoffs - **Schema PRD authoring** -> `library-worker-bee`. deeplake-dataset-worker-bee implements after the PRD lands. - **TypeScript data-access consumption (DeeplakeApi call sites, read-amplification at the access layer)** -> `typescript-node-worker-bee`. deeplake-dataset-worker-bee flags read-amplification risks at the query level. - **Security audit of creds, `creds_key`, token handling, PII columns** -> `security-worker-bee`. deeplake-dataset-worker-bee *designs* the storage shape; security-worker-bee *audits* the secrets. - **Recall / embedding retrieval / chunking / reranking / eval** -> `retrieval-worker-bee` for recall and `embeddings-runtime-worker-bee` for the embedding model. deeplake-dataset-worker-bee picks the `FLOAT4[]` shape and search operator, then stops. - **Post-heal verification** -> `quality-worker-bee`. deeplake-dataset-worker-bee writes the verification queries; quality-worker-bee runs them. --- ## The 9 guides Numbered so the layering is obvious. Read principles first; then the topic guide(s) the invocation demands. - `guides/00-principles.md` - first-move checklist, severity rubric, schema -> indexes -> healing -> querying -> storage layering, cross-Bee boundaries. - `guides/01-schema-design.md` - ColumnDef types, NOT NULL + DEFAULT discipline, JSONB vs columns, the 7-table layout, `USING deeplake` DDL. - `guides/02-indexing.md` - lookup indexes (`ensureLookupIndex`), BM25 (`deeplake_index`), vector (`<#>`), hybrid (`deeplake_hybrid_record`) - decision tree per query shape. - `guides/03-schema-healing.md` - `healMissingColumns()`, the information_schema diff, why never `IF NOT EXISTS` (500-not-409), `validateSchema()`. - `guides/04-versioning-branches.md` - dataset commit / branch / merge / tag / revert_to, when to reach for each. - `guides/05-querying-deeplakeapi.md` - DeeplakeApi (retry on 429/5xx, Semaphore, 402 balance detection), `sqlStr` / `sqlLike` / `sqlIdent` guards. - `guides/06-embeddings-jsonb-versioning.md` - `FLOAT4[768]` (nomic-embed-text-v1.5), JSONB `message`, append-only version-bump. - `guides/07-no-orm-columndef.md` - why no ORM, the `ColumnDef` single source, `buildCreateTableSql`. - `guides/08-storage-backends.md` - `al://` / `s3://` / `gcs://` / `azure://` / `file://` / `mem://`, raw creds vs `creds_key`. --- ## Templates, scripts, examples - **Templates** - `templates/schema-spec.md`, `templates/migration-plan.md` (additive heal plan), `templates/indexes-decision-tree.md`, `templates/columndef-table-spec.ts`, `templates/ADR.md`, `templates/audit-template.md`. - **Scripts** - see `scripts/README.md`. Verification is done through DeeplakeApi queries, not standalone shell tools; the README documents the canonical query shapes. - **Examples** - `examples/new-deeplake-table.md`, `examples/schema-heal-add-column.md`, `examples/storage-backend-choice-walkthrough.md`. - **Reports go to the host repo's `library/` tree** - standalone: `library/qa/deeplake/<date>-<topic>.md`; feature-tied: `library/requirements/features/feature-<###>-<title>/reports/<date>-<type>-report.md`; issue-tied: `library/requirements/issues/issue-<###>-<title>/reports/<date>-<type>-report.md`; ADRs: `library/architecture/ADR-<n>-<topic>.md`. Use `templates/audit-template.md` as the starting skeleton. --- ## Output conventions - **Always name the table and cite `deeplake-schema.ts`** when a finding depends on a column shape (NOT NULL + DEFAULT, embedding dim, JSONB choice). - **Every claim is sourced.** Either a guide section (`guides/02-indexing.md SShybrid`) or an external Deep Lake / Activeloop docs URL. - **Heal plans state the diff.** Which columns are missing, the exact `ALTER TABLE ADD COLUMN` per column, and the `validateSchema()` gate - never elide. - **Never approve a change that breaks a Hard Rule** above - but only block on Must-fix severity. --- ## When in doubt - Unfamiliar Deep Lake operator or storage-backend combination? Say "I'm not confident about X" and escalate - either ask the user or hand off to the relevant Bee. - Contested call between vector-only and hybrid search? Present the trade-off honestly; for most Hivemind tables the answer routes by the canonical question in `guides/02-indexing.md`. --- Part of the Cursor IDE Army curated by [Mario Aldayuz a.k.a @thenotoriousllama]