# Datacoolie Build

> Build, modify, materialize, run, and verify DataCoolie projects. Use for workspace bootstrap, metadata authoring, environment overlays, capability checks, runners, notebooks, custom functions, narrow unsupported adapters, local tests, immutable builds, and project-owned build/CI automation. This is the sole implementation skill for pipeline and build artifacts; it does not discover sources, make material design decisions, provision infrastructure, or deploy releases.

- Skill: `datacoolie/datacoolie-build` (Agent Skill, multi-file: 44 files)
- Install (CLI): `npx skillmds@latest add datacoolie/datacoolie-build`
- Raw SKILL.md: https://api.skillmd.com/api/skills/datacoolie/datacoolie-build/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: datacoolie (https://skillmd.com/u/datacoolie)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/datacoolie/datacoolie-build

---


# DataCoolie Build

## Outcome And Boundary

Turn current project intent into durable DataCoolie sources and an immutable
`.builds/artifacts/{build_id}` verified by executing the generated artifacts. Bootstrap only the
workspace structure required by the request; initialization is not a separate phase.

Own configuration, metadata, overlays, capability proof, runners/notebooks, functions, narrow
unsupported adapters, materialization, local execution, build evidence, and requested project-owned
automation. Return unknown source facts to discover, material decisions to design, missing resources
to provision with the exact requirements artifact and evidence, and deployment work to release.

Use the installed `datacoolie` package and public APIs. Resolve bundled resources relative to this
skill; generated projects must not depend on skill paths.

## Inputs And Gates

- Read the user request and only affected workspace sources.
- Use `architecture/current.md` when a new project or material contract requires it.
- When `architecture/current.md` exists, recompute its final-byte hash and reject a missing,
  malformed, or stale matching design receipt; reject misnamed receipts too. Architecture never
  self-declares an approval bypass.
- Require discovery evidence for every declared source in a new project. Use discovery artifacts
  only as authoring evidence; runtime code must not import them.
- Return to design before implementation if the requested change would alter a material contract.

## Resource Routing

| Need | Read or run |
|---|---|
| Build-tool dependencies | `scripts/requirements.txt`; add `requirements-excel.txt` only for Excel conversion |
| Workspace/config | `templates/project-structure.md`, `schemas/workspace-config.schema.json`, `scripts/validate_config.py` |
| Metadata fields and authoring | `references/schema-quick-reference.md`, `schemas/`, `scripts/validate.py` |
| Generated metadata layout | `templates/project-structure.md`, `scripts/materialize.py` |
| Metadata import/merge/lint | `scripts/convert.py`, `scripts/merge.py`, `scripts/lint.py` |
| Built-in capability inventory | `scripts/inspect_capabilities.py`, `references/capability-catalog.md` |
| Platform runtime, path, credential, or extra | `references/platform-contract.md`, then the matching runner template |
| Native versus custom boundary | `references/framework-boundary.md` |
| Python-function source or artifact | `references/python-functions-contract.md`, `scripts/validate_functions.py` |
| Common entrypoint and normal run | `references/runner-contract.md`, `templates/runners/README.md`, matching template |
| Polars Delta/Iceberg `source.query` | `references/polars-qualified-sql.md`, then `references/runner-contract.md` |
| Replay or maintenance extensions | load `references/runner-contract.md`, then `references/operations-contract.md` and matching templates |
| Immutable build, runnable current projection, and verification receipt | `scripts/materialize.py`, `scripts/validate_build.py`, `schemas/current-build.schema.json`, `schemas/build-verification-receipt.schema.json` |
| Requested project automation | `scripts/render_automation.py` |

Load only resources needed for the current outcome. Exact metadata layouts, runner names and
parameters, stage semantics, operation behavior, build identity, and manifest rules live in
the routed build resources rather than this prompt.

## Decision Workflow

### 1. Bind the environment

Keep `config.yaml` limited to project identity and environment-to-platform mapping. Validate it
against installed platform registrations. Engines, stages, runtime paths, secrets, and gate state
do not belong there. Environment names are project-defined non-empty values, not a fixed
`dev/test/prod` vocabulary.
Materialization always produces one complete snapshot of every configured environment. Environment
selection belongs to run, test-receipt, and release slices, never to Build scope or build identity.

### 2. Prove capability fit

Evaluate the installed combination of source, authentication, engine, transforms, destination,
load, platform, and dependencies. Inspect the installed registries before deciding; a missing
optional dependency is setup work, not evidence that a registered capability is unsupported. Use
metadata and `DataCoolieDriver.run(...)` for a supported path. Add custom code only around a
verified unsupported boundary, record the evidence, and leave the supported remainder native.
When platform execution context, path, credentials, or dependencies affect the combination, load
`references/platform-contract.md`; platform is the adapter and does not imply the execution host.

### 3. Author durable sources

Use the canonical metadata contract and environment overlays; do not clone full metadata per
environment. Use ordered selector patches for changes shared by matching canonical entities and
exact keyed overrides for additions or final exceptions; exact overrides win. Global
`schema_hints` selectors operate at connection/schema/table/column grain, while schema hints under
a selected dataflow remain local to that dataflow. Load `templates/project-structure.md` and
`references/schema-quick-reference.md` when authoring overlays. Create only required normal,
replay, or maintenance entrypoints. The selected file
fixes platform, engine, provider, and operation; runtime inputs carry only values allowed by the
runner and operation contracts. Keep credentials in environment or platform secret services.

Resolve metadata, log, and watermark paths inside the environment's approved persistent control
namespace and pass them unchanged. Deployed metadata is a build-scoped immutable projection; logs
and watermarks remain mutable and outside build artifacts. For a cloud platform used by an
on-premises runner, select the external runtime explicitly and keep the actual execution host
separate from the platform adapter.
Assume source query and action text can appear in framework logs. Do not embed secret literals;
apply the approved log classification, access, and retention policy to generated runtime paths.

Treat a file or lakehouse connection `base_path` as the root; the framework appends each non-empty
dataflow `schema_name` and `table`, so do not repeat those segments in the connection. Author exact
source-observed, broad, or shared type mappings once in `metadata/schema_hints.json`. Use
`transform.schema_hints` only for a small dataflow-specific cast or override. A non-empty transform
hint set prevents global hints from being attached, so do not assume the two sources merge. Select
the simplest native source address using the framework-boundary order.
Before adding audit or partition helper columns, compare their semantics with framework-generated
columns and native destination routing. Preserve distinct source/business timestamps, but do not
duplicate framework write-time or driver-managed dataflow run identity unless an explicit consumer
contract requires a separate named field. For flat-file load-time folder routing, prefer the
destination connection's `date_folder_partitions`; use `partition_columns` for data-value routing.
Use backward lookback primarily when discovery found no reliable change signal and a
transaction/business-date column must bound late corrections; pair it with an idempotent load that
can reprocess the window. For file sources, prefer `__file_modification_time` when file timestamps
are reliable. Add source `date_folder_partitions` only for an observed year/month/day/hour path
layout; it prunes folders and may be combined with file modification time. Destination
`date_folder_partitions` is a separate load-time routing concern.

When Polars executes a Delta or Iceberg `source.query`, load
`references/polars-qualified-sql.md`. Keep the SQL in normal metadata and register the required
relations on the same active `PolarsEngine` before constructing or running the driver. Registration
options are runner/bootstrap concerns, not `source.configure` fields.

### 4. Run fast source checks

Validate config and resolved metadata, lint affected paths, parse/compile entrypoints, and unit-test
helpers directly. These checks give fast feedback but do not prove the generated build.

### 5. Materialize and verify

Run `scripts/materialize.py`; select `single` (default), `split-connections`, or `split-all` metadata
layout explicitly when needed. It validates its inputs, renders typed metadata sets under each
environment's fixed `metadata/` component, packages zero or one function artifact under the fixed
`functions/` component, writes the manifest and checksums under `.builds/artifacts/{build_id}`, verifies the
immutable bytes, and replaces `.builds/current` with a verified runnable projection of that whole
build. The projection copies runtime files, omits artifact-only manifest/checksums, and records its
exact source ID in `current/build.json`. Never symlink or mutate immutable artifact contents.

Always validate the immutable build, resolved metadata, exact runner, and optional functions
artifact. Execute the generated runner on the Build host when that host is compatible and the
approved check is safe; record the result as useful Build-host evidence, not target qualification.
Do not block an artifact-qualified receipt solely because the runner requires staging on its target
execution host. Release always qualifies the exact staged runner slice before activation.

Keep any Build-host logs and watermarks under persistent `.runtime/{env}/` or another approved
isolated test namespace. Apply the runner contract for normal runs and the operations contract for
replay or maintenance, including their mutation confirmations.

Execute and validate `.builds/current` directly for the normal latest-build path. Select
`.builds/artifacts/{build_id}` only for a historical version. Write a typed successful or failed
receipt under `.builds/evidence/{build_id}/{env}/{receipt_id}.json`, using the exact ID from
`current/build.json` when current was tested. Release never consumes the moving projection.

### 6. Add automation only when requested

Use `scripts/render_automation.py` only for requested reproducible project-owned build/CI entrypoints.
Generated automation works with the installed framework and project sources without installed
skills. Release owns consume-only deployment automation. Do not generate speculative automation.

## Output And Handoff

```text
{workspace}/config.yaml
{workspace}/metadata/
{workspace}/runners/
{workspace}/functions/                          # optional
{workspace}/automation/                         # optional
{workspace}/.builds/artifacts/{build_id}/manifest.json
{workspace}/.builds/artifacts/{build_id}/SHA256SUMS
{workspace}/.builds/artifacts/{build_id}/{env}/...
{workspace}/.builds/evidence/{build_id}/{env}/*.json
{workspace}/.builds/current/build.json
{workspace}/.builds/current/{env}/...
{workspace}/.builds/current/functions/            # when functions were packaged
```

Release may receive `current` as a convenience selector, but resolves `current/build.json` once and
then consumes only the exact build ID, canonical local build directory or immutable remote artifact
identity, manifest/checksums, target slice, and successful matching artifact-verification receipt.
Build manifest v3 and Build receipt v4 bind the full metadata set and optional fixed-component
function artifact. The receipt requires `generated-artifact-validation`; Build-host runtime execution is
optional and never authorizes target activation. Build current is never a transfer source or
authorization identity. Build or design approval never authorizes deployment. End with verification
evidence, skipped checks, and unresolved questions.