Datacoolie Build

DataCoolie Build

Outcome And Boundary

Turn current project intent into durable DataCoolie sources and an immutable .builds/artifacts/{build_id} verified by executing the generated artifacts. Bootstrap only the workspace structure required by the request; initialization is not a separate phase.

Own configuration, metadata, overlays, capability proof, runners/notebooks, functions, narrow unsupported adapters, materialization, local execution, build evidence, and requested project-owned automation. Return unknown source facts to discover, material decisions to design, missing resources to provision with the exact requirements artifact and evidence, and deployment work to release.

Use the installed datacoolie package and public APIs. Resolve bundled resources relative to this skill; generated projects must not depend on skill paths.

Inputs And Gates

• Read the user request and only affected workspace sources.
• Use architecture/current.md when a new project or material contract requires it.
• When architecture/current.md exists, recompute its final-byte hash and reject a missing, malformed, or stale matching design receipt; reject misnamed receipts too. Architecture never self-declares an approval bypass.
• Require discovery evidence for every declared source in a new project. Use discovery artifacts only as authoring evidence; runtime code must not import them.
• Return to design before implementation if the requested change would alter a material contract.

Resource Routing

Load only resources needed for the current outcome. Exact metadata layouts, runner names and parameters, stage semantics, operation behavior, build identity, and manifest rules live in the routed build resources rather than this prompt.

Decision Workflow

1. Bind the environment

Keep config.yaml limited to project identity and environment-to-platform mapping. Validate it against installed platform registrations. Engines, stages, runtime paths, secrets, and gate state do not belong there. Environment names are project-defined non-empty values, not a fixed dev/test/prod vocabulary. Materialization always produces one complete snapshot of every configured environment. Environment selection belongs to run, test-receipt, and release slices, never to Build scope or build identity.

2. Prove capability fit

Evaluate the installed combination of source, authentication, engine, transforms, destination, load, platform, and dependencies. Inspect the installed registries before deciding; a missing optional dependency is setup work, not evidence that a registered capability is unsupported. Use metadata and DataCoolieDriver.run(...) for a supported path. Add custom code only around a verified unsupported boundary, record the evidence, and leave the supported remainder native. When platform execution context, path, credentials, or dependencies affect the combination, load references/platform-contract.md; platform is the adapter and does not imply the execution host.

3. Author durable sources

Use the canonical metadata contract and environment overlays; do not clone full metadata per environment. Use ordered selector patches for changes shared by matching canonical entities and exact keyed overrides for additions or final exceptions; exact overrides win. Global schema_hints selectors operate at connection/schema/table/column grain, while schema hints under a selected dataflow remain local to that dataflow. Load templates/project-structure.md and references/schema-quick-reference.md when authoring overlays. Create only required normal, replay, or maintenance entrypoints. The selected file fixes platform, engine, provider, and operation; runtime inputs carry only values allowed by the runner and operation contracts. Keep credentials in environment or platform secret services.

Resolve metadata, log, and watermark paths inside the environment's approved persistent control namespace and pass them unchanged. Deployed metadata is a build-scoped immutable projection; logs and watermarks remain mutable and outside build artifacts. For a cloud platform used by an on-premises runner, select the external runtime explicitly and keep the actual execution host separate from the platform adapter. Assume source query and action text can appear in framework logs. Do not embed secret literals; apply the approved log classification, access, and retention policy to generated runtime paths.

Treat a file or lakehouse connection base_path as the root; the framework appends each non-empty dataflow schema_name and table, so do not repeat those segments in the connection. Author exact source-observed, broad, or shared type mappings once in metadata/schema_hints.json. Use transform.schema_hints only for a small dataflow-specific cast or override. A non-empty transform hint set prevents global hints from being attached, so do not assume the two sources merge. Select the simplest native source address using the framework-boundary order. Before adding audit or partition helper columns, compare their semantics with framework-generated columns and native destination routing. Preserve distinct source/business timestamps, but do not duplicate framework write-time or driver-managed dataflow run identity unless an explicit consumer contract requires a separate named field. For flat-file load-time folder routing, prefer the destination connection's date_folder_partitions; use partition_columns for data-value routing. Use backward lookback primarily when discovery found no reliable change signal and a transaction/business-date column must bound late corrections; pair it with an idempotent load that can reprocess the window. For file sources, prefer __file_modification_time when file timestamps are reliable. Add source date_folder_partitions only for an observed year/month/day/hour path layout; it prunes folders and may be combined with file modification time. Destination date_folder_partitions is a separate load-time routing concern.

When Polars executes a Delta or Iceberg source.query, load references/polars-qualified-sql.md. Keep the SQL in normal metadata and register the required relations on the same active PolarsEngine before constructing or running the driver. Registration options are runner/bootstrap concerns, not source.configure fields.

4. Run fast source checks

Validate config and resolved metadata, lint affected paths, parse/compile entrypoints, and unit-test helpers directly. These checks give fast feedback but do not prove the generated build.

5. Materialize and verify

Run scripts/materialize.py; select single (default), split-connections, or split-all metadata layout explicitly when needed. It validates its inputs, renders typed metadata sets under each environment's fixed metadata/ component, packages zero or one function artifact under the fixed functions/ component, writes the manifest and checksums under .builds/artifacts/{build_id}, verifies the immutable bytes, and replaces .builds/current with a verified runnable projection of that whole build. The projection copies runtime files, omits artifact-only manifest/checksums, and records its exact source ID in current/build.json. Never symlink or mutate immutable artifact contents.

Always validate the immutable build, resolved metadata, exact runner, and optional functions artifact. Execute the generated runner on the Build host when that host is compatible and the approved check is safe; record the result as useful Build-host evidence, not target qualification. Do not block an artifact-qualified receipt solely because the runner requires staging on its target execution host. Release always qualifies the exact staged runner slice before activation.

Keep any Build-host logs and watermarks under persistent .runtime/{env}/ or another approved isolated test namespace. Apply the runner contract for normal runs and the operations contract for replay or maintenance, including their mutation confirmations.

Execute and validate .builds/current directly for the normal latest-build path. Select .builds/artifacts/{build_id} only for a historical version. Write a typed successful or failed receipt under .builds/evidence/{build_id}/{env}/{receipt_id}.json, using the exact ID from current/build.json when current was tested. Release never consumes the moving projection.

6. Add automation only when requested

Use scripts/render_automation.py only for requested reproducible project-owned build/CI entrypoints. Generated automation works with the installed framework and project sources without installed skills. Release owns consume-only deployment automation. Do not generate speculative automation.

Output And Handoff

{workspace}/config.yaml
{workspace}/metadata/
{workspace}/runners/
{workspace}/functions/                          # optional
{workspace}/automation/                         # optional
{workspace}/.builds/artifacts/{build_id}/manifest.json
{workspace}/.builds/artifacts/{build_id}/SHA256SUMS
{workspace}/.builds/artifacts/{build_id}/{env}/...
{workspace}/.builds/evidence/{build_id}/{env}/*.json
{workspace}/.builds/current/build.json
{workspace}/.builds/current/{env}/...
{workspace}/.builds/current/functions/            # when functions were packaged

Release may receive current as a convenience selector, but resolves current/build.json once and then consumes only the exact build ID, canonical local build directory or immutable remote artifact identity, manifest/checksums, target slice, and successful matching artifact-verification receipt. Build manifest v3 and Build receipt v4 bind the full metadata set and optional fixed-component function artifact. The receipt requires generated-artifact-validation; Build-host runtime execution is optional and never authorizes target activation. Build current is never a transfer source or authorization identity. Build or design approval never authorizes deployment. End with verification evidence, skipped checks, and unresolved questions.