Snowflake Native Pipelines And Governance
Overview
Use this skill when Snowflake is not just the storage target but the primary pipeline and governance surface. It helps agents reason about native ingestion, incremental patterns, task orchestration, governance controls, warehouse sizing, and secure publish paths inside Snowflake.
When to Use
- building warehouse-native incremental pipelines in
Snowflake
- designing with
Streams, Tasks, or Dynamic Tables
- evaluating
Snowpipe, Snowpark, or external orchestration boundaries
- defining masking, row access, tags, and secure sharing in the same workflow
- reviewing publish readiness for Snowflake-native data products
Do not assume Snowflake-native is automatically the right answer for every compute and orchestration step.
Workflow
Define the execution boundary.
Decide what should stay inside Snowflake versus what belongs in upstream landing, external orchestration, or other compute services.
Choose the incremental pattern.
Consider:
StreamsTasksDynamic TablesSnowpipe- external orchestration when boundaries are broader than Snowflake alone
Design governance and access together.
Include:
- roles
- masking policies
- row access policies
- tags and classification
- secure publish or sharing boundaries
Make warehouse operations explicit.
Cover:
- warehouse sizing
- concurrency and isolation
- cost controls
- retry and rerun behavior
- validation before publish
Define publish and recovery behavior.
Require:
- reconciliation or validation gates
- controlled reopen after failure
- rollback or forward-fix plan for warehouse-native publishes
Common Rationalizations
| Rationalization |
Reality |
| "Because the data is in Snowflake, the pipeline should also live there." |
Some landing, preprocessing, orchestration, or resilience controls still belong outside the warehouse. |
| "Tasks prove the pipeline is production-ready." |
Task success alone does not prove contract correctness, reconciliation, or secure publish behavior. |
| "Masking can be added after the model is stable." |
Governance controls are part of the design, not cleanup after delivery. |
Red Flags
- Snowflake-native execution is chosen without explicit boundary reasoning
- incremental behavior is vague across
Streams, Tasks, or Dynamic Tables
- warehouse cost and concurrency implications are not documented
- secure-sharing or publish controls are undefined
- governance is described in docs but not reflected in Snowflake-native policies
Verification
1---2name: snowflake-native-pipelines-and-governance3description: Guides agents through Snowflake-native pipeline and governance workflows. Use when building or reviewing Snowflake pipelines with Streams, Tasks, Dynamic Tables, Snowpipe, Snowpark, masking policies, row access, secure sharing, and warehouse-native operational controls.4---56# Snowflake Native Pipelines And Governance78## Overview910Use this skill when `Snowflake` is not just the storage target but the primary pipeline and governance surface. It helps agents reason about native ingestion, incremental patterns, task orchestration, governance controls, warehouse sizing, and secure publish paths inside Snowflake.1112## When to Use1314- building warehouse-native incremental pipelines in `Snowflake`15- designing with `Streams`, `Tasks`, or `Dynamic Tables`16- evaluating `Snowpipe`, `Snowpark`, or external orchestration boundaries17- defining masking, row access, tags, and secure sharing in the same workflow18- reviewing publish readiness for Snowflake-native data products1920Do not assume Snowflake-native is automatically the right answer for every compute and orchestration step.2122## Workflow23241. Define the execution boundary.25 Decide what should stay inside `Snowflake` versus what belongs in upstream landing, external orchestration, or other compute services.26272. Choose the incremental pattern.28 Consider:29 - `Streams`30 - `Tasks`31 - `Dynamic Tables`32 - `Snowpipe`33 - external orchestration when boundaries are broader than Snowflake alone34353. Design governance and access together.36 Include:37 - roles38 - masking policies39 - row access policies40 - tags and classification41 - secure publish or sharing boundaries42434. Make warehouse operations explicit.44 Cover:45 - warehouse sizing46 - concurrency and isolation47 - cost controls48 - retry and rerun behavior49 - validation before publish50515. Define publish and recovery behavior.52 Require:53 - reconciliation or validation gates54 - controlled reopen after failure55 - rollback or forward-fix plan for warehouse-native publishes5657## Common Rationalizations5859| Rationalization | Reality |60| --- | --- |61| "Because the data is in Snowflake, the pipeline should also live there." | Some landing, preprocessing, orchestration, or resilience controls still belong outside the warehouse. |62| "Tasks prove the pipeline is production-ready." | Task success alone does not prove contract correctness, reconciliation, or secure publish behavior. |63| "Masking can be added after the model is stable." | Governance controls are part of the design, not cleanup after delivery. |6465## Red Flags6667- Snowflake-native execution is chosen without explicit boundary reasoning68- incremental behavior is vague across `Streams`, `Tasks`, or `Dynamic Tables`69- warehouse cost and concurrency implications are not documented70- secure-sharing or publish controls are undefined71- governance is described in docs but not reflected in Snowflake-native policies7273## Verification7475- [ ] The Snowflake execution boundary is explicit and justified76- [ ] Incremental and orchestration behavior is designed intentionally77- [ ] Governance and access controls are defined with the pipeline design78- [ ] Warehouse sizing, cost, and isolation considerations are documented79- [ ] Publish, recovery, and validation behavior are explicit
vaquarkhan