Data Architecture Design (Framework + Router)
Architecture is decided by the objective and the constraints, not by tooling fashion. This skill runs the decision conversation top-down, then routes to the focused cloud and cost skills for the specifics. Don't restate cloud/pricing detail here — invoke the sub-skills.
Step 1 — Know the object first
Before anything, pin down what the platform is for. The object changes every downstream choice.
| Objective |
What it demands of the architecture |
| BI / analytics |
Batch-friendly, star/OBT models, warehouse-centric, cost per query matters, SQL consumers |
| ML |
Point-in-time-correct features, offline+online stores, reproducibility, feature freshness → hands to [[ml-feature-engineering]] |
| Both |
Shared curated layer feeding a BI mart and an ML feature layer — design the split explicitly |
| Operational / real-time |
Streaming backbone, low latency → hands to [[data-pipelines]] (streaming) |
Step 2 — Elicit requirements & constraints (ask, don't assume)
Work through these with the user explicitly. Missing answers are the #1 cause of wrong architecture.
- Deployment: cloud, on-prem, or hybrid? Any mandate (data residency, existing contracts)?
- Resource budget: how much compute can you actually run? How many concurrent jobs? Team size and skills (SQL-only? Spark? Kubernetes)?
- Cost ceiling: monthly budget, and is spend fixed (reserved) or variable (on-demand)?
- Latency / freshness SLA: real-time, hourly, daily? Drives batch vs. streaming.
- Volume & growth: current size and 12-month projection (from [[data-sourcing]] inventory + [[data-profiling]] sizing).
- Compliance: PII, PDPA/GDPR/HIPAA, audit → hands to [[data-reliability]] (governance).
On-prem vs. cloud — the short decision
- Cloud when: elastic/spiky load, small ops team, fast start, variable volume, want managed services.
- On-prem when: hard data-residency mandate, very stable predictable load at large scale where owned hardware beats rental, or existing heavy investment.
- Hybrid when: sensitive data must stay on-prem but burst compute is cheaper in cloud.
Step 3 — Design the layers, price each one
Design layer by layer. For each layer, choose the service and estimate cost — and this is where you route.
| Layer |
Purpose |
Decide |
Route to |
| Ingestion |
Get data in (batch/stream/CDC) |
Connector, batch vs. stream |
[[data-pipelines]] |
| Storage (raw) |
Durable landing zone |
Object store layout, format, partitioning |
cloud-infra-data |
| Processing |
Transform/curate |
Warehouse-native SQL vs. Spark vs. serverless |
cloud-infra-data |
| Serving |
BI marts / ML features / APIs |
Warehouse, feature store, cache |
cloud-infra-data, [[ml-feature-engineering]] |
| Cost of each |
Is this layer cost-effective? |
On-demand vs. reserved, tiering |
cost-optimization-data |
Routing is mandatory here: for any cloud service selection call Skill(skill="cloud-infra-data"); for any per-layer or whole-platform cost estimate call Skill(skill="cost-optimization-data"). Combine their output into the architecture — don't guess pricing or service limits from memory.
Step 4 — Cross-cloud cost-effectiveness
When the cloud isn't mandated, compare Azure / AWS / GCP per layer rather than picking a vendor wholesale — the cheapest storage and the cheapest compute are often on different clouds, but egress and integration cost usually favor consolidating. Get the concrete numbers from cost-optimization-data and the service mapping from cloud-infra-data, then recommend:
- A primary cloud (consolidation usually wins on egress + ops simplicity), and
- The warehouse/engine (BigQuery vs. Snowflake vs. Redshift vs. Synapse vs. Databricks) justified against the objective and budget.
Step 5 — Recommend & document
Deliver: a one-page architecture (layers + chosen services), the model recommendation (BI/ML/both), the constraint assumptions it rests on, and the estimated cost per layer. State what would change the recommendation (e.g. "if volume 10×, switch to reserved slots").
Hand-off
Feeds the build → [[data-pipelines]] and [[medallion-design]]; the model → [[data-modeling]]; governance → [[data-reliability]]. Upstream inputs come from [[data-sourcing]] and [[data-profiling]]. Lifecycle overview: [[data-lifecycle]].
1---2name: data-architecture3description: Data Architecture Design (Framework + Router)4---56# Data Architecture Design (Framework + Router)78Architecture is decided by the **objective and the constraints**, not by tooling fashion. This skill runs the decision conversation top-down, then routes to the focused cloud and cost skills for the specifics. Don't restate cloud/pricing detail here — **invoke the sub-skills**.910## Step 1 — Know the object first1112Before anything, pin down what the platform is *for*. The object changes every downstream choice.1314| Objective | What it demands of the architecture |15|---|---|16| **BI / analytics** | Batch-friendly, star/OBT models, warehouse-centric, cost per query matters, SQL consumers |17| **ML** | Point-in-time-correct features, offline+online stores, reproducibility, feature freshness → hands to [[ml-feature-engineering]] |18| **Both** | Shared curated layer feeding a BI mart and an ML feature layer — design the split explicitly |19| **Operational / real-time** | Streaming backbone, low latency → hands to [[data-pipelines]] (streaming) |2021## Step 2 — Elicit requirements & constraints (ask, don't assume)2223Work through these with the user explicitly. Missing answers are the #1 cause of wrong architecture.2425- **Deployment:** cloud, on-prem, or hybrid? Any mandate (data residency, existing contracts)?26- **Resource budget:** how much compute can you actually run? How many concurrent jobs? Team size and skills (SQL-only? Spark? Kubernetes)?27- **Cost ceiling:** monthly budget, and is spend fixed (reserved) or variable (on-demand)?28- **Latency / freshness SLA:** real-time, hourly, daily? Drives batch vs. streaming.29- **Volume & growth:** current size and 12-month projection (from [[data-sourcing]] inventory + [[data-profiling]] sizing).30- **Compliance:** PII, PDPA/GDPR/HIPAA, audit → hands to [[data-reliability]] (governance).3132### On-prem vs. cloud — the short decision3334- **Cloud** when: elastic/spiky load, small ops team, fast start, variable volume, want managed services.35- **On-prem** when: hard data-residency mandate, very stable predictable load at large scale where owned hardware beats rental, or existing heavy investment.36- **Hybrid** when: sensitive data must stay on-prem but burst compute is cheaper in cloud.3738## Step 3 — Design the layers, price each one3940Design layer by layer. For **each layer**, choose the service and estimate cost — and this is where you route.4142| Layer | Purpose | Decide | Route to |43|---|---|---|---|44| Ingestion | Get data in (batch/stream/CDC) | Connector, batch vs. stream | [[data-pipelines]] |45| Storage (raw) | Durable landing zone | Object store layout, format, partitioning | `cloud-infra-data` |46| Processing | Transform/curate | Warehouse-native SQL vs. Spark vs. serverless | `cloud-infra-data` |47| Serving | BI marts / ML features / APIs | Warehouse, feature store, cache | `cloud-infra-data`, [[ml-feature-engineering]] |48| Cost of each | Is this layer cost-effective? | On-demand vs. reserved, tiering | `cost-optimization-data` |4950**Routing is mandatory here:** for any cloud service selection call `Skill(skill="cloud-infra-data")`; for any per-layer or whole-platform cost estimate call `Skill(skill="cost-optimization-data")`. Combine their output into the architecture — don't guess pricing or service limits from memory.5152## Step 4 — Cross-cloud cost-effectiveness5354When the cloud isn't mandated, compare Azure / AWS / GCP **per layer** rather than picking a vendor wholesale — the cheapest storage and the cheapest compute are often on different clouds, but egress and integration cost usually favor consolidating. Get the concrete numbers from `cost-optimization-data` and the service mapping from `cloud-infra-data`, then recommend:5556- A **primary cloud** (consolidation usually wins on egress + ops simplicity), and57- The **warehouse/engine** (BigQuery vs. Snowflake vs. Redshift vs. Synapse vs. Databricks) justified against the objective and budget.5859## Step 5 — Recommend & document6061Deliver: a one-page architecture (layers + chosen services), the model recommendation (BI/ML/both), the constraint assumptions it rests on, and the estimated cost per layer. State what would change the recommendation (e.g. "if volume 10×, switch to reserved slots").6263## Hand-off6465Feeds the build → [[data-pipelines]] and [[medallion-design]]; the model → [[data-modeling]]; governance → [[data-reliability]]. Upstream inputs come from [[data-sourcing]] and [[data-profiling]]. Lifecycle overview: [[data-lifecycle]].
Methasit-Pun