[H1][OBSERVABILITY-STACK]
Dictum: Infrastructure generation follows canonical deploy.ts patterns.
Stack: Alloy -> Prometheus -> Grafana (metrics only). Logs/traces -> otelcol.exporter.debug (discarded).
Runtime: OTEL_LOGS_EXPORTER=none, OTEL_TRACES_EXPORTER=none, OTEL_METRICS_EXPORTER=otlp (deploy.ts:95-97).
Versions: Alloy 1.13.0 (OTel Collector v0.142.0), Prometheus 3.5.1+ (native histograms stable in 3.8+, flag no-op in 3.9+), Grafana 12.3.2.
| [INDEX] |
[USE_THIS_SKILL] |
[USE_OTHER_SKILL] |
| [1] |
Deploy observability via Pulumi. |
pulumi-k8s-generator: Non-observability K8s resources. |
| [2] |
OTEL collector configs (Alloy). |
k8s-debug: Debug deployed observability pods. |
| [3] |
Scrape/alert rules, dashboards. |
pulumi-k8s-validator: Audit existing infra code. |
| [4] |
Mode migration cloud<->selfhosted. |
promql-generator: PromQL query construction. |
Tasks:
- Read all three
references/ files before generating.
- Gather: mode (cloud/selfhosted), retention, HA, TLS via AskUserQuestion (ambiguous only).
- Generate Pulumi TypeScript resources following
deploy.ts patterns.
- Validate:
pnpm exec nx run infrastructure:typecheck then pulumi preview --diff.
[1][CANONICAL_IMPLEMENTATION]
Dictum: deploy.ts is the single source of truth for resource patterns.
Canonical: infrastructure/src/deploy.ts (207 LOC).
| [INDEX] |
[LINES] |
[SYMBOL] |
[PURPOSE] |
| [1] |
14 |
_CONFIG.images |
Image tags (:latest -- pin for production). |
| [2] |
22 |
_CONFIG.ports |
alloyGrpc:4317, alloyHttp:4318, alloyMetrics:12345, prom:9090. |
| [3] |
29-36 |
_Ops.alloy(promUrl) |
Alloy config: OTLP receiver, prometheus.remote_write, debug. |
| [4] |
59 |
_Ops.grafana(promUrl) |
Prometheus datasource provisioning YAML. |
| [5] |
63 |
_Ops.meta(ns,component,n?) |
Metadata factory with stack: 'parametric', tier: 'observe'. |
| [6] |
71 |
_Ops.prometheus(alloyHost) |
Scrape config: 15s interval, alloy:12345 + localhost:9090. |
| [7] |
120-126 |
_k8sObserve(ns, items) |
Array-driven factory: PVC + ConfigMap + Deployment + Service. |
[1.1][DATA_FLOW]
App (Effect OTEL) --OTLP--> Alloy(:4317/:4318) --remote_write--> Prometheus(:9090)
|--debug sink--> (logs discarded)
\--debug sink--> (traces discarded)
Grafana(:3000) --PromQL--> Prometheus(:9090)
[2][GENERATION]
Dictum: Cloud and selfhosted share topology; differ in resource types.
| [INDEX] |
[MODE] |
[INFRASTRUCTURE] |
[LINES] |
[USE_CASE] |
| [1] |
cloud |
K8s (EKS/GKE/AKS). |
131-176 |
Production, multi-node, HA. |
| [2] |
selfhosted |
Docker containers. |
178-194 |
Dev, single-node, staging. |
Cloud (deploy.ts:147-154):
- Alloy: ConfigMap + DaemonSet + Service (3 ports: grpc:4317, http:4318, metrics:12345). Resources:
200m/256Mi limits, 100m/128Mi requests.
_k8sObserve array: prometheus (cmd flags, 9090, PVC) + grafana (datasource provisioning, 3000, PVC).
Selfhosted (deploy.ts:187-189):
- 3
docker.Container resources with uploads (config) + named volumes via _Ops.dockerVol.
- Grafana:
GF_SECURITY_ADMIN_PASSWORD from _Ops.secret(). Shared network via _Ops.dockerNets(networkId).
All resources: Pulumi TypeScript. Never raw YAML, Helm, or docker-compose.
[3][VALIDATION]
Dictum: Validation gates prevent broken observability pipelines.
| [INDEX] |
[CHECK] |
[SCOPE] |
| [1] |
Alloy OTLP ports 4317/4318 exposed. |
All. |
| [2] |
Alloy metrics port 12345 exposed. |
All. |
| [3] |
Prometheus scrapes all components. |
All. |
| [4] |
Grafana datasource provisioned. |
All. |
| [5] |
Resource requests/limits set. |
Cloud. |
| [6] |
Named volumes for stateful data. |
Selfhosted. |
| [7] |
TLS for inter-component traffic. |
Cloud (prod). |
[4][PRODUCTION]
Dictum: Production gaps require explicit acknowledgment.
| [INDEX] |
[CONCERN] |
[STATUS] |
[RECOMMENDATION] |
| [1] |
TLS |
Not configured. |
cert-manager or AWS ACM for inter-component comms. |
| [2] |
Security contexts |
Missing on observe pods. |
runAsNonRoot: true, readOnlyRootFilesystem: true. |
| [3] |
Image tags |
:latest (deploy.ts:14). |
Pin to specific versions for production. |
| [4] |
NetworkPolicies |
Not defined. |
Restrict Alloy<-App, Prometheus<-Alloy, Grafana->Prom. |
| [5] |
Resource limits |
Alloy only. |
Add to Prometheus/Grafana via _k8sObserve extension. |
[5][DEPRECATIONS]
Dictum: Deprecated components require migration paths.
| [INDEX] |
[DEPRECATED] |
[REPLACEMENT] |
[STATUS] |
| [1] |
Promtail |
Alloy. |
EOL March 2, 2026. |
| [2] |
lokiexporter |
otlphttp. |
Removed OTEL 0.140+. |
| [3] |
Standalone OTEL Collector |
Alloy (includes v0.142.0). |
Superseded. |
| [4] |
River syntax (name) |
"Alloy configuration syntax". |
Renamed Alloy 1.8+. |
[6][EXTENSION_LOKI_TEMPO]
Dictum: Full 3-signal integration extends the metrics-only default.
| [INDEX] |
[SIGNAL] |
[IMAGE] |
[ALLOY_CHANGE] |
[RUNTIME_ENV] |
| [1] |
Logs (Loki) |
grafana/loki:3.6.x. |
debug -> loki exporter. |
OTEL_LOGS_EXPORTER=otlp. |
| [2] |
Traces (Tempo) |
grafana/tempo:2.7.x. |
debug -> otlp exporter to Tempo. |
OTEL_TRACES_EXPORTER=otlp. |
[REFERENCE] references/stack_architecture.md#extension-adding-loki--tempo.
[7][REFERENCES]
Dictum: Reference files provide detailed implementation patterns.
references/stack_architecture.md -- Topology, sizing, Prometheus version matrix, Loki/Tempo extension.references/alert_rules.md -- Alert templates, recording rules (classic + native), ConfigMap provisioning.references/grafana_dashboards.md -- Dashboard panels (Grafana 12.3+, schema v2), native histogram heatmap.
1---2name: observability-stack-23description: Generates Pulumi TypeScript observability infrastructure for Grafana Alloy 1.13, Prometheus 3.5+, Grafana 12.3 across K8s/Docker. Use when deploying metrics stacks, provisioning alert rules/dashboards, or migrating cloud↔selfhosted observability.4---5
6# [H1][OBSERVABILITY-STACK]
7>**Dictum:** *Infrastructure generation follows canonical deploy.ts patterns.*
8
9<br>
10
11Stack: Alloy -> Prometheus -> Grafana (metrics only). Logs/traces -> `otelcol.exporter.debug` (discarded).
12Runtime: `OTEL_LOGS_EXPORTER=none`, `OTEL_TRACES_EXPORTER=none`, `OTEL_METRICS_EXPORTER=otlp` (deploy.ts:95-97).
13Versions: Alloy 1.13.0 (OTel Collector v0.142.0), Prometheus 3.5.1+ (native histograms stable in 3.8+, flag no-op in 3.9+), Grafana 12.3.2.
14
15| [INDEX] | [USE_THIS_SKILL] | [USE_OTHER_SKILL] |
16| :-----: | -------------------------------------- | ---------------------------------------------------------- |
17| [1] | **Deploy observability via Pulumi.** | **pulumi-k8s-generator**: Non-observability K8s resources. |
18| [2] | **OTEL collector configs (Alloy).** | **k8s-debug**: Debug deployed observability pods. |
19| [3] | **Scrape/alert rules, dashboards.** | **pulumi-k8s-validator**: Audit existing infra code. |
20| [4] | **Mode migration cloud<->selfhosted.** | **promql-generator**: PromQL query construction. |
21
22**Tasks:**
231. Read all three `references/` files before generating.
242. Gather: mode (cloud/selfhosted), retention, HA, TLS via **AskUserQuestion** (ambiguous only).
253. Generate Pulumi TypeScript resources following `deploy.ts` patterns.
264. Validate: `pnpm exec nx run infrastructure:typecheck` then `pulumi preview --diff`.
27
28---
29## [1][CANONICAL_IMPLEMENTATION]
30>**Dictum:** *deploy.ts is the single source of truth for resource patterns.*
31
32<br>
33
34Canonical: `infrastructure/src/deploy.ts` (207 LOC).
35
36| [INDEX] | [LINES] | [SYMBOL] | [PURPOSE] |
37| :-----: | ----------- | ---------------------------- | --------------------------------------------------------------- |
38| [1] | **14** | `_CONFIG.images` | Image tags (`:latest` -- pin for production). |
39| [2] | **22** | `_CONFIG.ports` | alloyGrpc:4317, alloyHttp:4318, alloyMetrics:12345, prom:9090. |
40| [3] | **29-36** | `_Ops.alloy(promUrl)` | Alloy config: OTLP receiver, `prometheus.remote_write`, debug. |
41| [4] | **59** | `_Ops.grafana(promUrl)` | Prometheus datasource provisioning YAML. |
42| [5] | **63** | `_Ops.meta(ns,component,n?)` | Metadata factory with `stack: 'parametric'`, `tier: 'observe'`. |
43| [6] | **71** | `_Ops.prometheus(alloyHost)` | Scrape config: 15s interval, alloy:12345 + localhost:9090. |
44| [7] | **120-126** | `_k8sObserve(ns, items)` | Array-driven factory: PVC + ConfigMap + Deployment + Service. |
45
46<br>
47
48### [1.1][DATA_FLOW]
49
50```
51App (Effect OTEL) --OTLP--> Alloy(:4317/:4318) --remote_write--> Prometheus(:9090)
52 |--debug sink--> (logs discarded)
53 \--debug sink--> (traces discarded)
54Grafana(:3000) --PromQL--> Prometheus(:9090)
55```
56
57---
58## [2][GENERATION]
59>**Dictum:** *Cloud and selfhosted share topology; differ in resource types.*
60
61<br>
62
63| [INDEX] | [MODE] | [INFRASTRUCTURE] | [LINES] | [USE_CASE] |
64| :-----: | ---------------- | ------------------ | ------- | --------------------------- |
65| [1] | **`cloud`** | K8s (EKS/GKE/AKS). | 131-176 | Production, multi-node, HA. |
66| [2] | **`selfhosted`** | Docker containers. | 178-194 | Dev, single-node, staging. |
67
68**Cloud (deploy.ts:147-154):**
69- Alloy: ConfigMap + DaemonSet + Service (3 ports: grpc:4317, http:4318, metrics:12345). Resources: `200m/256Mi` limits, `100m/128Mi` requests.
70- `_k8sObserve` array: prometheus (cmd flags, 9090, PVC) + grafana (datasource provisioning, 3000, PVC).
71
72**Selfhosted (deploy.ts:187-189):**
73- 3 `docker.Container` resources with `uploads` (config) + named volumes via `_Ops.dockerVol`.
74- Grafana: `GF_SECURITY_ADMIN_PASSWORD` from `_Ops.secret()`. Shared network via `_Ops.dockerNets(networkId)`.
75
76All resources: Pulumi TypeScript. Never raw YAML, Helm, or docker-compose.
77
78---
79## [3][VALIDATION]
80>**Dictum:** *Validation gates prevent broken observability pipelines.*
81
82<br>
83
84| [INDEX] | [CHECK] | [SCOPE] |
85| :-----: | --------------------------------------- | ------------- |
86| [1] | **Alloy OTLP ports 4317/4318 exposed.** | All. |
87| [2] | **Alloy metrics port 12345 exposed.** | All. |
88| [3] | **Prometheus scrapes all components.** | All. |
89| [4] | **Grafana datasource provisioned.** | All. |
90| [5] | **Resource requests/limits set.** | Cloud. |
91| [6] | **Named volumes for stateful data.** | Selfhosted. |
92| [7] | **TLS for inter-component traffic.** | Cloud (prod). |
93
94---
95## [4][PRODUCTION]
96>**Dictum:** *Production gaps require explicit acknowledgment.*
97
98<br>
99
100| [INDEX] | [CONCERN] | [STATUS] | [RECOMMENDATION] |
101| :-----: | --------------------- | ------------------------- | ------------------------------------------------------ |
102| [1] | **TLS** | Not configured. | cert-manager or AWS ACM for inter-component comms. |
103| [2] | **Security contexts** | Missing on observe pods. | `runAsNonRoot: true`, `readOnlyRootFilesystem: true`. |
104| [3] | **Image tags** | `:latest` (deploy.ts:14). | Pin to specific versions for production. |
105| [4] | **NetworkPolicies** | Not defined. | Restrict Alloy<-App, Prometheus<-Alloy, Grafana->Prom. |
106| [5] | **Resource limits** | Alloy only. | Add to Prometheus/Grafana via `_k8sObserve` extension. |
107
108---
109## [5][DEPRECATIONS]
110>**Dictum:** *Deprecated components require migration paths.*
111
112<br>
113
114| [INDEX] | [DEPRECATED] | [REPLACEMENT] | [STATUS] |
115| :-----: | ----------------------------- | ----------------------------- | -------------------- |
116| [1] | **Promtail** | Alloy. | EOL March 2, 2026. |
117| [2] | **`lokiexporter`** | `otlphttp`. | Removed OTEL 0.140+. |
118| [3] | **Standalone OTEL Collector** | Alloy (includes v0.142.0). | Superseded. |
119| [4] | **River syntax (name)** | "Alloy configuration syntax". | Renamed Alloy 1.8+. |
120
121---
122## [6][EXTENSION_LOKI_TEMPO]
123>**Dictum:** *Full 3-signal integration extends the metrics-only default.*
124
125<br>
126
127| [INDEX] | [SIGNAL] | [IMAGE] | [ALLOY_CHANGE] | [RUNTIME_ENV] |
128| :-----: | ------------------ | ---------------------- | -------------------------------- | ---------------------------- |
129| [1] | **Logs (Loki)** | `grafana/loki:3.6.x`. | debug -> loki exporter. | `OTEL_LOGS_EXPORTER=otlp`. |
130| [2] | **Traces (Tempo)** | `grafana/tempo:2.7.x`. | debug -> otlp exporter to Tempo. | `OTEL_TRACES_EXPORTER=otlp`. |
131
132[REFERENCE] `references/stack_architecture.md#extension-adding-loki--tempo`.
133
134---
135## [7][REFERENCES]
136>**Dictum:** *Reference files provide detailed implementation patterns.*
137
138<br>
139
140- `references/stack_architecture.md` -- Topology, sizing, Prometheus version matrix, Loki/Tempo extension.
141- `references/alert_rules.md` -- Alert templates, recording rules (classic + native), ConfigMap provisioning.
142- `references/grafana_dashboards.md` -- Dashboard panels (Grafana 12.3+, schema v2), native histogram heatmap.
majiayu000