Platxa Monitoring

Platxa Monitoring

Guide for observability in the Platxa platform using Prometheus metrics and Loki logs.

Overview

This skill covers the complete observability stack:

Prerequisites

Verify monitoring stack is running:

kubectl get pods -n monitoring
# prometheus-*, loki-*, grafana-*, fluent-bit-*

# Access Grafana locally
kubectl port-forward svc/grafana 3000:80 -n monitoring

Prometheus Metrics

Common PromQL Queries

Instance Resource Metrics

# Memory usage (bytes)
container_memory_working_set_bytes{
  namespace="instance-{name}",
  container="odoo"
}

# CPU usage (millicores)
sum(rate(container_cpu_usage_seconds_total{
  namespace="instance-{name}",
  container="odoo"
}[5m])) * 1000

# Storage usage (ratio)
kubelet_volume_stats_used_bytes{namespace="instance-{name}"}
/
kubelet_volume_stats_capacity_bytes{namespace="instance-{name}"}

Waking Service Metrics

# Instance state distribution
waking_instances_by_state{state="running"}
waking_instances_by_state{state="sleeping"}
waking_instances_by_state{state="waking"}
waking_instances_by_state{state="error"}

# Total tracked instances
waking_instances_total

PostgreSQL Metrics

# Database size
pg_database_size{datname=~"instance_.*"}

# Active connections per database
sum by (datname) (pg_stat_activity_count{state="active"})

# Slow queries (>30s)
pg_slow_queries_count

Recording Rules

Pre-computed metrics for dashboard efficiency:

ServiceMonitors

Automatic scrape targets via Prometheus Operator:

Loki Logs

LogQL Query Patterns

Basic Label Filtering

# All logs from an instance
{namespace="instance-abc123xy"}

# Specific container
{namespace="instance-abc123xy", container="odoo"}

# Multiple namespaces (regex)
{namespace=~"instance-.*"}

Pattern Matching

# Contains error (case insensitive)
{namespace=~"instance-.*"} |~ "(?i)error"

# Exact match
{namespace=~"instance-.*"} |= "FATAL"

# Exclude pattern
{namespace=~"instance-.*"} != "healthcheck"

# Regex pattern
{namespace=~"instance-.*"} |~ "connection refused|timeout"

Aggregations

# Error count over time
count_over_time({namespace="instance-abc123xy"} |~ "ERROR" [5m])

# Error rate per minute
rate({namespace=~"instance-.*"} |~ "ERROR" [1m])

# Top namespaces by log volume
topk(10, sum by (namespace) (rate({namespace=~"instance-.*"}[5m])))

Log Labels

Fluent Bit enriches logs with Kubernetes metadata:

Multiline Log Handling

Python stack traces are automatically combined:

# Fluent Bit parser detects:
# - "Traceback (most recent call last):"
# - Indented continuation lines
# - "Error:", "Exception:", "Warning:"

Alerting

Alert Categories

Infrastructure Alerts

Instance Alerts

Database Alerts

Log-Based Alerts (Loki)

Alertmanager Routing

Routes:
  critical → platform-odoo (webhook, 10s group wait)
  warning  → platform-odoo (30s group wait)
  null     → silence (Watchdog, informational)

Webhook: Bearer token auth to Odoo platform
Grouping: By namespace and alertname

Workflow

Investigating Issues

Step 1: Identify Issue Type

Step 2: Scope to Target

# Find instance namespace
kubectl get ns -l platxa.io/tier=instance | grep {name}

# Identify components
kubectl get pods -n instance-{name}

Step 3: Query Data

Prometheus (via Grafana or API):

# API query
curl -G 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=instance:memory_usage:ratio{namespace="instance-abc123xy"}'

Loki (via Grafana or API):

# API query
curl -G 'http://localhost:3100/loki/api/v1/query' \
  --data-urlencode 'query={namespace="instance-abc123xy"} |~ "ERROR"'

Step 4: Correlate

• Match metric spikes with log timestamps
• Check alert history in Alertmanager
• Review recent events: kubectl get events -n instance-{name}

Step 5: Recommend Actions

• Use platxa-k8s-ops for operational commands
• Suggest resource adjustments
• Link to relevant troubleshooting guides

Examples

Example 1: Memory Investigation

User: "Instance abc123xy is running slow"

Response:

Check memory usage:

instance:memory_usage:ratio{namespace="instance-abc123xy"}
# Result: 0.92 (92% of limit)

Check for OOM events:

{namespace="instance-abc123xy"} |~ "OOM|killed|memory"

Recent restarts:

instance:restarts:1h{namespace="instance-abc123xy"}
# Result: 2 restarts

Finding: Memory pressure causing slowdown. Recommend increasing memory limit or reducing Odoo workers.

Example 2: Error Log Analysis

User: "Users report 500 errors on demo instance"

Response:

Search recent errors:

{namespace="instance-demo", container="odoo"}
  |~ "ERROR|Exception|Traceback"
  | json
  | line_format "{{.message}}"

Count errors by type:

sum by (level) (
  count_over_time({namespace="instance-demo"} |~ "ERROR|WARNING" [1h])
)

Check database connectivity:

{namespace="instance-demo"} |~ "could not connect|connection refused"

Finding: Database connection errors detected. Check PostgreSQL status with platxa-k8s-ops.

Example 3: Alert Investigation

User: "Got alert for OdooStorageCritical"

Response:

Check current storage usage:

instance:storage_usage:ratio{namespace=~"instance-.*"} > 0.90

Identify which PVC is full:

kubelet_volume_stats_used_bytes{namespace="instance-{name}"}
/
kubelet_volume_stats_capacity_bytes{namespace="instance-{name}"}

Results:

• odoo-{name}-addons: 45% used
• odoo-{name}-filestore: 97% used (CRITICAL)

Finding: Filestore PVC near capacity. Options:

1. Clean up old attachments
2. Expand PVC (if storage class supports)
3. Upgrade instance tier for larger storage

Example 4: Cold Start Performance

User: "Instances taking too long to wake"

Response:

Check waking service metrics:

# Current instance states
waking_instances_by_state

# Cold start timing (if exposed)
histogram_quantile(0.95, waking_cold_start_duration_bucket)

Check waking service logs:

{namespace="traefik-system", container="waking-service"}
  |~ "cold start|wake|scale"
  | json

Common causes:

• Large filestore extraction time
• Resource scheduling delays
• Init container timeouts

Grafana Dashboards

Pre-built dashboards available:

Access: Grafana → Dashboards → Browse

Troubleshooting

No Metrics Data

No Logs in Loki

Alerts Not Firing

Output Checklist

After monitoring investigation:

• Relevant PromQL/LogQL query constructed
• Time range appropriate for issue
• Results interpreted correctly
• Metrics and logs correlated
• Root cause identified or narrowed
• Actionable recommendations provided
• Follow-up monitoring suggested

Related Resources

• PromQL Queries: See references/promql-queries.md
• LogQL Queries: See references/logql-queries.md
• Alert Rules: See references/alert-rules.md
• K8s Operations: Use platxa-k8s-ops skill