Kubernetes Istio

Kubernetes Istio Service Mesh Manager

Implements Istio service mesh patterns for advanced traffic management, security, and observability in Kubernetes clusters. When loaded, the model generates Istio CRDs (VirtualService, DestinationRule, ServiceEntry, PeerAuthentication, AuthorizationPolicy) for traffic splitting, circuit breaking, retries, and mutual TLS enforcement.

TL;DR Checklist

• Apply Istio control plane (Istiod) via Helm or Istio Operator before creating any mesh resources
• Enable sidecar injection with sidecar.istio.io/inject: "true" annotation or namespace-level default
• Always define a DestinationRule alongside every VirtualService for traffic policies
• Use trafficPolicy.tls.mode: ISTIO_MUTUAL for mTLS — never plaintext in production
• Configure circuit breaker thresholds (maxConnections, httpMaxPendingRequests, consecutive5xxErrors) in DestinationRules
• Validate mesh connectivity with istioctl analyze before deploying traffic rules

When to Use

Use this skill when:

• Implementing canary or blue-green deployments with precise traffic splitting (e.g., 90% stable, 10% canary)
• Configuring circuit breaking to prevent cascade failures across microservices
• Enabling automatic retries with exponential backoff for transient failures
• Implementing mutual TLS (mTLS) between services for zero-trust security
• Adding request timeouts, rate limiting, or fault injection for resilience testing
• Routing traffic based on headers, weights, or source identities (not just host/path)

When NOT to Use

Avoid this skill for:

• Simple HTTP routing at the edge — use kubernetes-ingress instead
• Basic service discovery and load balancing within Kubernetes — use native ClusterIP Services
• Pod-level networking policies — use kubernetes-networkpolicy for L3/L4 rules
• Applications that cannot run sidecar containers — Istio requires the Istio proxy sidecar on every pod
• Clusters without Istio control plane installed — VirtualService and DestinationRule CRDs will not be recognized
• Stateful workloads needing stable network identities — use kubernetes-statefulset for that concern

Core Workflow

1. Verify Istio Control Plane — Confirm Istiod is running and sidecar injection is enabled in the target namespace. Checkpoint: Run kubectl get pods -n istio-system and verify istiod is Running. Run kubectl get namespace -L istio-injection to verify injection labels.

2. Define DestinationRule — Create a DestinationRule that sets the subset labels, connection pool limits, and TLS mode for the target service. Checkpoint: The trafficPolicy.tls.mode must be ISTIO_MUTUAL for mTLS or DISABLE for unencrypted — never use SIMPLE (deprecated in Istio 1.7+).

3. Define VirtualService — Create a VirtualService that routes traffic to subsets defined in the DestinationRule, using weight-based splitting, header matching, or fault injection. Checkpoint: Every route destination must reference a valid subset name defined in a corresponding DestinationRule.

4. Apply PeerAuthentication — Configure namespace-wide or global mTLS policy (STRICT for production, PERMISSIVE for transition periods). Checkpoint: STRICT mode rejects all non-mTLS traffic; verify no services are broken by the policy before applying cluster-wide.

5. Apply and Validate — Apply all manifests and run istioctl analyze to detect misconfigurations. Checkpoint: Confirm istioctl analyze reports zero warnings and that istioctl proxy-config shows the expected routes.

6. Verify Traffic Splitting — Send test requests to the ingress point and confirm the expected distribution across subsets. Checkpoint: Use curl or a load generator to send enough requests to verify the weight split matches the VirtualService configuration.

Implementation Patterns

Pattern 1: Canary Deployment with 10% Traffic Split

A VirtualService and DestinationRule for routing 10% of traffic to a canary subset and 90% to stable.

# DestinationRule: defines subsets for stable and canary versions
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: frontend-destination
  namespace: production
spec:
  host: frontend.production.svc.cluster.local
  subsets:
    - name: stable
      labels:
        version: v1
    - name: canary
      labels:
        version: v2
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 60s
      maxEjectionPercent: 50

---
# VirtualService: splits traffic 90/10 between stable and canary
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: frontend-routing
  namespace: production
spec:
  hosts:
    - frontend.production.svc.cluster.local
  http:
    - route:
        - destination:
            host: frontend.production.svc.cluster.local
            subset: stable
          weight: 90
        - destination:
            host: frontend.production.svc.cluster.local
            subset: canary
          weight: 10

Pattern 2: Circuit Breaking and Retry Configuration (BAD vs GOOD)

Improper circuit breaker and retry settings can either mask failures or cause excessive backpressure.

# ❌ BAD — No circuit breaker; cascading failures will propagate unbounded
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: backend-dr-bad
  namespace: production
spec:
  host: backend.production.svc.cluster.local
  subsets:
    - name: default
      labels:
        app: backend
  # No trafficPolicy at all — no connection limits, no outlier detection
  # ❌ No retries configured in VirtualService — transient failures are fatal

# ✅ GOOD — Circuit breaker prevents cascade, retries handle transient failures
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: backend-dr-good
  namespace: production
spec:
  host: backend.production.svc.cluster.local
  subsets:
    - name: default
      labels:
        app: backend
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100          # Hard cap on connections to the upstream
      http:
        http1MaxPendingRequests: 100  # Buffer for queued HTTP requests
        http2MaxRequests: 1000
    outlierDetection:
      consecutive5xxErrors: 3         # Eject after 3 consecutive 5xx responses
      interval: 10s                   # Check every 10 seconds
      baseEjectionTime: 30s           # Eject for 30 seconds
      maxEjectionPercent: 50          # Eject at most half the hosts

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: backend-routing
  namespace: production
spec:
  hosts:
    - backend.production.svc.cluster.local
  http:
    - route:
        - destination:
            host: backend.production.svc.cluster.local
            subset: default
      retries:
        attempts: 3                   # Retry up to 3 times
        perTryTimeout: 2s            # Each retry attempt has a 2s timeout
        retryOn: gateway-error,connect-failure,refused-stream

Pattern 3: mTLS Enforcement with Peer and Authorization Policies

# PeerAuthentication: enforce mTLS namespace-wide
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: production-mtls
  namespace: production
spec:
  mtls:
    mode: STRICT    # Reject all plaintext traffic

---
# AuthorizationPolicy: allow frontend to call backend on /api/* only
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: backend-authz
  namespace: production
spec:
  selector:
    matchLabels:
      app: backend
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/production/sa/frontend"]
      to:
        - operation:
            paths: ["/api/*"]
            methods: ["GET", "POST"]

---
# ServiceEntry: allow outbound traffic to external payment API
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-payment-api
  namespace: production
spec:
  hosts:
    - api.payment-provider.com
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL

Pattern 4: Header-Based Routing for A/B Testing

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ab-test-routing
  namespace: production
spec:
  hosts:
    - checkout.production.svc.cluster.local
  http:
    - match:
        - headers:
            x-ab-group:
              exact: group-b
      route:
        - destination:
            host: checkout.production.svc.cluster.local
            subset: group-b
          weight: 100
    - route:
        - destination:
            host: checkout.production.svc.cluster.local
            subset: group-a
          weight: 100

Constraints

MUST DO

• Always create a DestinationRule for every service that will receive routed traffic — VirtualService subsets must be defined in a corresponding DestinationRule
• Set trafficPolicy.tls.mode: ISTIO_MUTUAL in DestinationRules for mTLS between mesh services
• Configure outlierDetection with consecutive5xxErrors (3–5) and baseEjectionTime (30–60s) in DestinationRules for circuit breaking
• Set retries.attempts to 1–3 and retries.perTryTimeout to 1–5s in VirtualService for retry policies
• Use PeerAuthentication with mode: STRICT in production namespaces to enforce mTLS
• Label pods with version or release labels so subsets can match them accurately
• Run istioctl analyze -A after applying any Istio CRD to catch configuration errors
• Set connectionPool.tcp.maxConnections and connectionPool.http.http1MaxPendingRequests to cap backpressure

MUST NOT DO

• Never set PeerAuthentication mode to STRICT without verifying all services can use mTLS first — it will cut off plaintext communication instantly
• Never create a VirtualService route to a subset that is not defined in a DestinationRule — Istio will reject the configuration
• Never omit outlierDetection from production DestinationRules — services without circuit breaking will cascade failures
• Never set retries.attempts higher than 3 — excessive retries amplify load spikes during incidents
• Never use retries.retryOn: 5xx as the only retry condition — 503 from circuit breakers should not trigger more retries
• Never use mode: DISABLE on PeerAuthentication in production — it allows unencrypted communication between services
• Never apply Istio CRDs to the kube-system or istio-system namespaces without explicit justification

Output Template

When implementing an Istio service mesh configuration, produce the following:

1. DestinationRule YAML — Complete DestinationRule with subset definitions, connection pool limits, outlier detection, and TLS mode.
2. VirtualService YAML — Complete VirtualService with route rules, weight splits, header matches, retries, and fault injection as required.
3. Security Policies — PeerAuthentication (mTLS mode) and AuthorizationPolicy (access control) if mesh security is needed.
4. Traffic Matrix — A table mapping each route rule to its subset, weight, and matching criteria for operational reference.

Related Skills

Live References

Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content.

• Istio VirtualService Documentation — VirtualService routing rules, weights, retries, and fault injection
• Istio DestinationRule Documentation — Subset definitions, connection pools, outlier detection, and TLS
• Istio Traffic Management — Overview of traffic management concepts and the service mesh data plane
• Istio PeerAuthentication — mTLS enforcement modes (PERMISSIVE, STRICT, DISABLE)
• Istio AuthorizationPolicy — Fine-grained access control within the mesh
• Istio ServiceEntry — Configuring outbound traffic to external services
• Istio Sidecar Injection — How sidecar injection works and how to control it
• Istio Outlier Detection — Circuit breaker configuration and ejection behavior