401 Kubelet Logs 082b8dd5

AKS Kubelet Logs Guide

Overview

Kubelet logs provide critical information about node-level operations, pod lifecycle management, and container runtime interactions in AKS clusters.

Access Methods

Method A: kubectl Raw API (Quick - No SSH)

# Set node name
export NODE_NAME="aks-agentpool-xxxxxxx-0"

# Get kubelet logs via raw API
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep kubelet

Advantages:

• Quickest method
• No SSH access required
• Useful for rapid diagnostics

Method B: SSH Connection (Detailed Access)

# 1. Establish SSH connection to node
# 2. Enter host environment
chroot /host

# 3. View kubelet logs
journalctl -u kubelet -o cat

Method C: Container Insights

Azure Monitor Container Insights provides:

• Syslog collection for kubelet logs
• Integration with Log Analytics
• Centralized log aggregation

Log Locations

Linux Nodes

Windows Nodes

Log Format

I0508 12:26:17.905042    8672 kubelet_node_status.go:497] Using Node Hostname from cloudprovider: "aks-agentpool-xxxxxxx-0"
I0508 12:26:28.920125    8672 server.go:796] GET /stats/summary: (10.370874ms) 200 [[Ruby] 10.244.0.x:52492]

Format Components

CLI Commands

Prerequisites

export RESOURCE_GROUP_NAME="<ResourceGroupName>"
export AKS_CLUSTER_NAME="<AKSClusterName>"
az aks get-credentials --resource-group $RESOURCE_GROUP_NAME --name $AKS_CLUSTER_NAME

Retrieve Kubelet Logs

# Via kubectl raw API
export NODE_NAME="aks-agentpool-xxxxxxx-0"
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep kubelet

# Via az aks command invoke
az aks command invoke -g $RESOURCE_GROUP_NAME -n $AKS_CLUSTER_NAME \
  --command "chroot /host && journalctl -u kubelet -o cat"

Filter Logs

# Last 100 lines
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep kubelet | tail -100

# Errors only
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep -E "^E.*kubelet"

# Warnings and errors
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep -E "^[EW].*kubelet"

# Time range (via journalctl on node)
journalctl -u kubelet --since "2024-01-01 00:00:00" --until "2024-01-01 12:00:00"

Common Issues in Kubelet Logs

Node Status Issues

Connectivity Issues

Resource Issues

Pod Lifecycle Issues

Troubleshooting Workflow

Step 1: Identify Issue Category

• Cluster creation failures
• Upgrade/scaling operations
• Node status problems
• Connectivity issues
• Performance degradation

Step 2: Collect Logs

# Quick collection
kubectl get --raw "/api/v1/nodes/$NODE_NAME/proxy/logs/messages" | grep kubelet > kubelet-logs.txt

# Detailed collection (SSH)
journalctl -u kubelet -o cat > kubelet-detailed.txt

Step 3: Analyze Patterns

# Search for errors
grep -E "^E" kubelet-logs.txt

# Search for specific issues
grep -i "certificate" kubelet-logs.txt
grep -i "connection refused" kubelet-logs.txt
grep -i "timeout" kubelet-logs.txt
grep -i "evicted" kubelet-logs.txt

Step 4: Correlate with Events

# Get node events
kubectl get events --field-selector involvedObject.name=$NODE_NAME

# Describe node
kubectl describe node $NODE_NAME

Key Diagnostic Patterns

Error Patterns to Search

Log Level Reference

Additional Tools

AKS Diagnose and Solve Problems

• Automated self-diagnostic in Azure Portal
• No additional configuration needed
• Covers connectivity, best practices, cluster health

AKS Periscope

• Detailed log collection tool
• Comprehensive node diagnostics
• Network connectivity checks

Data Collection Methods

• Real-time system insights capture
• TCP dump from nodes
• TCP packet capture from pods
• Container dumps (Windows)

Access Methods Comparison

Best Practices

1. Start with kubectl raw API for quick diagnostics
2. Use SSH for deep investigation when raw API is insufficient
3. Enable Container Insights for continuous log collection
4. Set up log-based alerts for critical patterns
5. Correlate with Kubernetes events for context
6. Document common patterns for your workloads