How do I install the Fluentd skill?

Run npx skillmds@latest add paulpas/fluentd in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Fluentd skill do?

"Fluentd unified logging layer for collecting, transforming, and routing" log data in cloud-native environments It is listed under DevOps & Infra on SkillMD.

Is Fluentd safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Fluentd?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Fluentd free to use?

Yes. Installing skills from SkillMD is free. This skill is licensed under MIT.

Who published Fluentd?

paulpas (@paulpas) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Fluentd by paulpas — Agent Skill

Name: Fluentd
Author: paulpas

related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-gcp-autoscaling

Tutorial

This tutorial will guide you through installing, configuring, and using Fluentd for centralized logging collection.

Prerequisites

Before beginning, ensure you have:

A running Kubernetes cluster (minikube, kind, EKS, GKE, AKS)
kubectl configured to access your cluster
Basic understanding of logging concepts and Kubernetes architecture
A logging destination (Elasticsearch, Fluentd, S3, etc.)

Verify your setup:

# Check cluster connectivity
kubectl cluster-info

# Verify kubectl configuration
kubectl get nodes

# Check cluster version
kubectl version --client --short

1. Installation

Fluentd can be installed in various ways depending on your environment and requirements.

Method 1: Using Helm (Recommended for Kubernetes)

# Add the Fluentd Helm repository
helm repo add fluent-stable https://fluent.github.io/helm-charts
helm repo add fluentd https://fluent.github.io/helm-charts

# Update repositories
helm repo update

# Install Fluentd with default configuration
helm install fluentd fluent-stable/fluentd \
  --namespace logging \
  --create-namespace \
  --wait

# Install with custom values
helm install fluentd fluent-stable/fluentd \
  --namespace logging \
  --create-namespace \
  --set fluentd.conf.type="s3" \
  --set output.host="elasticsearch.logging.svc.cluster.local"

# Verify installation
helm list -n logging
kubectl -n logging get pods

Method 2: Using kubectl apply (Direct Manifests)

# Download the Fluentd manifest
curl -O https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-kubernetes-1.16.yaml

# Or use the simplified version
kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch.yaml

# Verify deployment
kubectl get daemonset fluentd -n kube-system
kubectl get pods -n kube-system -l app=fluentd

Method 3: Installing Fluentd on Bare Metal

# Install Fluentd using RubyGems
gem install fluentd --no-document

# Or use td-agent (Fluentd's distribution)
# For Ubuntu/Debian
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-focal-td-agent4.sh | sh

# For CentOS/RHEL
curl -fsSL https://toolbelt.treasuredata.com/sh/install-redhat-centos-td-agent4.sh | sh

# Verify installation
td-agent --version
fluentd --version

Method 4: Using Docker

# Pull the Fluentd Docker image
docker pull fluent/fluentd:v1.16

# Run Fluentd with custom configuration
docker run -d \
  -p 24224:24224 \
  -v /path/to/fluent.conf:/fluentd/etc/fluent.conf \
  -v /var/log:/fluentd/log \
  fluent/fluentd:v1.16

# Verify container is running
docker ps | grep fluentd

2. Basic Configuration

Fluentd Configuration File Structure

# fluent.conf - Basic configuration
<system>
  log_level info
  suppress_config_dump true
</system>

# Source: Kubernetes container logs
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  <parse>
    @type json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

# Filter: Enrich logs with Kubernetes metadata
<filter kubernetes.**>
  @type kubernetes_metadata
  kubernetes_url https://kubernetes.default.svc
  verify_ssl false
</filter>

# Match: Send logs to Elasticsearch
<match **>
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
  logstash_format true
  logstash_prefix kubernetes
  flatten_hashes true
  <buffer>
    @type file
    path /var/log/fluentd/buffer
    flush_interval 5s
    flush_thread_count 2
    retry_type exponential_backoff
  </buffer>
</match>

Environment Variables for Configuration

# Set environment variables for runtime configuration
export FLUENTD_CONF=fluent.conf
export FLUENTD_ARGS=--no-config-autoreload

# Kubernetes environment variables (for kubernetes_metadata filter)
export KUBERNETES_SERVICE_HOST=kubernetes.default.svc
export KUBERNETES_SERVICE_PORT=443

ConfigMap for Kubernetes

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <system>
      log_level info
    </system>
    
    <source>
      @type forward
      port 24224
      bind 0.0.0.0
    </source>
    
    <match **>
      @type file
      path /var/log/fluentd/output
      append true
      <format>
        @type json
      </format>
      <buffer>
        @type file
        path /var/log/fluentd/buffer
        flush_interval 1s
      </buffer>
    </match>
  
  parsers.conf: |
    <parse>
      @type json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </parse>

related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-gcp-autoscaling

3. Usage Examples

Viewing Fluentd Logs

# View Fluentd logs in Kubernetes
kubectl -n logging logs -f deployment/fluentd

# Follow Fluentd logs with timestamp
kubectl -n logging logs -f deployment/fluentd --timestamps

# View logs from a specific pod
kubectl -n logging logs fluentd-xyz123 -f

# Check Fluentd pod status
kubectl -n logging get pods -l app=fluentd

Testing Log Collection

# Create a test pod to generate logs
kubectl run test-logger --image=busybox --restart=Never -- tail -f /dev/null

# In another terminal, send a test log
kubectl exec test-logger -- sh -c 'echo '{"level":"info","message":"Test log message"}' | fluent-cat test.tag'

# Verify the log was collected
kubectl -n logging logs -l app=fluentd | grep "Test log"

Loading Configuration from ConfigMap

# Create ConfigMap from fluent.conf
kubectl create configmap fluentd-config \
  --from-file=fluent.conf \
  --namespace=logging

# Or update existing ConfigMap
kubectl create configmap fluentd-config \
  --from-file=fluent.conf \
  --namespace=logging \
  --dry-run=client \
  -o yaml | kubectl apply -f -

# Verify ConfigMap
kubectl -n logging get configmap fluentd-config -o yaml

Setting up Multiple Outputs

# Multiple output configuration
<match **>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch.logging.svc.cluster.local
    port 9200
    logstash_format true
  </store>
  <store>
    @type s3
    s3_bucket my-logs-bucket
    s3_region us-east-1
    path logs/%Y/%m/%d/
    <format>
      @type json
    </format>
    <buffer>
      @type file
      path /var/log/fluentd/s3
    </buffer>
  </store>
  <store>
    @type stdout
  </store>
</match>

4. Common Operations

Monitoring Fluentd

# Check Fluentd metrics (if Prometheus plugin is enabled)
kubectl -n logging get pods -l app=fluentd -o wide

# View Fluentd pod resources
kubectl -n logging describe pod -l app=fluentd

# Check Fluentd configuration reload status
kubectl -n logging logs deployment/fluentd | grep "config file"

# View Fluentd buffer status
kubectl -n logging exec -it deployment/fluentd -- ls -la /var/log/fluentd/buffer

Restarting Fluentd DaemonSet

# Rolling restart of Fluentd pods
kubectl -n logging delete pods -l app=fluentd

# Wait for pods to restart
kubectl -n logging rollout status daemonset/fluentd

# Verify all pods are running
kubectl -n logging get pods -l app=fluentd

Updating Configuration

# Update ConfigMap
kubectl create configmap fluentd-config \
  --from-file=fluent.conf \
  --namespace=logging \
  --dry-run=client \
  -o yaml | kubectl apply -f -

# Delete pods to trigger reload
kubectl -n logging delete pods -l app=fluentd

Scaling Fluentd

# Fluentd typically runs as a DaemonSet (one per node)
# To scale, update the DaemonSet
kubectl -n logging patch daemonset fluentd -p '{"spec":{"template":{"spec":{"containers":[{"name":"fluentd","resources":{"limits":{"cpu":"500m","memory":"512Mi"}}}]}}}}'

# Or edit the daemonset
kubectl -n logging edit daemonset fluentd

Backup and Restore

# Backup Fluentd configuration
kubectl -n logging get configmap fluentd-config -o yaml > fluentd-config-backup.yaml

# Export buffer state (if needed)
kubectl -n logging exec deployment/fluentd -- tar czf /tmp/buffer-backup.tar.gz /var/log/fluentd/buffer

# Restore configuration
kubectl -n logging apply -f fluentd-config-backup.yaml

5. Best Practices

Configuration Best Practices

Use Separate Sources: Define separate <source> blocks for different log types
Filter Early: Apply filters as close to the source as possible
Buffer Properly: Configure buffer to prevent data loss during outages
Use Tags: Use meaningful tags for route matching
Separate Concerns: Keep configuration modular with @include directives

Performance Optimization

Optimize Buffer: Adjust buffer settings based on log volume

<buffer>
  @type file
  flush_interval 5s
  flush_thread_count 2
  chunk_limit_size 8MB
  queued_chunks_limit_size 32
</buffer>

Use Multiple Threads: Configure flush_thread_count for parallel processing
Reduce Parsing: Use forward input to receive pre-parsed logs when possible
Compress Buffers: Enable compression for large buffers

Security Best Practices

RBAC: Configure proper RBAC for Fluentd service account
Network Policies: Restrict Fluentd network access to required destinations
Secrets Management: Use Kubernetes Secrets for sensitive configuration
Log Redaction: Implement filters to redact sensitive data
TLS Encryption: Use TLS for log transmission to external systems

Monitoring and Alerting

Prometheus Metrics: Enable Fluentd's Prometheus plugin

<source>
  @type prometheus
  bind 0.0.0.0
  port 24231
</source>

Health Checks: Configure readiness/liveness probes
Alert on Backlog: Monitor buffer size and alert on growth
Log Volume Monitoring: Track log volume trends

Kubernetes-Specific Tips

Use DaemonSet: Deploy Fluentd as a DaemonSet for node-level log collection
Namespace Isolation: Use separate Fluentd instances per namespace for isolation
Label Selectors: Use label selectors to filter logs by application
Resource Limits: Set appropriate CPU/memory limits based on cluster size

Examples

Basic Configuration

# Basic configuration example
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <system>
      log_level info
    </system>
    
    <source>
      @type forward
      port 24224
      bind 0.0.0.0
    </source>
    
    <match **>
      @type file
      path /var/log/fluentd/output
      append true
      <format>
        @type json
      </format>
      <buffer>
        @type file
        path /var/log/fluentd/buffer
        flush_interval 1s
      </buffer>
    </match>

Kubernetes Deployment

# Fluentd DaemonSet for Kubernetes
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.16
        env:
        - name: FLUENT_UID
          value: "0"
        resources:
          limits:
            memory: "512Mi"
            cpu: "500m"
        volumeMounts:
        - name: config-volume
          mountPath: /fluentd/etc
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: config-volume
        configMap:
          name: fluentd-config
      - name: varlog
        hostPath:
          path: /var/log

Kubernetes Service

# Kubernetes service for Fluentd
apiVersion: v1
kind: Service
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    app: fluentd
  ports:
  - protocol: TCP
    port: 24224
    targetPort: 24224
    name: fluentd
  - protocol: TCP
    port: 24231
    targetPort: 24231
    name: prometheus
  type: ClusterIP

Kubernetes Container Log Collection

# Collect logs from Kubernetes containers
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  <parse>
    @type json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

# Enrich with Kubernetes metadata
<filter kubernetes.**>
  @type kubernetes_metadata
  kubernetes_url https://kubernetes.default.svc
  verify_ssl false
</filter>

# Send to Elasticsearch
<match kubernetes.**>
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
  logstash_format true
  logstash_prefix kubernetes
  <buffer>
    @type file
    path /var/log/fluentd/buffer
    flush_interval 5s
  </buffer>
</match>

Multi-Output Configuration

# Copy logs to multiple destinations
<match **>
  @type copy
  <store>
    @type elasticsearch
    host es1.example.com
    port 9200
  </store>
  <store>
    @type elasticsearch
    host es2.example.com
    port 9200
  </store>
  <store>
    @type s3
    s3_bucket my-logs
    s3_region us-east-1
  </store>
</match>

When to Use

Use this skill when:

Integrating a CNCF project into Kubernetes infrastructure — You need to configure, deploy, or troubleshoot a cloud-native tool within a cluster
Designing cloud-native architecture — You are selecting and integrating CNCF tools to solve specific infrastructure challenges
Resolving operational issues — A CNCF component is misbehaving, underperforming, or needs configuration changes

Core Workflow

Assess Requirements — Understand the use case, scale, integration needs, and existing infrastructure. Checkpoint: Document requirements, constraints, and success criteria.
Design Architecture — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. Checkpoint: Verify the architecture addresses all requirements and follows CNCF conventions.
Implement & Configure — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. Checkpoint: Validate all YAML against schema and test in a staging environment.
Deploy & Monitor — Apply manifests to the cluster, verify component health, and confirm observability is working. Checkpoint: Confirm all pods/services are running, probes passing, and metrics/alerts configured.

Fluentd

File contents