Fluent Bit Knowledge Patch

Fluent Bit Knowledge Patch

Use this skill when configuring, upgrading, operating, or extending Fluent Bit and the work depends on recent pipeline, plugin, security, storage, or observability behavior.

How to use this skill

1. Inspect the deployed Fluent Bit version and its YAML or classic-mode configuration before applying guidance.
2. Start with the compatibility traps below when upgrading an existing fleet.
3. Open the reference matching the affected pipeline stage; reference pages contain the full option names, limits, defaults, and operational effects.
4. Apply version-attributed behavior only where the deployed release supports it, including patch-level defaults and fixes.
5. Test producers, receivers, storage recovery, authentication failures, and monitoring queries under load before rolling out a change.

Reference index

Breaking changes and compatibility traps

Prefer canonical shared HTTP listener settings

HTTP-family inputs share canonical http_server.* listener keys for HTTP/2, chunk and maximum buffer sizes, connections, workers, and ingress queue event and byte limits. Use those names in new configuration.

The older http2, buffer_chunk_size, and buffer_max_size spellings remain compatibility aliases, but canonical names make shared listener behavior explicit across HTTP, Splunk, Elasticsearch, OpenTelemetry, and Prometheus Remote Write inputs.

Disable Forward metadata for Fluentd receivers

Starting in v5.0.4, the Forward output defaults retain_metadata_in_forward_mode to true. That sends extended MessagePack metadata which Fluentd receivers reject. Disable it on a Fluentd-bound output:

pipeline:
  outputs:
    - name: forward
      match: "*"
      host: fluentd-host
      port: 24224
      retain_metadata_in_forward_mode: false

Treat hot-reload counts as a counter

fluentbit_hot_reloaded_times is a counter rather than a gauge. Rewrite Prometheus dashboards and alerts to use rate() or increase(); delta() and other gauge-oriented calculations no longer describe the metric correctly.

Recheck JSON Unicode escaping

escape_unicode is honored by every JSON-outputting plugin. An upgrade can therefore change serialized output for plugins that previously ignored it. Compare payload bytes and downstream parsing wherever escaping is significant.

Reject malformed OpenTelemetry identifiers

OpenTelemetry log and trace IDs receive strict decoding. Fix producers that emit malformed identifiers rather than relying on permissive ingestion.

Plan for emitter backpressure

With filesystem storage, emitter-backed filters automatically enable storage.pause_on_chunks_overlimit when the setting is absent and pause at storage.max_chunks_up. Pending emitter bytes also count toward mem_buf_limit, paused records are retained, and Rewrite Tag keeps the original record if retagging cannot enqueue it.

Set storage.pause_on_chunks_overlimit off on the relevant input only when the old over-limit accumulation behavior is an intentional requirement. Monitor memory, filesystem chunks, and retained originals during migration.

Validate Secure Forward peers

Secure Forward now validates inbound PING messages and outbound PONG server digests. An output configured with a username or password but no shared key is rejected. Custom peers must also accept base64-encoded 128-bit chunk acknowledgment identifiers.

High-value routing and processing

Route directly from an input

Direct routes can send input events to outputs without the traditional routing path. Routes may use conditions and context variables, match output labels or plugin names from chunk metadata, survive restored chunks and reloads, and are expressible in YAML. Define each route under the exact input section that owns it; this prevents multiple instances of one input plugin from cross-routing.

Use the routing metrics for matched, unmatched, and dropped event counts and for route performance. See Routing, processing, and storage for the storage and input-scoping details.

Make processor decisions explicit

Log processors accept conditions built from and, or, in, gt, lt, and regex operators. Trace sampling supports probabilistic head sampling and conditional tail sampling based on status, latency, typed attributes, span count, or trace state. During a tail-sampling migration, use legacy_reconcile to compare the optimized reconciler with earlier behavior.

Metrics pipelines can convert cumulative monotonic samples to delta temporality, and can run topology-based analysis. Keep temporality conversion close to the backend contract so cumulative and delta series are not mixed.

Preserve and inspect invalid chunks

The dead-letter queue can quarantine invalid filesystem-backed chunks with safe deletion checks. storage_backlog preserves that quarantine across restarts. Exercise a restart in recovery tests so corrupt input is neither silently lost nor repeatedly delivered.

High-value ingestion and security

Secure HTTP-family listeners

TLS-capable inputs can require sender certificates with tls.verify_client_cert on alongside tls.crt_file and tls.key_file. HTTP and OpenTelemetry inputs can validate bearer tokens with oauth2.* settings. The HTTP output can obtain client-credentials tokens using basic, post-body, or private-key JWT client authentication.

Choose mutual TLS, OAuth 2.0, or both from the producer trust model. Keep TLS protocol floors, cipher selection, issuer, audience, client allowlists, and JWKS refresh behavior explicit; see Security and configuration.

Ingest the agent's own logs

The fluentbit_logs input turns internal logs into structured pipeline records. The Kubernetes filter can enrich those records from local metadata, with kube_namespace_file available to override the namespace file path. Add loop prevention when forwarding agent diagnostics through the same pipeline.

Distinguish the two health endpoints

The built-in server exposes JSON health at /api/v2/health, returning 200 when healthy and 500 on failure. Separately, an HTTP input can expose GET /health as an ingestion readiness probe. Point each load balancer or orchestrator probe at the endpoint whose scope it actually needs.

Account for ingestion response semantics

HTTP, Splunk, and Elasticsearch inputs preserve busy, ingestion-error, and payload status behavior more consistently under worker load. Splunk HEC also returns compatible 401 and 403 responses for missing, malformed, or invalid authentication. Configure senders to retry only retryable responses.

High-value outputs and observability

Select compression per destination

OpenTelemetry output supports zstd; HTTP supports zstd and snappy. Kinesis Data Streams supports gzip, zstd, and snappy; Firehose and S3 support snappy; Azure Blob supports zstd transfer compression. S3 Parquet output uses a pure-C Apache Arrow backend and can use zstd.

Benchmark CPU, network, and receiver compatibility rather than assuming one codec is optimal for every destination.

Revalidate operational queries

Output monitoring includes chunk-latency and downstream-backpressure timing. S3 accounting more accurately represents logical records, while grouped logs, retries, routed traffic, worker ownership, and ingress flow have clearer signals. Recheck SLO, chargeback, and delivery-validation queries after an upgrade.

Treat missing CloudWatch streams as terminal for the chunk

CloudWatch Logs output treats a missing stream as unrecoverable for the current chunk and evicts stale stream state. Alerting must account for the chunk not being retried through that stale stream.

Upgrade checklist

• Inventory HTTP-family listener aliases and adopt canonical http_server.* keys.
• Test OpenTelemetry IDs, null values, JSON metrics and traces, compressed gRPC, and maximum message size.
• Compare JSON Unicode escaping and Forward payloads byte-for-byte.
• Exercise direct-route matches, reloads, restored chunks, and duplicate input plugin instances.
• Fill storage limits, pause emitters, restart with dead-letter data, and check Rewrite Tag originals.
• Validate TLS versions, ciphers, client certificates, OAuth failures, and Secure Forward handshakes against real peers.
• Verify HTTP-family status codes and both health-endpoint scopes under load.
• Update reload, latency, backpressure, skipped-line, and accounting queries.
• Test cloud codecs, S3 Parquet, and missing-stream failure paths with their actual receivers.