Vector

Working With Vector

Vector is a high-performance observability data pipeline (written in Rust) that collects, transforms, and routes logs, metrics, and traces through a directed acyclic graph (DAG) of sources, transforms, and sinks.

Pipeline Anatomy

Every Vector config defines a DAG of three component types wired together via inputs:

sources --> transforms --> sinks

• Sources ingest data (file, kafka, http_server, vector, kubernetes_logs, ...)
• Transforms process data (remap/VRL, filter, route, sample, dedupe, aggregate, log_to_metric, ...)
• Sinks deliver data (elasticsearch, aws_s3, kafka, loki, datadog_logs, prometheus_exporter, vector, ...)

Components connect through the inputs field forming the DAG. Fan-in and fan-out are both supported. The route transform produces named outputs referenced as component_id.output_name.

Workflow

Always validate and test before deploying:

vector validate config.toml       # check config correctness
vector test config.toml           # run unit tests
vector --config config.toml       # start pipeline

Use vector vrl to prototype transforms interactively.

Configuration

Vector supports TOML, YAML, and JSON. Prefer TOML or YAML -- JSON is verbose and does not support comments, making configs harder to maintain.

Minimal example (TOML):

data_dir = "/var/lib/vector"

[sources.app_logs]
type = "file"
include = ["/var/log/app/*.log"]

[transforms.parse]
type = "remap"
inputs = ["app_logs"]
source = '''
  . = parse_json!(.message)
  del(.password)
'''

[sinks.out]
type = "elasticsearch"
inputs = ["parse"]
endpoints = ["https://es:9200"]
bulk.index = "app-%Y.%m.%d"

Key global options:

• data_dir -- persistent state directory (checkpoints, disk buffers). Must persist across restarts.
• [api] -- enable the management API (/health, GraphQL, vector top/vector tap)
• [log_schema] -- override default field names (host, message, timestamp)
• Environment variable interpolation: ${VAR}, ${VAR:-default}, ${VAR:?error}

Multi-file configs: vector --config-dir /etc/vector/ or vector -c sources.toml -c transforms.toml -c sinks.toml. Wildcard inputs are supported: inputs = ["parse_*"].

VRL (Vector Remap Language)

VRL is the primary transform language. It compiles at config load time (no runtime interpretation), is safe (no loops, no I/O, guaranteed termination), and requires explicit error handling.

Core syntax

# Access/set fields
.new_field = "value"
.nested.field = 42
del(.unwanted)

# Variables
my_var = parse_json!(.message)

# Error handling -- fallible functions require one of:
result = parse_json!(.msg)           # ! = abort on error (event goes to .dropped)
result, err = parse_json(.msg)       # explicit error capture
result = parse_json(.msg) ?? {}      # coalesce with default

# Conditionals
if .status >= 500 { .severity = "error" } else { .severity = "info" }

# Null coalescing
.host = .hostname ?? .host ?? "unknown"

# Abort (drop event when drop_on_abort = true)
if .level == "debug" { abort }

# Iteration -- VRL has no general loops but provides for_each for objects and arrays
for_each(object!(.tags)) -> |key, value| {
  # process each key-value pair
}
for_each(array!(.items)) -> |index, value| {
  # process each element
}

Key functions

Standard string, type-check, and encoding functions work as expected (downcase, contains, to_int!, is_string, encode_json, etc.). These are Vector-specific or commonly needed:

Prototype complex parse_regex! and parse_grok! patterns interactively with vector vrl before embedding in config.

Emitting multiple events (fan-out)

Setting . to an array in a remap transform causes Vector to emit one event per element. This is the idiomatic way to "explode" a single log into multiple events without Lua:

payload = object!(.payload)
events = []
for_each(payload) -> |key, value| {
  event = {"name": key, "value": value, "host": .host}
  events = push(events, event)
}
. = events

drop_on_abort vs reroute_dropped

Both options control what happens to events that hit abort in a remap transform, but they are mutually exclusive:

• drop_on_abort = true (the default) -- aborted events are silently discarded. Use this when you simply want to filter out events and do not need to inspect failures.
• reroute_dropped = true -- aborted events are sent to a .dropped output (e.g., transform_id.dropped) instead of being discarded. Use this when you need a dead-letter queue or want visibility into what was dropped. Setting reroute_dropped = true implicitly sets drop_on_abort = true.

If neither is set, drop_on_abort defaults to true and aborted events vanish. To capture dropped events, explicitly set reroute_dropped = true and wire a sink to the .dropped output.

VRL constraints (by design)

• No general-purpose loops (single-pass transforms only)
• No I/O (no file/network access)
• No custom functions
• No recursion
• No cross-event state
• Metric events are restrictive -- only tags and certain fields can be modified

Timezone handling

Logs often lack timezone info in the timestamp string. Since v0.44, parse_timestamp accepts an IANA timezone as a third argument:

# Explicit timezone on the function call
.timestamp = parse_timestamp!(.raw_ts, format: "%Y-%m-%d %H:%M:%S%.f", timezone: "America/Sao_Paulo")

Alternatively, set timezone on the remap transform to apply it to all VRL timestamp functions in that block:

[transforms.parse]
type = "remap"
inputs = ["source"]
timezone = "America/Sao_Paulo"
source = '''
  # parse_timestamp will use the transform-level timezone for naive timestamps
  .timestamp = parse_timestamp!(.raw_ts, format: "%Y-%m-%d %H:%M:%S%.f")
'''

The global config also supports timezone as a default for all transforms. Resolution order: timezone in the string itself > function argument > transform-level > global > UTC.

Do not hardcode UTC offsets (e.g. appending " -0300") -- this breaks across DST transitions and is fragile if timezone rules change.

Idiomatic VRL patterns

Parse-and-restructure:

. = parse_json!(.message)
.timestamp = parse_timestamp!(.timestamp, format: "%Y-%m-%dT%H:%M:%S%.fZ")
.severity = if .status >= 500 { "error" } else if .status >= 400 { "warn" } else { "info" }
del(.raw_body)

Redact sensitive data:

.message = replace(.message, r'[\w.+-]+@[\w-]+\.[\w.]+', "[REDACTED]")
del(.password)
del(.api_key)

Dead-letter routing (use reroute_dropped = true on the remap transform):

parsed, err = parse_json(.message)
if err != null { abort }  # sends to <transform>.dropped output
. = parsed

Unit Testing

Test transforms inline in config files. Run with vector test <config>.

[[tests]]
name = "parse_app_logs"

[[tests.inputs]]
insert_at = "parse"
type = "log"
[tests.inputs.log_fields]
message = '{"user":"alice","action":"login","ts":"2024-01-15T10:00:00Z"}'

[[tests.outputs]]
extract_from = "parse"
[[tests.outputs.conditions]]
type = "vrl"
source = '''
  assert_eq!(.user, "alice")
  assert_eq!(.action, "login")
  assert!(is_timestamp(.timestamp))
  assert!(!exists(.password))
'''

Use no_outputs_from to verify a filter drops events:

[[tests]]
name = "filter_drops_debug"
no_outputs_from = ["debug_filter"]

[[tests.inputs]]
insert_at = "debug_filter"
type = "log"
[tests.inputs.log_fields]
level = "debug"

Test chains by setting insert_at to the first transform and extract_from to the last.

Pipeline Design Patterns

See references/patterns.md for detailed pipeline patterns:

• Agent + Aggregator topology
• Routing and fan-out
• Dead-letter queues
• Sampling and cost reduction
• Metrics derivation from logs
• Multi-tenant routing
• Enrichment tables

Production Deployment

See references/production.md for:

• Deployment topologies (distributed, centralized, stream-based)
• Sizing and capacity planning
• Buffering strategies (memory vs disk vs overflow)
• Hardening (TLS, secrets, non-root, disk encryption)
• High availability patterns
• Monitoring Vector itself (internal_metrics, internal_logs)
• Troubleshooting workflow

Common Mistakes

Avoid these frequent pitfalls (see references/production.md for detailed production guidance):

• Generic component IDs (transform_1, sink_2) -- use descriptive names like parse_nginx_logs, route_by_severity
• Hardcoded secrets -- always use ${ENV_VAR} interpolation
• Deploying without running vector validate and vector test
• Using Lua when VRL can handle the use case. In particular, VRL fan-out (. = array) replaces the most common Lua use case (emitting multiple events). The Lua event API is also version-dependent -- Event.new() does not exist; creating new events requires constructing {log = {field = value}} tables, and deep-copying event objects is fragile due to metatables.
• Blindly using ! on every fallible function instead of handling errors explicitly
• Placing filters late in the pipeline after expensive transforms -- wastes CPU on events that will be discarded
• Using memory buffers for data you cannot afford to lose
• Ephemeral data_dir (loses checkpoints and disk buffers on restart) -- causes reprocessing of already-seen data and loss of disk buffer contents
• Running without self-monitoring (internal_metrics source) -- no visibility into pipeline health, cannot detect data loss