agentskillexchange Redpanda Connect Declarative Stream Processor
Redpanda Connect (formerly Benthos) is a high-performance stream processor that connects data sources and sinks through declarative YAML pipelines. It supports hundreds of connectors and a built-in mapping language called Bloblang for data transformation.
Installation
Use the upstream install or setup path that matches your environment:
- brew install redpanda-data/tap/redpanda
- docker pull docker.redpanda.com/redpandadata/connect
- docker run --rm -v /path/to/your/config.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run
- docker run --rm -p 4195:4195 docker.redpanda.com/redpandadata/connect run \
Requirements and caveats from upstream:
- It comes with a [powerful mapping language][bloblang-about], is easy to deploy and monitor, and ready to drop into your pipeline either as a static binary or docker image, making it cloud native as heck.
- Or pull the docker image:
- Or, with docker:
Basic usage or getting-started notes:
shell
curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
unzip rpk-linux-amd64.zip -d ~/.local/bin/
Extracted from upstream docs: https://raw.githubusercontent.com/redpanda-data/connect/HEAD/README.md