Injector Dev

injector-dev

injector-dev is a CLI that turns the manual loop of build image → push → manage a cluster → deploy workloads by hand into a single declarative command. You describe a scenario in YAML — which Agent components to deploy, how to configure them, and what test workloads to run — and injector-dev apply brings the whole environment up on a local Kubernetes cluster.

Scenarios are reproducible and shareable: tear an environment down and recreate it identically at any time.

When to use this skill

• Iterating on local datadog-agent, datadog-operator, or Helm chart changes and needing them running on a real cluster.
• Reproducing an APM auto-instrumentation / injection bug locally.
• Writing, editing, or debugging a scenario.yaml.
• Pinning a test environment against a specific released version, a CI pipeline artifact, or the agent main branch.

Prerequisites

• A local Kubernetes platform. Supported drivers: kind (recommended), colima, minikube, nvkind, and none (use an existing cluster/context). kind is fast, reliable across machines, and the easiest to reset. There is also a workspace driver that runs the cluster on a remote Datadog Workspace VM — see Remote clusters.
• A Docker runtime installed and running.
• helm and kubectl on your PATH. kind must be installed to use the recommended kind platform.
• Datadog API + App keys (a real API key is needed for the Agent to report). Set them however you prefer:

  export DD_API_KEY="<your-api-key>"
  export DD_APP_KEY="<your-app-key>"
  

If you already have colima and/or minikube running, shut them down before using injector-dev to avoid conflicts.

Installation

git clone https://github.com/DataDog/injector-dev
cd injector-dev
make install   # builds and installs the binary to /usr/local/bin (uses sudo)

Configuration (~/.injector-dev/config.yaml)

A practical starting config:

---
platform: kind              # kind (recommended) | colima | minikube | nvkind | workspace | none
builder:
  code_root: "/Users/<USER>/dd"   # parent dir containing datadog-agent/, auto_inject/, datadog-operator/, ...
  dev_container:
    name: "injector-dev-builder"
    enabled: true
    persist: true           # keep the build container alive between builds → much faster rebuilds
installer:
  repo_root: "/Users/<USER>/dd/injector-dev"
  # api_key / app_key are optional here; DD_API_KEY / DD_APP_KEY env vars are the fallback.

Key points:

• builder.code_root — parent directory holding your component repos. The tool derives each repo path as code_root/<repo-name> (e.g. code_root/datadog-agent), so all repos must be siblings under this dir.
• builder.dev_container.persist: true — leaves the build container running between applies. First build takes several minutes (installing deps); later builds drop to ~30 seconds.
• installer.repo_root — path to your local injector-dev checkout.
• Every config key can be overridden by env var: prefix with INJECTOR_DEV_ and replace dots with underscores, e.g. INJECTOR_DEV_INSTALLER_API_KEY, INJECTOR_DEV_BUILDER_CODE_ROOT.
• API/App key resolution order: config.yaml (installer.api_key/app_key) → then DD_API_KEY / DD_APP_KEY env vars.

Profiles (multiple configs)

To manage several environments (staging, sandbox, org2, …), drop additional config files at ~/.injector-dev/<profile>.yaml and select one per command:

injector-dev apply -f scenario.yaml --profile sandbox

Omitting --profile uses config.yaml.

Remote clusters: the workspace platform

Instead of a local cluster, injector-dev can run the kind cluster on a remote Datadog Workspace VM. Image builds still happen locally — only the cluster and workloads run on the workspace. Useful when your laptop is resource-constrained or you want a beefier, disposable environment.

How it works: docker build runs locally; the image is streamed to the workspace (docker save | ssh … kind load); kind is installed on the workspace automatically on first use (workspaces ship with docker but not kind); and the remote cluster's API server is exposed to your machine through a persistent SSH tunnel, with an injector-dev-ws-<name> context merged into ~/.kube/config and made current — so local kubectl/helm/k9s work against it transparently.

Prerequisites

• A workspace reachable over SSH as ssh workspace-<name> (be connected to Appgate). Create one with:

  workspaces create <name> --repo dd/datadog-agent
  

  injector-dev does not create the workspace; if it's missing it fails fast and prints this command.
• Local docker for building images, as usual. The workspace only needs docker (kind is installed for you).

Selecting the workspace

The workspace name comes from the --workspace flag or a scenario's platform.workspace (flag wins). It is intentionally not read from the global config.yaml.

In a scenario (picked up by apply):

platform:
  type: workspace
  workspace: firstname-lastname   # SSH host = workspace-firstname-lastname
  name: my-cluster                # optional kind cluster name on the VM (default: "kind")
  reset: false
helm:
  # ... same as any other scenario ...

Or by flag — required for start/stop/reset, which don't read a scenario:

injector-dev apply -f scenario.yaml --workspace firstname-lastname --build
injector-dev stop  --platform=workspace --workspace firstname-lastname

Usage

# bring the remote cluster up + deploy (build local, load remote)
injector-dev apply -f scenario.yaml --workspace <name> --build

# local kubectl now targets the remote cluster through the tunnel
kubectl get pods -A

# tear down the remote cluster, tunnel, and kube-context
injector-dev stop --platform=workspace --workspace <name>

Notes

• start/apply are non-destructive when the cluster already exists: they reuse it and just re-establish the tunnel and switch your kube-context (so apply --reset=false still re-points kubectl at the workspace).
• The SSH tunnel is persistent (survives after the command exits) so local kubectl keeps working. injector-dev stop closes it.
• Inspect open tunnels: ls ~/.injector-dev/*.sock, and check one with ssh -O check -S ~/.injector-dev/workspace-<name>.sock workspace-<name>.
• You can also inspect the cluster on the workspace directly: ssh workspace-<name> then kubectl (kind writes a kubeconfig there too).

The core loop: apply

apply is the primary command. It (optionally) resets/starts the cluster, installs the Datadog stack via Helm or the Operator, deploys any apps/manifests, and waits for health.

injector-dev apply -f workloads/my-feature/scenario.yaml            # deploy
injector-dev apply -f workloads/my-feature/scenario.yaml --build    # build local source first

apply flags

Scenario files

A scenario is helm: or operator:, optionally preceded by a platform: block. Keep each scenario in its own directory alongside its manifests:

workloads/
├── hello-world/
│   └── scenario.yaml
├── my-feature/
│   ├── scenario.yaml
│   └── redis.yaml

Generate a starter template with injector-dev new --type helm --output scenario.yaml (add --edit to open it in $EDITOR).

platform block

platform:
  type: kind              # kind (recommended) | colima | minikube | nvkind | workspace | none
  name: my-dev-cluster    # unique cluster/profile name — give each scenario its own
  reset: false            # false → reuse the cluster if it exists (fast); true → recreate each apply

Precedence for both platform and reset: CLI flag > scenario platform: block > default.

Deploying a pre-built version (simplest case)

---
platform:
  type: kind              # recommended
  name: hello-world
  reset: false
helm:
  versions:
    agent:
      version: "7.81.0"       # use the latest available agent version
    cluster_agent:
      version: "7.81.0"       # use the latest available cluster-agent version
    injector: "0.60.0"        # use the latest available injector version
  config:
    datadog:
      kubelet:
        tlsVerify: false      # needed locally; the kubelet cert usually isn't trusted
    clusterAgent:
      enabled: true

Building from local source

Add build: {} to any component and pass --build:

---
platform:
  type: kind              # recommended
  name: my-dev-cluster
  reset: false
helm:
  versions:
    agent:
      version: "7.81.0"       # use the latest available version
      build: {}             # build agent from local source at code_root/datadog-agent
    cluster_agent:
      version: "7.81.0"       # use the latest available version
      build: {}
    injector:
      version: "0.60.0"       # use the latest available version
      build: {}             # build auto_inject from code_root/auto_inject
  config:
    datadog:
      kubelet:
        tlsVerify: false
    clusterAgent:
      enabled: true

injector-dev apply -f scenario.yaml --build

You can pin a build tag with build: { tag: "dev.1" } (defaults to a git-derived tag otherwise).

Version / image field reference

Each of agent, cluster_agent, injector, csi, (and operator in operator scenarios) accepts either a string or a map:

injector: "0.60.0"          # string → pull this tag from the default repo

agent:                       # map form
  tag: "7.81.0"
  repository: registry.ddbuild.io/ci/datadog-agent/agent   # override the image repo
  pullPolicy: IfNotPresent
  build:                     # presence of `build` → build locally (needs --build)
    tag: "dev.1"

Pin to a CI pipeline / branch artifact — reproduce a coworker's PR build (or any pipeline build) without compiling locally:

helm:
  versions:
    agent:
      repository: registry.ddbuild.io/ci/datadog-agent/agent
      tag: v<PIPELINE>-<COMMIT>-7-amd64
    cluster_agent:
      repository: registry.ddbuild.io/ci/datadog-agent/cluster-agent
      tag: v<PIPELINE>-<COMMIT>-amd64

Full helm: schema

Test apps

Apps are deployed through a shared base chart (schema in apps/base/values.yaml). Sample apps live in apps/: c, dotnet, java, js, php, python, ruby.

helm:
  apps:
    - name: python
      namespace: application
      values:
        image:
          repository: registry.ddbuild.io/ci/injector-dev/python
          tag: "2cd78ded"
        service:
          port: "8080"
        podLabels:
          language: python
          tags.datadoghq.com/env: local
        env:
          - name: DD_TRACE_DEBUG
            value: "true"
          - name: DD_APM_INSTRUMENTATION_DEBUG
            value: "true"

App fields: name, namespace, values (or valuesFile), build (build the app image locally), injector (override injector image per-app), wait.

Kubernetes health checks hit each pod's endpoints, so a running sample app automatically produces traces once instrumentation is enabled — a quick way to confirm injection is working.

Raw manifests & namespaces

helm:
  namespaces:
    - name: cache
      labels:
        team: platform
  manifests:
    - path: "redis-with-password.yaml"   # relative to the scenario file
      namespace: cache                    # auto-created if missing

Local Helm chart

If you're also changing the Datadog Helm chart, point at a local copy. injector-dev then skips the repo add/update and installs from the path:

helm:
  localChartPath: ~/dd/helm-charts/charts/datadog
  versions:
    agent: { version: "7.81.0", build: {} }         # use the latest available version
    cluster_agent: { version: "7.81.0", build: {} }
    injector: "0.60.0"
  config:
    datadog:
      kubelet: { tlsVerify: false }
    clusterAgent: { enabled: true }

Pair with --helm-skip-schema-validation if your local chart adds values the published schema doesn't know about yet.

Operator scenarios

Switch the top-level key to operator:. config becomes the DatadogAgent CRD spec rather than Helm values:

---
platform:
  type: kind              # recommended
  name: operator-example
  reset: false
operator:
  versions:
    operator: "1.28.0"        # use the latest available operator version
    agent: "7.81.0"           # use the latest available version
    cluster_agent: "7.81.0"
    injector: "0.60.0"
  config:
    apiVersion: datadoghq.com/v2alpha1
    kind: DatadogAgent
    metadata:
      name: datadog
    spec:
      features:
        apm:
          instrumentation:
            enabled: true

The operator itself can be built from local source too: set versions.operator.build: {} and pass --build.

Other commands

Standalone builds

build --type accepts: app, agent, cluster-agent, injector, operator, csi.

injector-dev build --type injector
injector-dev build --type cluster-agent
injector-dev build --type app --context ./apps/python

build flags: -t/--type, -n/--name, -c/--context, -f/--dockerfile, -r/--repository, -g/--tag.

Fast-iteration tips

• Set platform.reset: false and a stable platform.name per scenario so applies reuse the cluster instead of recreating it. Override with --reset=true only when you need a clean slate.
• Keep dev_container.persist: true for ~30s rebuilds after the first build.
• Use --skip-agent-validation when the Agent intentionally won't fully start (e.g. testing a failure path) so apply doesn't error out.
• --app-image-tag/-t sets one image tag across all apps at once.
• tlsVerify: false under datadog.kubelet is almost always needed locally.

Troubleshooting

• Agent won't report / auth errors → check DD_API_KEY/DD_APP_KEY (or the active profile's config) and that you're pointed at the right org.
• Kubelet TLS errors → set datadog.kubelet.tlsVerify: false.
• Platform flag "sticks" → if you passed --platform to start, pass it to every apply too, or set platform: in the scenario/config.
• Stale/unhealthy cluster → injector-dev reset (or reset --hard on colima).
• Helm schema rejects new values → --helm-skip-schema-validation.
• Colima/minikube conflicts → stop any pre-existing instances first.
• Add --debug to any command for verbose logs.