Jenkins Agent L3 Deploy

Jenkins-agent L3 deploy (no SSH)

Deploy a docker-compose stack where a Jenkins inbound agent on the target host runs deploy.py + docker compose locally. The controller never SSHes in. Runtime secrets are written to tmpfs /run/payroll-sec (file 0444, dir 0700), mounted as Docker secrets at /run/secrets/<name>, and a tiny entrypoint shim exports them to env at container start — so they are absent from docker inspect and from .env.

When to use vs the SSH variant

• Agent (this skill, preferred): an inbound agent runs on the target → sh steps execute locally; only a Secret file credential (the cred bundle) is needed. Controller opens no SSH outbound.
• SSH variant: controller ssh/scp/rsync into the target (needs an SSH Username with private key credential). Use only when you cannot run an agent on the target.

Architecture (L3)

secret source (cred bundle)
   → deploy provisioner writes 7 files to tmpfs /run/payroll-sec  (file 0444, dir 0700 root)
   → docker-compose `secrets:` mounts each at /run/secrets/<name> (tmpfs, ro)
   → secrets-entrypoint.sh in each image: for f in /run/secrets/*; export $(UPPER basename)=$(cat f); exec "$@"
   ⇒ secret NOT in `docker inspect` (set at runtime), NOT on disk (RAM only)

postgres uses the official POSTGRES_PASSWORD_FILE; app containers use the entrypoint shim so no application code changes (avoids the Edge-runtime fs trap if a framework imports the secret module in middleware).

Setup the inbound agent (once)

1. Jenkins → Manage Jenkins → Nodes → New Node: Permanent Agent, label = <target>, Remote root = /home/<user>/jenkins-agent, Launch = inbound (JNLP); copy the secret.
2. On the target (agent must reach controller :8080 + :50000):

   sudo apt-get install -y openjdk-21-jre-headless          # Java 21 — NOT 17 (see gotcha 1)
   J21=$(ls /usr/lib/jvm/java-21-openjdk-*/bin/java | head -1)
   mkdir -p ~/jenkins-agent && cd ~/jenkins-agent
   curl -sO http://<CONTROLLER>:8080/jnlpJars/agent.jar
   nohup "$J21" -jar agent.jar -url http://<CONTROLLER>:8080/ \
     -secret <AGENT_SECRET> -name <target> -workDir ~/jenkins-agent > agent.log 2>&1 &
   

   Run the agent as a user that is in the docker group AND has sudo NOPASSWD. For production replace nohup with a systemd unit (Restart=always).
3. Credential: one Secret file credential (the cred bundle). No SSH credential.

Pipeline template (declarative)

pipeline {
  agent { label '<target>' }                 // runs ON the target, no ssh
  environment { APP = '/home/<user>/<app-dir>' }
  stages {
    stage('Sync') { steps { sh '''
      set -e
      cp -f docker-compose.yml deploy.py.example "$APP/"
      cp -f web/Dockerfile web/secrets-entrypoint.sh "$APP/web/"
      cp -f etl/Dockerfile etl/secrets-entrypoint.sh "$APP/etl/"
      cd "$APP" && cp -f deploy.py.example deploy.py
    ''' } }
    stage('Deploy') { steps { withCredentials([file(credentialsId:'app-cred', variable:'CRED')]) { sh '''
      set -e
      mkdir -p /dev/shm/pl && chmod 700 /dev/shm/pl
      cp "$CRED" /dev/shm/pl/cred.txt && chmod 600 /dev/shm/pl/cred.txt   # gotcha 3
      cd "$APP" && ln -sf /dev/shm/pl/cred.txt cred.txt
      sudo python3 deploy.py --no-nginx --etl-url /etl
      rm -f cred.txt; shred -uf /dev/shm/pl/cred.txt 2>/dev/null || rm -f /dev/shm/pl/cred.txt
    ''' } } }
    stage('Verify') { steps { sh '''
      set -e
      for c in <containers>; do
        docker inspect "$c" --format "{{range .Config.Env}}{{println .}}{{end}}" \
          | grep -iqE "SESSION_SECRET=|AUTH_USERS=|PASSWORD=|DATABASE_URL=" && { echo "LEAK $c"; exit 1; } || echo "CLEAN $c"
        docker exec "$c" ls /run/secrets >/dev/null 2>&1 && echo "MOUNT $c" || { echo "NOMOUNT $c"; exit 1; }
      done
    ''' } }
  }
}

Provide the workspace code via SCM checkout, or pre-populate the agent workspace.

secrets-entrypoint.sh (baked into each app image; ENTRYPOINT before CMD)

#!/bin/sh
set -e
if [ -d /run/secrets ]; then
  for f in /run/secrets/*; do [ -f "$f" ] || continue
    name=$(basename "$f" | tr '[:lower:]' '[:upper:]'); export "$name=$(cat "$f")"
  done
fi
exec "$@"

The provisioner (e.g. deploy.py) writes the secret files to a tmpfs dir mode 0444 (files) / 0700 (dir), then docker compose up -d.

Gotchas (each cost a failed real run — bake these in)

Verify success (real signals)

• Build console shows Running on <target> (proves it ran on the agent, not controller).
• CLEAN <c> for every container (secret absent from docker inspect).
• MOUNT <c> (secret present at /run/secrets).
• App smoke (e.g. login) returns 200.
• The cred bundle is shredded from /dev/shm (not left on disk).

Selective deploy (deploy only some services, not full)

Expose booleanParam checkboxes (WEB/ETL/DB/NGINX/ALL) and pass the chosen compose services to the provisioner. Use --no-deps so a tick rebuilds only that service (not its dependency chain):

if [ "$ALL" = true ]; then SVC=""; else
  S=""; [ "$WEB" = true ] && S="$S web"; [ "$ETL" = true ] && S="$S etl"; [ "$DB" = true ] && S="$S postgres"
  [ -z "$(echo $S|xargs)" ] && SVC=none || SVC=$(echo $S|xargs)
fi
deploy.py --services "$SVC"        # "" = full ; "web etl" = those ; "none" = secrets only, no up
# provisioner runs: docker compose up -d --build --no-deps $SVC
[ "$NGINX" = true ] && docker restart <proxy-container>   # proxy is a separate container, not compose

Dependency graph (depends_on) — document it so operators know a fresh deploy needs the chain: web → etl → postgres (proxy/nginx independent). With --no-deps, tick web = ONLY web; for a clean first deploy tick the whole chain (or ALL). Verified: single ticks + combos all touch exactly the chosen services.

Pulling code from git

Production: configure the job as Pipeline script from SCM (git repo + scriptPath); Jenkins auto-checkout scm into the agent workspace, then the pipeline runs. The git URL must be reachable by BOTH controller (fetch Jenkinsfile) and agent (checkout workspace) — file:// only works on the side that owns it; in a lab without a git server, have the agent git clone file:///…/repo.git.

Private repo (tested with GitHub, build SUCCESS):

• HTTPS anonymous clone fails for a private repo — needs auth.
• Use an SSH deploy key (read-only) on the repo; put the private key in a Jenkins SSH Username with private key credential (username git); SCM URL git@github.com:owner/repo.git with that credentialsId.
• Host key gotcha: add github.com to known_hosts on controller AND agent (ssh-keyscan github.com >> ~/.ssh/known_hosts) or set the git host-key strategy to "Accept first connection", else the SSH clone fails host-key verification.
• The deploy key's public part on GitHub must match the private key in Jenkins (compare ssh-keygen -lf key.pub fingerprint vs what GitHub shows).

Don't

• Don't pass secrets via compose environment: (visible in docker inspect, written to .env).
• Don't store the secret files 0600 (non-root container can't read) — use 0444 + dir 0700.
• Don't rely on the SSH variant if you can run an agent — agent keeps SSH closed and runs locally.
• Don't point a file:// SCM URL at a path the controller can't see — use a network git URL.