# Jenkins Agent L3 Deploy

> Deploy a docker-compose app via a Jenkins INBOUND AGENT running on the target server (no SSH from controller), with Docker-secrets "L3" hardening — runtime secrets live on tmpfs and never appear in `docker inspect` nor in `.env`. Use when the user wants: "deploy via jenkins agent", "no-ssh deploy", "agent thay vì ssh", "docker secrets L3", "ẩn secret khỏi docker inspect", "secret không nằm trên đĩa", or is wiring a Jenkins pipeline that deploys docker-compose to a Linux host. Encodes the gotchas learned from a real 1:1 run (Java 21, 0444 secret perms, 0400 file-credential, agent user needs docker+sudo).

- Skill: `rheinmir/jenkins-agent-l3-deploy` (Agent Skill)
- Install (CLI): `npx skillmds@latest add rheinmir/jenkins-agent-l3-deploy`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rheinmir/jenkins-agent-l3-deploy/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: rheinmir (https://skillmd.com/u/rheinmir)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/rheinmir/jenkins-agent-l3-deploy

---


# Jenkins-agent L3 deploy (no SSH)

Deploy a docker-compose stack where **a Jenkins inbound agent on the target host
runs `deploy.py` + `docker compose` locally**. The controller never SSHes in.
Runtime secrets are written to **tmpfs `/run/payroll-sec` (file 0444, dir 0700)**,
mounted as Docker secrets at `/run/secrets/<name>`, and a tiny entrypoint shim
exports them to env at container start — so they are absent from `docker inspect`
and from `.env`.

## When to use vs the SSH variant
- **Agent (this skill, preferred):** an inbound agent runs on the target → `sh`
  steps execute locally; only a `Secret file` credential (the cred bundle) is
  needed. Controller opens no SSH outbound.
- **SSH variant:** controller `ssh/scp/rsync` into the target (needs an
  `SSH Username with private key` credential). Use only when you cannot run an
  agent on the target.

## Architecture (L3)
```
secret source (cred bundle)
   → deploy provisioner writes 7 files to tmpfs /run/payroll-sec  (file 0444, dir 0700 root)
   → docker-compose `secrets:` mounts each at /run/secrets/<name> (tmpfs, ro)
   → secrets-entrypoint.sh in each image: for f in /run/secrets/*; export $(UPPER basename)=$(cat f); exec "$@"
   ⇒ secret NOT in `docker inspect` (set at runtime), NOT on disk (RAM only)
```
`postgres` uses the official `POSTGRES_PASSWORD_FILE`; app containers use the
entrypoint shim so **no application code changes** (avoids the Edge-runtime `fs`
trap if a framework imports the secret module in middleware).

## Setup the inbound agent (once)
1. Jenkins → Manage Jenkins → Nodes → New Node: Permanent Agent, label = `<target>`,
   Remote root = `/home/<user>/jenkins-agent`, Launch = **inbound (JNLP)**; copy the secret.
2. On the target (agent must reach controller `:8080` + `:50000`):
   ```bash
   sudo apt-get install -y openjdk-21-jre-headless          # Java 21 — NOT 17 (see gotcha 1)
   J21=$(ls /usr/lib/jvm/java-21-openjdk-*/bin/java | head -1)
   mkdir -p ~/jenkins-agent && cd ~/jenkins-agent
   curl -sO http://<CONTROLLER>:8080/jnlpJars/agent.jar
   nohup "$J21" -jar agent.jar -url http://<CONTROLLER>:8080/ \
     -secret <AGENT_SECRET> -name <target> -workDir ~/jenkins-agent > agent.log 2>&1 &
   ```
   Run the agent as a user that is in the **docker group** AND has **sudo NOPASSWD**.
   For production replace `nohup` with a systemd unit (`Restart=always`).
3. Credential: one `Secret file` credential (the cred bundle). No SSH credential.

## Pipeline template (declarative)
```groovy
pipeline {
  agent { label '<target>' }                 // runs ON the target, no ssh
  environment { APP = '/home/<user>/<app-dir>' }
  stages {
    stage('Sync') { steps { sh '''
      set -e
      cp -f docker-compose.yml deploy.py.example "$APP/"
      cp -f web/Dockerfile web/secrets-entrypoint.sh "$APP/web/"
      cp -f etl/Dockerfile etl/secrets-entrypoint.sh "$APP/etl/"
      cd "$APP" && cp -f deploy.py.example deploy.py
    ''' } }
    stage('Deploy') { steps { withCredentials([file(credentialsId:'app-cred', variable:'CRED')]) { sh '''
      set -e
      mkdir -p /dev/shm/pl && chmod 700 /dev/shm/pl
      cp "$CRED" /dev/shm/pl/cred.txt && chmod 600 /dev/shm/pl/cred.txt   # gotcha 3
      cd "$APP" && ln -sf /dev/shm/pl/cred.txt cred.txt
      sudo python3 deploy.py --no-nginx --etl-url /etl
      rm -f cred.txt; shred -uf /dev/shm/pl/cred.txt 2>/dev/null || rm -f /dev/shm/pl/cred.txt
    ''' } } }
    stage('Verify') { steps { sh '''
      set -e
      for c in <containers>; do
        docker inspect "$c" --format "{{range .Config.Env}}{{println .}}{{end}}" \
          | grep -iqE "SESSION_SECRET=|AUTH_USERS=|PASSWORD=|DATABASE_URL=" && { echo "LEAK $c"; exit 1; } || echo "CLEAN $c"
        docker exec "$c" ls /run/secrets >/dev/null 2>&1 && echo "MOUNT $c" || { echo "NOMOUNT $c"; exit 1; }
      done
    ''' } }
  }
}
```
Provide the workspace code via SCM checkout, or pre-populate the agent workspace.

## secrets-entrypoint.sh (baked into each app image; ENTRYPOINT before CMD)
```sh
#!/bin/sh
set -e
if [ -d /run/secrets ]; then
  for f in /run/secrets/*; do [ -f "$f" ] || continue
    name=$(basename "$f" | tr '[:lower:]' '[:upper:]'); export "$name=$(cat "$f")"
  done
fi
exec "$@"
```
The provisioner (e.g. `deploy.py`) writes the secret files to a tmpfs dir
**mode 0444 (files) / 0700 (dir)**, then `docker compose up -d`.

## Gotchas (each cost a failed real run — bake these in)
| # | Symptom | Fix |
|---|---------|-----|
| 1 | Agent dies `UnsupportedClassVersionError: class 65.0 … up to 61.0` | Jenkins 2.5xx agent.jar needs **Java 21**, not 17. |
| 2 | App container crash-loops `Permission denied /run/secrets/*` | Compose non-swarm bind-mounts the source file's perms; write secret files **0444** (dir 0700) so the non-root container user can read. |
| 3 | `shred /dev/shm/.../cred: Permission denied` | Jenkins `file` credential is delivered **0400 (read-only)**; `cp` then `chmod 600`, and use `shred -uf … || rm -f`. |
| 4 | `docker`/`sudo` "permission denied" inside the pipeline | The agent process user must be in the **docker group** and have **sudo NOPASSWD**. |
| 5 | Reboot → containers fail to mount secret | `/run` is tmpfs → secrets cleared on reboot; re-run the deploy (or a boot hook) to re-materialize before `compose up`. |

## Verify success (real signals)
- Build console shows `Running on <target>` (proves it ran on the agent, not controller).
- `CLEAN <c>` for every container (secret absent from `docker inspect`).
- `MOUNT <c>` (secret present at `/run/secrets`).
- App smoke (e.g. login) returns 200.
- The cred bundle is shredded from `/dev/shm` (not left on disk).

## Selective deploy (deploy only some services, not full)
Expose `booleanParam` checkboxes (WEB/ETL/DB/NGINX/ALL) and pass the chosen compose
services to the provisioner. Use `--no-deps` so a tick rebuilds **only that service**
(not its dependency chain):
```sh
if [ "$ALL" = true ]; then SVC=""; else
  S=""; [ "$WEB" = true ] && S="$S web"; [ "$ETL" = true ] && S="$S etl"; [ "$DB" = true ] && S="$S postgres"
  [ -z "$(echo $S|xargs)" ] && SVC=none || SVC=$(echo $S|xargs)
fi
deploy.py --services "$SVC"        # "" = full ; "web etl" = those ; "none" = secrets only, no up
# provisioner runs: docker compose up -d --build --no-deps $SVC
[ "$NGINX" = true ] && docker restart <proxy-container>   # proxy is a separate container, not compose
```
**Dependency graph (depends_on) — document it** so operators know a fresh deploy needs the chain:
`web → etl → postgres` (proxy/nginx independent). With `--no-deps`, tick web = ONLY web; for a
clean first deploy tick the whole chain (or ALL). Verified: single ticks + combos all touch exactly
the chosen services.

## Pulling code from git
Production: configure the job as **Pipeline script from SCM** (git repo + scriptPath); Jenkins
auto-`checkout scm` into the agent workspace, then the pipeline runs. The git URL must be reachable
by BOTH controller (fetch Jenkinsfile) and agent (checkout workspace) — `file://` only works on the
side that owns it; in a lab without a git server, have the agent `git clone file:///…/repo.git`.

**Private repo (tested with GitHub, build SUCCESS):**
- HTTPS anonymous clone fails for a private repo — needs auth.
- Use an **SSH deploy key** (read-only) on the repo; put the private key in a Jenkins
  `SSH Username with private key` credential (username `git`); SCM URL `git@github.com:owner/repo.git`
  with that credentialsId.
- **Host key gotcha:** add `github.com` to known_hosts on controller AND agent
  (`ssh-keyscan github.com >> ~/.ssh/known_hosts`) or set the git host-key strategy to
  "Accept first connection", else the SSH clone fails host-key verification.
- The deploy key's public part on GitHub must match the private key in Jenkins (compare
  `ssh-keygen -lf key.pub` fingerprint vs what GitHub shows).

## Don't
- Don't pass secrets via compose `environment:` (visible in `docker inspect`, written to `.env`).
- Don't store the secret files 0600 (non-root container can't read) — use 0444 + dir 0700.
- Don't rely on the SSH variant if you can run an agent — agent keeps SSH closed and runs locally.
- Don't point a `file://` SCM URL at a path the controller can't see — use a network git URL.