Concourse CI Pipeline Development
Expert guidance for writing, refactoring, and optimizing Concourse CI pipelines (v8.0+).
When to Use
- Creating or modifying Concourse pipelines
- Configuring resources (git, registry-image, custom types)
- Building container images with
oci-build-task
- Troubleshooting resource check failures or build issues
- Migrating from legacy patterns (docker-image, duplicate jobs)
Quick Reference
| Task |
Modern (Recommended) |
Legacy (Avoid) |
| Building images |
oci-build-task + registry-image |
docker-image resource |
| Multi-env deploys |
across step modifier |
Duplicate jobs per env |
| Dynamic pipelines |
set_pipeline + instanced pipelines |
Manual pipeline duplication |
| Notification symbols |
UTF-8 characters (e.g. \u2714 for checkmark, \u274c for X) |
HTML entities (e.g. ✓, ✗) |
| Resource styling |
Always use icon: property |
No icon |
Core Concepts
Pipelines consist of resources (external versioned artifacts), jobs (sequences of steps), and optional groups (UI organization). All execution runs in containers.
Key step types: get, put, task, set_pipeline, in_parallel, do, try, load_var. Job hooks: on_success, on_failure, on_error, on_abort, ensure. Note: on_failure (non-zero exit) differs from on_error (infrastructure crash/OOM) -- handle both. Use fly execute to test tasks locally.
See references/core-concepts.md for step types table, lifecycle hooks, and fly CLI essentials.
Critical Gotchas
- Git tag detection after force-push -- Escape regex dots, enable
clean_tags: true, separate read/write resources, force recheck with fly -t T check-resource -r pipeline/resource. See references/resources-guide.md.
- registry_mirror format mismatch --
registry-image expects an object (host: mirror), docker-image expects a URL string. Provide separate formats in CONCOURSE_BASE_RESOURCE_TYPE_DEFAULTS. See references/resources-guide.md.
- GitLab Container Registry JWT auth -- The JWT endpoint lives on the GitLab host, not the registry host. Discover via
Www-Authenticate header. See references/resources-guide.md.
- git push --mirror and default branch -- Target repo's default branch must exist upstream. If absent, the pre-receive hook rejects the push with "pre-receive hook declined". Set it before the first mirror push.
References
references/pipeline-syntax.md -- Complete YAML schema for pipelines, jobs, resourcesreferences/core-concepts.md -- Step types, lifecycle hooks, fly CLI essentialsreferences/resources-guide.md -- Git-resource, registry-image, docker-image migration, gotcha detailsreferences/best-practices.md -- Optimization, troubleshooting, notifications, deployment patternsreferences/resource-types-catalog.md -- Available resource types (Ansible, Terraform, etc.)
Examples
Working examples in examples/:
basic-pipeline.yml -- Build-test-deploy with versioningmodern-ci-cd.yml -- oci-build-task, across, build_log_retentionmulti-branch.yml -- Dynamic branch pipelines with set_pipelinedocker-build.yml -- Container image build and pushvars-template.yml -- Variable file organization
Validation
Use scripts/validate-pipeline.sh to check pipeline syntax before deployment.
1---2name: concourse-ci3description: Use when working with ANY Concourse CI task: writing pipelines, configuring resources, building images with oci-build-task, troubleshooting failing jobs, migrating from legacy patterns, or optimizing CI/CD. Triggers on: Concourse, pipeline, fly CLI, resource type, oci-build-task, set_pipeline, concourse.yml.4license: (MIT AND CC-BY-SA-4.0)5---67# Concourse CI Pipeline Development89Expert guidance for writing, refactoring, and optimizing Concourse CI pipelines (v8.0+).1011## When to Use1213- Creating or modifying Concourse pipelines14- Configuring resources (git, registry-image, custom types)15- Building container images with `oci-build-task`16- Troubleshooting resource check failures or build issues17- Migrating from legacy patterns (docker-image, duplicate jobs)1819## Quick Reference2021| Task | Modern (Recommended) | Legacy (Avoid) |22|------|---------------------|----------------|23| Building images | `oci-build-task` + `registry-image` | `docker-image` resource |24| Multi-env deploys | `across` step modifier | Duplicate jobs per env |25| Dynamic pipelines | `set_pipeline` + instanced pipelines | Manual pipeline duplication |26| Notification symbols | UTF-8 characters (e.g. `\u2714` for checkmark, `\u274c` for X) | HTML entities (e.g. `✓`, `✗`) |27| Resource styling | Always use `icon:` property | No icon |2829## Core Concepts3031Pipelines consist of **resources** (external versioned artifacts), **jobs** (sequences of steps), and optional **groups** (UI organization). All execution runs in containers.3233Key step types: `get`, `put`, `task`, `set_pipeline`, `in_parallel`, `do`, `try`, `load_var`. Job hooks: `on_success`, `on_failure`, `on_error`, `on_abort`, `ensure`. Note: `on_failure` (non-zero exit) differs from `on_error` (infrastructure crash/OOM) -- handle both. Use `fly execute` to test tasks locally.3435See `references/core-concepts.md` for step types table, lifecycle hooks, and fly CLI essentials.3637## Critical Gotchas38391. **Git tag detection after force-push** -- Escape regex dots, enable `clean_tags: true`, separate read/write resources, force recheck with `fly -t T check-resource -r pipeline/resource`. See `references/resources-guide.md`.402. **registry_mirror format mismatch** -- `registry-image` expects an object (`host: mirror`), `docker-image` expects a URL string. Provide separate formats in `CONCOURSE_BASE_RESOURCE_TYPE_DEFAULTS`. See `references/resources-guide.md`.413. **GitLab Container Registry JWT auth** -- The JWT endpoint lives on the GitLab host, not the registry host. Discover via `Www-Authenticate` header. See `references/resources-guide.md`.424. **git push --mirror and default branch** -- Target repo's default branch must exist upstream. If absent, the pre-receive hook rejects the push with "pre-receive hook declined". Set it before the first mirror push.4344## References4546- `references/pipeline-syntax.md` -- Complete YAML schema for pipelines, jobs, resources47- `references/core-concepts.md` -- Step types, lifecycle hooks, fly CLI essentials48- `references/resources-guide.md` -- Git-resource, registry-image, docker-image migration, gotcha details49- `references/best-practices.md` -- Optimization, troubleshooting, notifications, deployment patterns50- `references/resource-types-catalog.md` -- Available resource types (Ansible, Terraform, etc.)5152### Examples5354Working examples in `examples/`:55- `basic-pipeline.yml` -- Build-test-deploy with versioning56- `modern-ci-cd.yml` -- oci-build-task, across, build_log_retention57- `multi-branch.yml` -- Dynamic branch pipelines with set_pipeline58- `docker-build.yml` -- Container image build and push59- `vars-template.yml` -- Variable file organization6061### Validation6263Use `scripts/validate-pipeline.sh` to check pipeline syntax before deployment.
netresearch