# Rseng CI CD > Covers continuous integration and delivery for research software: CI/CD concepts, automating builds and tests with GitHub Actions and GitLab CI/CD, and wiring an organization's GitLab CI infrastructure to a GitHub-hosted project. Use when the user asks to set up CI, write a pipeline or workflow file (.github/workflows, .gitlab-ci.yml), add automated builds/tests on push or pull request, publish a package or deploy docs when a tag or release is cut, cache dependencies to speed up CI runs, choose between GitHub Actions and GitLab CI, use self-hosted or GPU runners, mirror a repo, or report external CI status back to GitHub. For what to test and shrinking a test matrix see rseng-testing; for linter and pre-commit policy see rseng-code-quality. - Skill: `fdiblen/rseng-ci-cd` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add fdiblen/rseng-ci-cd` - Raw SKILL.md: https://api.skillmd.com/api/skills/fdiblen/rseng-ci-cd/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: CC-BY-4.0 - Author: fdiblen (https://skillmd.com/u/fdiblen) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/fdiblen/rseng-ci-cd --- # Continuous integration and delivery for research software Use this skill when a project needs to automate its build, test and release path, or when someone is deciding how to run CI for research software. It covers CI/CD concepts, GitHub Actions, GitLab CI/CD, and hybrid setups where a GitHub project runs on an organization's GitLab runners. Prefer concrete, minimal pipelines first, then grow them; do not scaffold monitoring, mirroring or multi-runner infrastructure a small project will not use. ## Core concepts Distinguish the three practices before recommending tooling: - Continuous Integration (CI): integrate code changes into a shared repository frequently, and verify every change with an automated build that compiles and runs tests. Catches integration errors early. - Continuous Delivery (CD): keep software always in a deployable state; the release to production stays a manual decision. Value: fast, safe rollbacks and few manual release steps. - Continuous Deployment (CD): go further and deploy every passing change to production automatically, with no manual gate. When advising, name which of the three the user actually wants. Most research projects need solid CI plus Continuous Delivery, not fully automated Continuous Deployment. ## What a pipeline should cover Map the project's needs onto these stages before writing YAML: - Automated builds triggered on every change (compile plus tests). - Version-control integration so commits and pull requests trigger runs. - Automated unit, integration and regression tests across the platforms and dependency versions the software claims to support. - Code-quality gates: static analysis or linters, and pre-commit (https://pre-commit.com/) hooks run in CI, not only locally. - Deployment pipelines that package artifacts ready to ship. - Monitoring and rollback for anything deployed to production. ## Choosing a platform Match the tool to where the repository lives and what hardware it needs: - GitHub-hosted, standard hardware: use GitHub Actions. It is integrated, has a large marketplace, and free runners for public repos. - GitLab-hosted: use GitLab CI/CD. It is the GitLab-native equivalent of GitHub Actions. - Migrating from Actions to GitLab: point the user at GitLab's dedicated Actions-migration guide rather than rewriting by hand. - Specialized hardware (GPUs, ARM/Power CPUs, HPC) or heavy test matrices that exceed GitHub's free tier: consider running on an organization's GitLab runners, even for a GitHub-hosted project (see the hybrid section). - Other established options exist (Jenkins, CircleCI, Travis CI, Azure Pipelines, Bitbucket Pipelines); recommend them only when the project is already on that ecosystem. ## GitHub Actions Set up a workflow with these steps: - Create a `.github/workflows/` directory; put one YAML file (`.yml` or `.yaml`) per workflow there. - Choose triggers under `on:` -- typically `push`, `pull_request`, or a `schedule`. Pick the events that match how the team collaborates. - Select a runner OS with `runs-on:` (Ubuntu, Windows, or macOS); use a matrix when the software must work across several. - Reuse marketplace actions instead of hand-rolling steps: `actions/checkout` to check out the code and `actions/setup-python` (or the relevant language setup) to provision the toolchain. A major tag like `@v7` is readable but mutable; pinning means a full commit SHA with the tag in a trailing comment, which is what rseng-security asks for on third-party actions. - Store credentials and API keys as repository secrets; never inline them in the workflow file. - After pushing, read results in the Actions tab: the run graph shows job sequence, and expanding a job exposes per-step logs for debugging. Minimal test workflow for a Python project: ```yaml name: Tests on: [push, pull_request] jobs: test: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v7 - name: Setup Python uses: actions/setup-python@v7 with: python-version: '3.13' - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Run tests run: pytest ``` Adapt the language setup and install command to the project; keep the first working pipeline this small, then extend. ## GitLab CI/CD Set up a pipeline with these steps: - Create a `.gitlab-ci.yml` at the repository root; it defines pipelines and jobs in YAML. - Runners: on gitlab.com, instance runners are provided automatically; on a self-hosted GitLab, confirm runners are available first. - Structure work into `stages`; jobs in the same stage run in parallel (given enough runners), and a later stage starts only after the previous one finishes. Each job needs at least `stage` and `script`. - If one job needs another's output, put them in separate stages and pass results as artifacts -- parallel jobs in one stage cannot depend on each other. - Reuse pre-built components from the GitLab CI/CD Catalog rather than writing everything from scratch; most ship a README with configuration. - Watch runs in the Pipelines tab, drilling into stages and jobs for status and logs. Skeleton `.gitlab-ci.yml` showing the stage/job structure: ```yaml stages: - build - deploy build-job: stage: build script: - echo "Starting build" deploy-job: stage: deploy script: - echo "Deploy" ``` For real pipelines, point users at GitLab's quick-start and advanced tutorials and its per-language examples rather than inventing job scripts. ## Hybrid: GitLab CI for a GitHub-hosted project Only recommend this when a GitHub project genuinely needs hardware or capacity beyond GitHub's free runners (GPUs, ARM/Power, HPC, large matrices) and the organization already runs GitLab with suitable runners. It is real operational overhead -- flag that cost explicitly to small teams before they commit. The moving parts to design: - Repository mirroring: use GitLab native mirroring to sync the GitHub repo into GitLab. Mirror only main/protected branches initially to limit load and exposure; mirroring needs a deploy key or token with write access to the GitHub repo. - Fork integration: forks complicate external CI. A webhook-driven bot can watch pull-request events and create matching branches in the GitLab mirror using a systematic naming convention; clean those branches up when the PR closes. - Webhook security: validate webhook signatures with a secret, return HTTP 200 immediately and process asynchronously, and retry failed GitLab API calls with exponential backoff (dead-letter events that never succeed). - Status reporting: push GitLab pipeline results back to GitHub via the commit status API, keyed on commit hash, using stable check names like `gitlab-ci/` so PR checks reflect the external run. - Runners: tag runners by capability (for example `gpu`, `nvidia`, `arm64`) and target them from jobs with matching `tags:`. Organize as shared, group, and project-specific runners so specialized hardware is allocated fairly. - Access and security: map GitHub permissions to GitLab roles (admin -> maintainer, write -> developer, read -> reporter), grant external contributors time-limited guest access to logs, and keep audit logs of webhooks, mirroring, access changes and pipeline triggers. - Reliability and monitoring: rotate API tokens (updating webhook configs in lockstep so auth does not desync), and monitor webhook success rate, mirror-sync delay and runner availability with alert thresholds. Document the hybrid setup for contributors: how to read status checks, reach GitLab logs, and request runner access. A concrete workflow set for research Python packages (NLeSC python-template): build, link-check, docs build, SonarCloud analysis and cffconvert citation validation - plus its next_steps.yml pattern, which opens onboarding issues for manual setup steps (Zenodo, SonarCloud, Read the Docs) so scaffolded projects finish their setup. ## Working with this skill The generated references.md beside this file lists the source material and pointers: - references.md - verified Learn more pointers Learn more (verified): - https://docs.github.com/en/actions - GitHub Actions documentation - https://docs.gitlab.com/ci/ - GitLab CI/CD documentation - https://coderefinery.github.io/testing/ - CodeRefinery automated testing lesson - https://book.the-turing-way.org/reproducible-research/ci - Turing Way continuous integration chapter ## Related skills Check whether any of these applies before moving on: - rseng-green-computing - pipeline energy and runner cost - rseng-hpc-computing - GPU/HPC runners for heavy jobs - rseng-publishing-releasing - release automation triggered from CI - rseng-reproducible-environments - testing inside the shipped container - rseng-security - secrets management in pipelines - rseng-testing - what the pipeline should actually run