Azure Pipelines

azure-pipelines Best Practices

Azure Pipelines are the backbone of modern CI/CD. This guide enforces a YAML-first, security-conscious, and highly automated approach to pipeline development.

Code Organization and Structure

Always define pipelines using YAML for version control, reusability, and clarity. Structure your pipelines using multi-stage designs and leverage templates for common patterns.

1. Embrace Multi-Stage Pipelines

Separate distinct phases (Build, Test, Deploy) into stages. This provides clear logical separation, enables independent quality gates, and improves visibility.

❌ BAD: Monolithic pipeline with all steps in one job.

# azure-pipelines.yml
jobs:
- job: BuildAndDeploy
  steps:
  - script: dotnet build
  - script: dotnet test
  - script: az webapp deploy

✅ GOOD: Clearly defined stages for build, test, and deployment.

# azure-pipelines.yml
stages:
- stage: Build
  jobs:
  - job: BuildApp
    steps:
    - script: dotnet build
    - publish: $(Build.ArtifactStagingDirectory)
      artifact: drop

- stage: Test
  dependsOn: Build
  jobs:
  - job: RunTests
    steps:
    - download: current
      artifact: drop
    - script: dotnet test --logger "trx;LogFileName=test-results.trx"
    - task: PublishTestResults@2
      inputs:
        testResultsFormat: 'VSTest'
        testResultsFiles: '**/*.trx'

- stage: Deploy
  dependsOn: Test
  condition: succeeded('Test') # Only deploy if tests pass
  jobs:
  - deployment: DeployApp
    environment: 'Production' # Use environments for approvals and checks
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: drop
          - script: az webapp deploy --resource-group MyResourceGroup --name MyApp --src-path $(Pipeline.Workspace)/drop

2. Leverage Templates for Reusability

Define common stages, jobs, or steps in reusable templates. This keeps your pipelines DRY, consistent, and easier to maintain.

❌ BAD: Copy-pasting build steps across multiple azure-pipelines.yml files.

# service-a/azure-pipelines.yml
stages:
- stage: Build
  jobs:
  - job: BuildServiceA
    steps:
    - script: npm install
    - script: npm run build
    # ... more steps

# service-b/azure-pipelines.yml
stages:
- stage: Build
  jobs:
  - job: BuildServiceB
    steps:
    - script: npm install
    - script: npm run build
    # ... identical steps

✅ GOOD: Centralize common build logic in a template.

# templates/build-node-app.yml
parameters:
- name: appName
  type: string
  default: 'app'

jobs:
- job: BuildNodeApp
  displayName: Build ${{ parameters.appName }}
  steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '18.x'
    displayName: 'Install Node.js'
  - script: npm install
    displayName: 'Install Dependencies'
  - script: npm run build
    displayName: 'Build App'
  - publish: $(Build.ArtifactStagingDirectory)
    artifact: drop-${{ parameters.appName }}
    displayName: 'Publish Build Artifact'

# service-a/azure-pipelines.yml
stages:
- stage: Build
  jobs:
  - template: ../templates/build-node-app.yml
    parameters:
      appName: ServiceA

# service-b/azure-pipelines.yml
stages:
- stage: Build
  jobs:
  - template: ../templates/build-node-app.yml
    parameters:
      appName: ServiceB

Common Patterns and Anti-patterns

3. Secure Secrets with Azure Key Vault

Never hardcode secrets. Always use Azure Key Vault linked to Variable Groups for secure secret management.

❌ BAD: Storing secrets directly in Variable Groups or YAML.

# Variable Group 'MySecrets' (in Azure DevOps UI)
#   MyApiKey: "SuperSecretKey123" (plaintext)

# azure-pipelines.yml
variables:
  MyApiKey: 'SuperSecretKey123' # Directly in YAML

✅ GOOD: Link Variable Group to Azure Key Vault.

# Azure Key Vault: Create a secret named 'MyApiKey'
# Azure DevOps Variable Group 'MySecrets': Link to Key Vault, select 'MyApiKey'

# azure-pipelines.yml
variables:
- group: MySecrets # Link to the Variable Group
stages:
- stage: Deploy
  jobs:
  - job: DeployApp
    steps:
    - script: echo "Using API Key: $(MyApiKey)" # Access as a regular variable

Note: $(MyApiKey) will be masked in logs if it's a secret variable.

4. Use Workload Identity Federation for Service Connections

Prefer Workload Identity Federation (OIDC) over Service Principals for Azure Service Connections. It eliminates the need for manual secret management.

❌ BAD: Service Connection using a Service Principal with a client secret.

# Azure DevOps Service Connection: Azure Resource Manager
#   Authentication Method: Service principal (manual)
#   Client secret: Manually managed and rotated

✅ GOOD: Service Connection using Workload Identity Federation.

# Azure DevOps Service Connection: Azure Resource Manager
#   Authentication Method: Workload Identity Federation (automatic)
#   No client secret to manage

When creating a new Azure Resource Manager service connection, choose "Workload Identity Federation (automatic)" if available.

Performance Considerations

5. Cache Dependencies

Speed up builds by caching frequently used dependencies (e.g., node_modules, pip packages, dotnet NuGet packages).

❌ BAD: Installing dependencies from scratch on every build.

# azure-pipelines.yml
steps:
- script: npm install
  displayName: 'Install Node Modules'
- script: dotnet restore
  displayName: 'Restore NuGet Packages'

✅ GOOD: Use the Cache@2 task.

# azure-pipelines.yml
steps:
- task: Cache@2
  inputs:
    key: 'npm | "$(Agent.OS)" | **/package-lock.json'
    path: '$(npm_config_cache)'
    cacheHitVar: 'NPM_CACHE_RESTORED'
  displayName: 'Cache npm packages'
- script: npm install
  displayName: 'Install Node Modules'
  condition: ne(variables.NPM_CACHE_RESTORED, 'true')

- task: Cache@2
  inputs:
    key: 'nuget | "$(Agent.OS)" | $(Build.SourcesDirectory)/**/*.csproj'
    path: '$(NUGET_PACKAGES)'
    cacheHitVar: 'NUGET_CACHE_RESTORED'
  displayName: 'Cache NuGet packages'
- script: dotnet restore
  displayName: 'Restore NuGet Packages'
  condition: ne(variables.NUGET_CACHE_RESTORED, 'true')

6. Run Jobs in Parallel

For independent tasks, run jobs in parallel to reduce overall pipeline execution time.

❌ BAD: Sequential jobs when they could run concurrently.

jobs:
- job: Build
  # ...
- job: Lint
  dependsOn: Build # Unnecessary dependency
  # ...
- job: UnitTests
  dependsOn: Build # Unnecessary dependency
  # ...

✅ GOOD: Parallelize independent jobs.

jobs:
- job: Build
  # ...
- job: Lint
  dependsOn: [] # No dependency on Build
  # ...
- job: UnitTests
  dependsOn: Build # Unit tests might need build artifacts
  # ...

Common Pitfalls and Gotchas

7. Avoid Exposing Secrets to Fork Builds

When working with public repositories, prevent secrets from being exposed to builds from forks.

❌ BAD: Enabling "Make secrets available to builds of forks" in pipeline settings.

✅ GOOD: Keep "Make secrets available to builds of forks" disabled. Manually trigger fork builds after review if necessary.

8. Use Project-Scoped Build Identities

Restrict pipeline permissions to the project level to minimize lateral exposure.

❌ BAD: Using a collection-level build identity (Project Collection Build Service (YourCollectionName)).

✅ GOOD: Use the default project-level build identity (YourProjectName Build Service (YourOrganizationName)). This is the default for new YAML pipelines. Verify Job authorization scope is set to Project Collection or Current project as appropriate for your needs, but generally Current project is more secure.

Configuration Management

9. Define Infrastructure as Code (IaC)

Manage your infrastructure (Azure resources, environments) as code (e.g., Bicep, ARM templates, Terraform) and deploy it via pipelines.

❌ BAD: Manually provisioning environments or using UI-driven deployments only.

✅ GOOD: Deploy infrastructure using a dedicated IaC stage.

# azure-pipelines.yml
stages:
- stage: DeployInfrastructure
  jobs:
  - deployment: DeployBicep
    environment: 'DevEnvironment'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureCLI@2
            inputs:
              azureSubscription: 'MyAzureSubscription'
              scriptType: 'bash'
              scriptLocation: 'inlineScript'
              inlineScript: |
                az group create --name $(ResourceGroupName) --location $(Location)
                az deployment group create \
                  --resource-group $(ResourceGroupName) \
                  --template-file ./infra/main.bicep \
                  --parameters environmentType=dev

Environment Variables

10. Use Predefined Variables Correctly

Leverage Azure Pipelines' predefined variables for consistent and reliable access to build, release, and agent information.

❌ BAD: Hardcoding paths or trying to guess build numbers.

steps:
- script: echo "Build number: 1.0.0.$(date +%s)" # Inconsistent
- script: cp ./dist/* /home/vsts/work/1/a/ # Fragile path

✅ GOOD: Use predefined variables.

steps:
- script: echo "Build number: $(Build.BuildNumber)"
- script: cp $(Build.SourcesDirectory)/dist/* $(Build.ArtifactStagingDirectory)/

Logging

11. Mask Sensitive Information in Logs

Ensure secrets are never printed to logs. Azure Pipelines automatically masks variables marked as secret.

❌ BAD: Printing a secret directly.

steps:
- script: echo "My secret key is $(MyApiKey)" # Key will be visible

✅ GOOD: Access secret variables, which are automatically masked.

steps:
- script: echo "Attempting to use API Key..."
- script: az login --service-principal -u $(ServicePrincipalId) -p $(ServicePrincipalKey) --tenant $(TenantId) # ServicePrincipalKey is a secret variable

Testing Approaches

12. Integrate Comprehensive Testing Early

Run unit tests and static analysis (SAST) as early as possible in the CI stage. Dedicate separate stages for integration and end-to-end tests.

❌ BAD: Only running manual tests or basic compilation.

✅ GOOD: Multi-layered testing strategy.

stages:
- stage: Build
  jobs:
  - job: BuildAndScan
    steps:
    - script: dotnet build
    - task: SonarQubePrepare@5 # SAST integration
    - task: SonarQubeAnalyze@5
    - task: SonarQubePublish@5

- stage: Test
  dependsOn: Build
  jobs:
  - job: UnitAndIntegrationTests
    steps:
    - script: dotnet test --filter "Category=Unit"
    - script: dotnet test --filter "Category=Integration"
    - task: PublishTestResults@2
      inputs:
        testResultsFormat: 'VSTest'
        testResultsFiles: '**/*.trx'

- stage: E2ETest
  dependsOn: Test
  condition: succeeded('Test')
  jobs:
  - deployment: DeployToTestEnv
    environment: 'Test'
    strategy:
      runOnce:
        deploy:
          steps:
          - script: npm run e2e-tests # Run Cypress/Selenium tests