Pipeline Best Practices

What Makes a Good Pipeline?

A well-designed CI/CD pipeline is fast, reliable, maintainable, and provides clear feedback. Following established best practices ensures pipelines support development velocity while maintaining code quality and security standards.

Effective pipelines balance speed with thoroughness, catching issues early while minimizing friction in the development workflow.

Core Pipeline Principles

Fast Feedback

Provide quick feedback on code changes to enable rapid iteration.

# Optimize for fast feedback
stages:
  - stage: QuickChecks
    jobs:
      - job: Lint
        steps:
          - script: npm run lint
      
      - job: UnitTests
        steps:
          - script: npm test -- --watch=false
  
  - stage: SlowerChecks
    dependsOn: QuickChecks
    jobs:
      - job: IntegrationTests
        steps:
          - script: npm run test:integration

Why it matters: Developers context-switch frequently. Fast feedback reduces waiting time and maintains focus.

Fail Fast

Stop pipelines early when critical issues are detected.

stages:
  - stage: Validate
    jobs:
      - job: Lint
        steps:
          - script: npm run lint
      
      - job: CompileCheck
        steps:
          - script: npm run build
  
  # Only run if validation passes
  - stage: Test
    dependsOn: Validate
    condition: succeeded()
    jobs:
      - job: Tests
        steps:
          - script: npm test

Build Once, Deploy Many

Build artifacts once and deploy the same artifact to all environments.

stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - script: docker build -t myapp:$(Build.BuildId) .
          - script: docker push myapp:$(Build.BuildId)
  
  - stage: DeployDev
    jobs:
      - job: Deploy
        steps:
          - script: docker pull myapp:$(Build.BuildId)
          - script: docker run myapp:$(Build.BuildId)
  
  - stage: DeployProd
    jobs:
      - job: Deploy
        steps:
          - script: docker pull myapp:$(Build.BuildId)
          - script: docker run myapp:$(Build.BuildId)

Why it matters: Ensures consistency across environments and eliminates "works in staging but not production" issues.

Pipeline Architecture Pattern

graph TD
    A["Code Commit"] -->B["Quick Validation
Lint, Format, Compile"]
    B -->|Pass| C["Build Artifacts
Once"]
    C -->D["Unit Tests
Fast"]
    D -->E["Integration Tests
Slower"]
    E -->F["Security Scan"]
    F -->G["Deploy Dev"]
    G -->H["Deploy Staging"]
    H -->I["Deploy Production"]
    
    B -->|Fail| Z["Stop Pipeline"]
    
    style B fill:#e1f5ff
    style C fill:#fff3e0
    style G fill:#e8f5e9
    style H fill:#e8f5e9
    style I fill:#ccffcc

Performance Optimization

Parallel Execution

Run independent jobs in parallel.

jobs:
  # These run in parallel
  - job: UnitTests
    steps:
      - script: npm run test:unit
  
  - job: Lint
    steps:
      - script: npm run lint
  
  - job: SecurityScan
    steps:
      - script: npm audit
  
  # This waits for all above
  - job: IntegrationTests
    dependsOn:
      - UnitTests
      - Lint
      - SecurityScan
    steps:
      - script: npm run test:integration

Caching Dependencies

Cache package installations to reduce build time.

# GitHub Actions
steps:
  - uses: actions/cache@v4
    with:
      path: ~/.npm
      key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
  
  - run: npm ci

# Azure Pipelines
steps:
  - task: Cache@2
    inputs:
      key: 'npm | "$(Agent.OS)" | package-lock.json'
      path: $(npm_config_cache)
  
  - script: npm ci

Incremental Builds

Build only changed components in monorepos.

# Example with Nx
steps:
  - script: npx nx affected:build --base=origin/main
  - script: npx nx affected:test --base=origin/main

Code Quality Gates

Automated Quality Checks

stages:
  - stage: QualityGate
    jobs:
      - job: CodeQuality
        steps:
          - script: npm run lint
            displayName: 'Linting'
          
          - script: npm run format:check
            displayName: 'Format check'
          
          - script: npm test -- --coverage
            displayName: 'Unit tests with coverage'
          
          - script: |
              # Fail if coverage below 80%
              COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
              if (( $(echo "$COVERAGE < 80" | bc -l) )); then
                echo "Coverage $COVERAGE% is below 80%"
                exit 1
              fi
            displayName: 'Coverage threshold check'

Branch Protection

Require pipeline success before merging:

# GitHub branch protection rules
# Settings > Branches > Branch protection rules
# - Require status checks to pass before merging
# - Require branches to be up to date before merging

# Azure DevOps branch policies
# Repos > Branches > Branch policies
# - Require a build to succeed
# - Minimum number of reviewers

Security Best Practices

Secret Management

# BAD - Never hardcode secrets
steps:
  - script: |
      export API_KEY="hardcoded-secret"
      curl -H "Authorization: Bearer hardcoded-secret"

# GOOD - Use secret management
steps:
  - script: |
      curl -H "Authorization: Bearer ${{ secrets.API_KEY }}"
    env:
      API_KEY: ${{ secrets.API_KEY }}

Container Image Scanning

steps:
  - script: docker build -t myapp:latest .
  
  - script: |
      # Scan image for vulnerabilities
      trivy image --severity HIGH,CRITICAL myapp:latest
    displayName: 'Security scan'
  
  - script: docker push myapp:latest
    condition: succeeded()

Dependency Scanning

steps:
  - script: npm audit --audit-level=high
    displayName: 'Audit dependencies'
  
  - script: |
      # Check for outdated dependencies
      npx npm-check-updates --errorLevel 2
    displayName: 'Check for outdated packages'

Error Handling and Resilience

Retry Logic

# GitHub Actions
steps:
  - name: Deploy with retry
    uses: nick-invision/retry@v2
    with:
      timeout_minutes: 10
      max_attempts: 3
      command: npm run deploy

# Azure Pipelines
steps:
  - powershell: |
      $retries = 3
      $success = $false
      for ($i = 0; $i -lt $retries; $i++) {
        try {
          npm run deploy
          $success = $true
          break
        } catch {
          Write-Host "Attempt $($i+1) failed"
          Start-Sleep -Seconds 10
        }
      }
      if (-not $success) {
        throw "Deployment failed after $retries attempts"
      }

Conditional Execution

steps:
  - script: npm test
    displayName: 'Run tests'
  
  - script: echo "Tests passed"
    condition: succeeded()
  
  - script: echo "Tests failed, sending notification"
    condition: failed()
  
  - script: echo "Always runs cleanup"
    condition: always()

Timeout Configuration

jobs:
  - job: Build
    timeoutInMinutes: 30
    steps:
      - script: npm run build
        timeoutInMinutes: 10

Maintainability

Pipeline as Code

Store pipeline definitions in version control:

project/
  .github/
    workflows/
      ci.yml
      deploy.yml
  .azure/
    pipelines/
      ci-pipeline.yml
      deploy-pipeline.yml

Template and Reusability

# templates/node-build.yml
parameters:
  - name: nodeVersion
    type: string
    default: '18'

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: ${{ parameters.nodeVersion }}
  
  - script: npm ci
  - script: npm run build
  - script: npm test

# Main pipeline
jobs:
  - job: Build
    steps:
      - template: templates/node-build.yml
        parameters:
          nodeVersion: '20'

Clear Naming and Documentation

# GOOD - Clear names
stages:
  - stage: BuildAndTest
    displayName: 'Build Application and Run Tests'
    jobs:
      - job: CompileCode
        displayName: 'Compile TypeScript Code'
        steps:
          - script: npm run build
            displayName: 'Run TypeScript compiler'

# AVOID - Vague names
stages:
  - stage: Stage1
    jobs:
      - job: Job1
        steps:
          - script: npm run build

Testing Strategy

Test Pyramid

graph TD
    A["E2E Tests
Slow, Expensive"] -->B["Integration Tests
Medium Speed"]
    B -->C["Unit Tests
Fast, Cheap"]
    
    style A fill:#ffcccc
    style B fill:#fff3e0
    style C fill:#ccffcc

Pipeline implementation:

stages:
  - stage: FastTests
    jobs:
      - job: UnitTests
        steps:
          - script: npm run test:unit
            displayName: 'Unit tests (70% of tests)'
  
  - stage: MediumTests
    dependsOn: FastTests
    jobs:
      - job: IntegrationTests
        steps:
          - script: npm run test:integration
            displayName: 'Integration tests (20% of tests)'
  
  - stage: SlowTests
    dependsOn: MediumTests
    condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
    jobs:
      - job: E2ETests
        steps:
          - script: npm run test:e2e
            displayName: 'E2E tests (10% of tests)'

Smoke Tests

stages:
  - stage: Deploy
    jobs:
      - deployment: DeployApp
        steps:
          - script: kubectl apply -f deployment.yaml
  
  - stage: SmokeTests
    dependsOn: Deploy
    jobs:
      - job: Smoke
        steps:
          - script: |
              # Wait for deployment
              sleep 30
              
              # Health check
              curl -f https://myapp.com/health || exit 1
              
              # Basic functionality
              curl -f https://myapp.com/api/status || exit 1
            displayName: 'Smoke tests'

Notifications and Monitoring

Pipeline Notifications

# GitHub Actions
steps:
  - name: Notify Slack on failure
    if: failure()
    uses: slackapi/slack-github-action@v1
    with:
      payload: |
        {
          "text": "Build failed: ${{ github.repository }}"
        }
    env:
      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

# Azure Pipelines - Configure in project settings
# Project Settings > Notifications > New subscription
# Event: Build completed
# Filter: Build status = Failed

Pipeline Metrics

Track key metrics:

• Build success rate
• Average build duration
• Deployment frequency
• Mean time to recovery (MTTR)

steps:
  - script: |
      # Send metrics to monitoring system
      curl -X POST https://metrics.example.com/pipeline \
        -d "build_duration=$(($SECONDS))" \
        -d "build_status=${{ job.status }}" \
        -d "build_id=${{ github.run_id }}"

Environment-Specific Configuration

Configuration Management

# Use environment variables
stages:
  - stage: DeployDev
    variables:
      ENVIRONMENT: 'development'
      API_URL: 'https://api-dev.example.com'
      DATABASE_NAME: 'myapp_dev'
    jobs:
      - job: Deploy
        steps:
          - script: |
              echo "Deploying to $ENVIRONMENT"
              echo "API_URL=$API_URL" > .env
              echo "DATABASE_NAME=$DATABASE_NAME" >> .env

  - stage: DeployProd
    variables:
      ENVIRONMENT: 'production'
      API_URL: 'https://api.example.com'
      DATABASE_NAME: 'myapp_prod'
    jobs:
      - job: Deploy
        steps:
          - script: |
              echo "Deploying to $ENVIRONMENT"
              echo "API_URL=$API_URL" > .env
              echo "DATABASE_NAME=$DATABASE_NAME" >> .env

Documentation

Pipeline Documentation

# Add comments to complex logic
steps:
  # Calculate semantic version based on commit messages
  # Format: major.minor.patch (e.g., 1.2.3)
  - script: |
      VERSION=$(conventional-changelog-cli --preset angular --output-unreleased)
      echo "##vso[task.setvariable variable=version]$VERSION"
    displayName: 'Calculate semantic version'
  
  # Tag repository with new version
  - script: |
      git tag v$(version)
      git push origin v$(version)
    displayName: 'Create and push git tag'

README for Pipelines

# CI/CD Pipeline Documentation

## Overview
This pipeline builds, tests, and deploys the application.

## Triggers
- Push to `main` or `develop` branches
- Pull requests to `main`
- Nightly scheduled builds at 2 AM UTC

## Stages
1. Build - Compile and package application
2. Test - Run unit and integration tests
3. Deploy Dev - Auto-deploy to development
4. Deploy Staging - Deploy to staging (main branch only)
5. Deploy Production - Deploy to production (manual approval required)

## Variables
- `buildConfiguration`: Release/Debug build mode
- `nodeVersion`: Node.js version to use

## Secrets
- `AZURE_CREDENTIALS`: Azure service principal
- `DATABASE_PASSWORD`: Production database password

Common Pitfalls to Avoid

Don't Mix Configuration with Code

# BAD
steps:
  - script: |
      if [ "$BRANCH" == "main" ]; then
        export API_URL="https://api.example.com"
      else
        export API_URL="https://api-dev.example.com"
      fi

# GOOD
variables:
  - ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
    - name: API_URL
      value: 'https://api.example.com'
  - ${{ else }}:
    - name: API_URL
      value: 'https://api-dev.example.com'

Don't Ignore Failed Tests

# BAD
steps:
  - script: npm test || true

# GOOD
steps:
  - script: npm test
  - script: echo "Cleanup even if tests fail"
    condition: always()

Don't Skip Security Scans

# Always include security checks
stages:
  - stage: Security
    jobs:
      - job: Scan
        steps:
          - script: npm audit
          - script: trivy image myapp:latest

Real-World Example: Complete Pipeline

name: $(Date:yyyyMMdd)$(Rev:.r)

trigger:
  branches:
    include:
      - main
      - develop

variables:
  - group: shared-variables
  - name: buildConfiguration
    value: 'Release'
  - name: nodeVersion
    value: '18.x'

stages:
  - stage: Validate
    displayName: 'Quick Validation'
    jobs:
      - job: Lint
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: $(nodeVersion)
          
          - script: npm ci
            displayName: 'Install dependencies'
          
          - script: npm run lint
            displayName: 'Run linter'
          
          - script: npm run format:check
            displayName: 'Check formatting'

  - stage: Build
    displayName: 'Build Application'
    dependsOn: Validate
    jobs:
      - job: BuildJob
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - template: templates/node-setup.yml
          
          - script: npm run build
            displayName: 'Build application'
          
          - task: CopyFiles@2
            inputs:
              SourceFolder: 'dist'
              Contents: '**'
              TargetFolder: '$(Build.ArtifactStagingDirectory)'
          
          - task: PublishBuildArtifacts@1
            inputs:
              pathToPublish: '$(Build.ArtifactStagingDirectory)'
              artifactName: 'webapp'

  - stage: Test
    displayName: 'Run Tests'
    dependsOn: Build
    jobs:
      - job: UnitTests
        steps:
          - template: templates/node-setup.yml
          - script: npm run test:unit -- --coverage
            displayName: 'Unit tests'
          - task: PublishCodeCoverageResults@1
            inputs:
              codeCoverageTool: 'Cobertura'
              summaryFileLocation: 'coverage/cobertura-coverage.xml'
      
      - job: IntegrationTests
        steps:
          - template: templates/node-setup.yml
          - script: npm run test:integration
            displayName: 'Integration tests'

  - stage: Security
    displayName: 'Security Scanning'
    dependsOn: Build
    jobs:
      - job: SecurityScan
        steps:
          - script: npm audit --audit-level=moderate
            displayName: 'Audit dependencies'
          
          - script: trivy fs --severity HIGH,CRITICAL .
            displayName: 'Trivy scan'

  - stage: DeployDev
    displayName: 'Deploy to Development'
    dependsOn:
      - Test
      - Security
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/develop'))
    jobs:
      - deployment: DeployDev
        environment: development
        strategy:
          runOnce:
            deploy:
              steps:
                - template: templates/deploy-steps.yml
                  parameters:
                    environment: 'dev'

  - stage: DeployProd
    displayName: 'Deploy to Production'
    dependsOn:
      - Test
      - Security
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeployProd
        environment: production
        strategy:
          runOnce:
            deploy:
              steps:
                - template: templates/deploy-steps.yml
                  parameters:
                    environment: 'prod'
            
            on:
              failure:
                steps:
                  - script: echo "Sending failure notification"
              success:
                steps:
                  - script: echo "Deployment successful"

Key Takeaways

• Optimize for fast feedback by running quick checks first
• Build artifacts once and deploy the same artifact to all environments
• Run independent jobs in parallel to reduce pipeline duration
• Cache dependencies to speed up builds significantly
• Implement quality gates with code coverage thresholds
• Use secret management instead of hardcoding credentials
• Add retry logic for flaky external dependencies
• Store pipeline definitions in version control for collaboration
• Use templates to promote reusability and consistency
• Follow the test pyramid with more unit tests than integration tests
• Include smoke tests after deployments for quick validation
• Configure notifications for pipeline failures
• Document complex pipeline logic with comments and README files
• Avoid mixing configuration with code using variables
• Always include security scanning in pipelines

Next Steps: Review existing pipelines against these best practices, implement caching and parallelization, add quality gates, and create reusable templates for common patterns.