YAML Pipelines in Azure DevOps

What are YAML Pipelines?

YAML pipelines define CI/CD workflows as code using YAML syntax, stored directly in the repository alongside application code. This infrastructure-as-code approach enables version control, code review, and collaboration on pipeline definitions.

YAML pipelines provide flexibility, reusability, and portability that classic visual pipelines cannot match.

Why Use YAML Pipelines?

• Version Control - Pipeline definitions tracked in Git with code
• Code Review - Changes reviewed via pull requests
• Reusability - Templates and extends enable DRY principles
• Portability - Move pipelines between projects easily
• Branching - Different pipeline behavior per branch
• Automation - Generate or modify pipelines programmatically

YAML Pipeline Structure

# Triggers
trigger:
  branches:
    include:
      - main

# Variables
variables:
  buildConfiguration: 'Release'

# Stages
stages:
  - stage: Build
    jobs:
      - job: BuildJob
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - script: echo "Building..."

Pipeline Schema Hierarchy

graph TD
    A["Pipeline"] -->B["Trigger"]
    A -->C["Variables"]
    A -->D["Stages"]
    D -->E["Stage 1"]
    D -->F["Stage 2"]
    E -->G["Job 1"]
    E -->H["Job 2"]
    G -->I["Step 1"]
    G -->J["Step 2"]
    G -->K["Step 3"]
    
    style A fill:#e1f5ff
    style D fill:#fff3e0
    style E fill:#e8f5e9
    style G fill:#fffacd

Basic YAML Pipeline

# azure-pipelines.yml
name: $(Date:yyyyMMdd)$(Rev:.r)

trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

variables:
  NODE_VERSION: '18.x'

steps:
  - task: NodeTool@0
    displayName: 'Install Node.js'
    inputs:
      versionSpec: $(NODE_VERSION)
  
  - script: |
      npm ci
      npm run build
      npm test
    displayName: 'Build and test'
  
  - task: PublishTestResults@2
    displayName: 'Publish test results'
    condition: succeededOrFailed()
    inputs:
      testResultsFiles: '**/test-results.xml'
      testRunTitle: 'Node.js Tests'

Stages, Jobs, and Steps

Multi-Stage Pipeline

stages:
  - stage: Build
    displayName: 'Build Application'
    jobs:
      - job: BuildJob
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - script: echo "Building"
          - script: dotnet build
  
  - stage: Test
    displayName: 'Run Tests'
    dependsOn: Build
    jobs:
      - job: UnitTests
        steps:
          - script: dotnet test
      
      - job: IntegrationTests
        steps:
          - script: npm run test:integration
  
  - stage: Deploy
    displayName: 'Deploy to Production'
    dependsOn: Test
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeployWeb
        environment: production
        strategy:
          runOnce:
            deploy:
              steps:
                - script: echo "Deploying"

Stage Dependencies and Conditions

graph LR
    A["Build Stage"] -->B["Test Stage"]
    B -->C["Deploy Staging"]
    B -->D["Deploy Production"]
    C -->E{Manual
Approval}
    E -->F["Smoke Tests"]
    
    style A fill:#e1f5ff
    style B fill:#fff3e0
    style C fill:#e8f5e9
    style D fill:#e8f5e9

Conditional Stages

stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - script: echo "Building"
  
  - stage: DeployDev
    condition: eq(variables['Build.SourceBranch'], 'refs/heads/develop')
    dependsOn: Build
    jobs:
      - job: Deploy
        steps:
          - script: echo "Deploy to dev"
  
  - stage: DeployProd
    condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
    dependsOn: Build
    jobs:
      - job: Deploy
        steps:
          - script: echo "Deploy to production"

Variables in YAML

Pipeline Variables

variables:
  # Simple variables
  buildConfiguration: 'Release'
  dotnetVersion: '7.x'
  
  # Computed variables
  buildNumber: $[counter(format('{0:yyyyMMdd}', pipeline.startTime), 1)]

steps:
  - script: echo "Build config is $(buildConfiguration)"

Variable Groups

variables:
  - group: 'shared-variables'
  - group: 'production-secrets'
  - name: localVar
    value: 'local-value'

steps:
  - script: echo "Using variable group values"

Runtime Variables

steps:
  - script: |
      echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]Hello"
    name: SetVar
  
  - script: echo "Value is $(SetVar.myOutputVar)"

Variable Templates

# variables-template.yml
variables:
  buildConfiguration: 'Release'
  vmImage: 'ubuntu-latest'

# azure-pipelines.yml
variables:
  - template: variables-template.yml

pool:
  vmImage: $(vmImage)

Templates

Templates enable reusability and consistency across pipelines.

Step Template

# templates/build-steps.yml
parameters:
  - name: buildConfiguration
    type: string
    default: 'Release'

steps:
  - task: DotNetCoreCLI@2
    displayName: 'Restore packages'
    inputs:
      command: 'restore'
  
  - task: DotNetCoreCLI@2
    displayName: 'Build project'
    inputs:
      command: 'build'
      arguments: '--configuration ${{ parameters.buildConfiguration }}'
  
  - task: DotNetCoreCLI@2
    displayName: 'Run tests'
    inputs:
      command: 'test'

# Main pipeline
steps:
  - template: templates/build-steps.yml
    parameters:
      buildConfiguration: 'Release'

Job Template

# templates/test-job.yml
parameters:
  - name: testType
    type: string
  - name: vmImage
    type: string
    default: 'ubuntu-latest'

jobs:
  - job: ${{ parameters.testType }}
    pool:
      vmImage: ${{ parameters.vmImage }}
    steps:
      - script: echo "Running ${{ parameters.testType }}"
      - script: npm test

# Main pipeline
jobs:
  - template: templates/test-job.yml
    parameters:
      testType: 'UnitTests'
  
  - template: templates/test-job.yml
    parameters:
      testType: 'IntegrationTests'
      vmImage: 'windows-latest'

Stage Template

# templates/deploy-stage.yml
parameters:
  - name: environment
    type: string
  - name: serviceConnection
    type: string

stages:
  - stage: Deploy_${{ parameters.environment }}
    jobs:
      - deployment: Deploy
        environment: ${{ parameters.environment }}
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: ${{ parameters.serviceConnection }}

# Main pipeline
stages:
  - template: templates/deploy-stage.yml
    parameters:
      environment: 'staging'
      serviceConnection: 'AzureStaging'
  
  - template: templates/deploy-stage.yml
    parameters:
      environment: 'production'
      serviceConnection: 'AzureProduction'

Extends Templates

Extends templates provide pipeline structure inheritance.

Base Template

# templates/base-pipeline.yml
parameters:
  - name: stages
    type: stageList

stages:
  - stage: Security
    jobs:
      - job: SecurityScan
        steps:
          - script: echo "Running security scan"
  
  - ${{ each stage in parameters.stages }}:
    - ${{ stage }}
  
  - stage: Compliance
    jobs:
      - job: ComplianceCheck
        steps:
          - script: echo "Running compliance check"

Extending Base Template

# azure-pipelines.yml
extends:
  template: templates/base-pipeline.yml
  parameters:
    stages:
      - stage: Build
        jobs:
          - job: BuildJob
            steps:
              - script: dotnet build
      
      - stage: Deploy
        jobs:
          - job: DeployJob
            steps:
              - script: echo "Deploying"

Result: Security stage runs first, then Build and Deploy, then Compliance stage.

Template Flow

graph TD
    A["Main Pipeline"] -->B["Extends Base Template"]
    B -->C["Security Stage
(from base)"]
    C -->D["Build Stage
(from main)"]
    D -->E["Deploy Stage
(from main)"]
    E -->F["Compliance Stage
(from base)"]
    
    style A fill:#e1f5ff
    style B fill:#fff3e0
    style C fill:#ffcccc
    style D fill:#ccffcc
    style E fill:#ccffcc
    style F fill:#ffcccc

Conditions and Expressions

Built-in Conditions

steps:
  - script: echo "Always runs"
    condition: always()
  
  - script: echo "Runs if previous succeeded"
    condition: succeeded()
  
  - script: echo "Runs if previous failed"
    condition: failed()
  
  - script: echo "Runs if previous succeeded or failed"
    condition: succeededOrFailed()

Custom Conditions

steps:
  - script: echo "Production deployment"
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  
  - script: echo "PR validation"
    condition: eq(variables['Build.Reason'], 'PullRequest')
  
  - script: echo "Manual or scheduled"
    condition: or(eq(variables['Build.Reason'], 'Manual'), eq(variables['Build.Reason'], 'Schedule'))

Variable-Based Conditions

variables:
  deployToProduction: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]

stages:
  - stage: Deploy
    condition: eq(variables.deployToProduction, true)
    jobs:
      - job: DeployJob
        steps:
          - script: echo "Deploying"

Container Jobs

Run jobs inside Docker containers for consistent environments.

resources:
  containers:
    - container: node
      image: node:18
    - container: python
      image: python:3.11

jobs:
  - job: NodeJob
    container: node
    steps:
      - script: |
          node --version
          npm --version
  
  - job: PythonJob
    container: python
    steps:
      - script: |
          python --version
          pip --version

Service Containers

Run databases or services alongside jobs.

resources:
  containers:
    - container: postgres
      image: postgres:15
      env:
        POSTGRES_PASSWORD: postgres
        POSTGRES_DB: testdb
      ports:
        - 5432:5432

jobs:
  - job: IntegrationTests
    services:
      postgres: postgres
    
    steps:
      - script: |
          echo "Postgres available at localhost:5432"
          npm run test:integration
        env:
          DATABASE_URL: postgres://postgres:postgres@localhost:5432/testdb

Matrix Strategy

Test across multiple configurations.

strategy:
  matrix:
    Linux_Node16:
      vmImage: 'ubuntu-latest'
      nodeVersion: '16.x'
    Linux_Node18:
      vmImage: 'ubuntu-latest'
      nodeVersion: '18.x'
    Windows_Node18:
      vmImage: 'windows-latest'
      nodeVersion: '18.x'
    Mac_Node18:
      vmImage: 'macOS-latest'
      nodeVersion: '18.x'

pool:
  vmImage: $(vmImage)

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: $(nodeVersion)
  
  - script: npm test

Parallel and Sequential Execution

Parallel Jobs

jobs:
  - job: Job1
    steps:
      - script: echo "Job 1"
  
  - job: Job2
    steps:
      - script: echo "Job 2"
  
  # Both run in parallel by default

Sequential Jobs

jobs:
  - job: Job1
    steps:
      - script: echo "Job 1"
  
  - job: Job2
    dependsOn: Job1
    steps:
      - script: echo "Job 2"
  
  - job: Job3
    dependsOn:
      - Job1
      - Job2
    steps:
      - script: echo "Job 3"

Deployment Jobs

Deployment jobs track deployments to environments with history and approvals.

stages:
  - stage: Deploy
    jobs:
      - deployment: DeployWeb
        displayName: 'Deploy Web App'
        environment: production
        pool:
          vmImage: 'ubuntu-latest'
        strategy:
          runOnce:
            deploy:
              steps:
                - download: current
                  artifact: webapp
                
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: 'AzureConnection'
                    appName: 'my-web-app'
                    package: '$(Pipeline.Workspace)/webapp/**/*.zip'

Deployment Strategies

# Rolling deployment
strategy:
  rolling:
    maxParallel: 2
    deploy:
      steps:
        - script: echo "Deploying"

# Canary deployment
strategy:
  canary:
    increments: [10, 20, 50, 100]
    deploy:
      steps:
        - script: echo "Deploying to canary"

Real-World Multi-Environment Pipeline

trigger:
  branches:
    include:
      - main
      - develop

variables:
  - group: 'shared-variables'
  - name: buildConfiguration
    value: 'Release'

stages:
  - stage: Build
    displayName: 'Build Application'
    jobs:
      - job: BuildJob
        pool:
          vmImage: 'ubuntu-latest'
        
        steps:
          - task: UseDotNet@2
            inputs:
              version: '7.x'
          
          - task: DotNetCoreCLI@2
            displayName: 'Restore dependencies'
            inputs:
              command: 'restore'
          
          - task: DotNetCoreCLI@2
            displayName: 'Build'
            inputs:
              command: 'build'
              arguments: '--configuration $(buildConfiguration)'
          
          - task: DotNetCoreCLI@2
            displayName: 'Test'
            inputs:
              command: 'test'
              arguments: '--configuration $(buildConfiguration) --collect:"XPlat Code Coverage"'
          
          - task: DotNetCoreCLI@2
            displayName: 'Publish'
            inputs:
              command: 'publish'
              publishWebProjects: true
              arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'
          
          - task: PublishBuildArtifacts@1
            inputs:
              pathToPublish: '$(Build.ArtifactStagingDirectory)'
              artifactName: 'drop'

  - stage: DeployDev
    displayName: 'Deploy to Development'
    dependsOn: Build
    condition: eq(variables['Build.SourceBranch'], 'refs/heads/develop')
    
    jobs:
      - deployment: DeployDev
        environment: development
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: 'AzureDev'
                    appName: 'myapp-dev'
                    package: '$(Pipeline.Workspace)/drop/**/*.zip'

  - stage: DeployStaging
    displayName: 'Deploy to Staging'
    dependsOn: Build
    condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
    
    jobs:
      - deployment: DeployStaging
        environment: staging
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: 'AzureStaging'
                    appName: 'myapp-staging'
                    package: '$(Pipeline.Workspace)/drop/**/*.zip'

  - stage: DeployProduction
    displayName: 'Deploy to Production'
    dependsOn: DeployStaging
    condition: succeeded()
    
    jobs:
      - deployment: DeployProduction
        environment: production
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: 'AzureProduction'
                    appName: 'myapp-prod'
                    package: '$(Pipeline.Workspace)/drop/**/*.zip'
                
                - task: PowerShell@2
                  displayName: 'Smoke test'
                  inputs:
                    targetType: 'inline'
                    script: |
                      $response = Invoke-WebRequest -Uri "https://myapp-prod.azurewebsites.net/health"
                      if ($response.StatusCode -ne 200) {
                        throw "Health check failed"
                      }

Best Practices

Keep Pipelines DRY

Use templates to avoid repetition:

# Bad - repetitive
jobs:
  - job: TestWindows
    pool:
      vmImage: 'windows-latest'
    steps:
      - script: npm install
      - script: npm test
  
  - job: TestLinux
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - script: npm install
      - script: npm test

# Good - template
jobs:
  - template: templates/test-job.yml
    parameters:
      vmImage: 'windows-latest'
  
  - template: templates/test-job.yml
    parameters:
      vmImage: 'ubuntu-latest'

Validate YAML Syntax

# Install Azure CLI
az pipelines runs validate --yaml-path azure-pipelines.yml

Use Meaningful Names

# Good
- stage: BuildAndTest
  displayName: 'Build Application and Run Tests'
  jobs:
    - job: CompileCode
      displayName: 'Compile Source Code'

# Avoid
- stage: Stage1
  jobs:
    - job: Job1

Key Takeaways

• YAML pipelines store CI/CD definitions as code in version control
• Pipeline structure includes stages, jobs, and steps in hierarchical organization
• Templates enable reusability through step, job, and stage templates
• Extends templates provide base pipeline structures with inherited behavior
• Variables support simple values, variable groups, and runtime computation
• Conditions control when stages, jobs, or steps execute based on expressions
• Container jobs provide consistent environments using Docker images
• Service containers run databases and dependencies for integration testing
• Matrix strategies test across multiple configurations simultaneously
• Deployment jobs track deployment history and integrate with environment approvals
• Templates and extends eliminate duplication and enforce standards

Next Steps: Convert existing classic pipelines to YAML, create reusable templates for common patterns, and implement multi-stage pipelines with environment approvals.