Azure Pipelines Introduction

What are Azure Pipelines?

Azure Pipelines is a cloud-based continuous integration and continuous delivery (CI/CD) service in Azure DevOps that automates building, testing, and deploying applications. It supports multiple programming languages, platforms, and cloud providers, enabling teams to deliver software faster and more reliably.

Azure Pipelines provides both classic (visual) and YAML-based pipeline definitions, offering flexibility in how automation workflows are created and maintained.

Core Concepts

Pipeline

An automated workflow that builds, tests, and deploys code. Pipelines are triggered by events like code commits, pull requests, or schedules.

Agent

A compute resource (virtual machine) that executes pipeline jobs. Azure provides Microsoft-hosted agents (managed) and supports self-hosted agents (custom infrastructure).

Stage

A logical boundary in a pipeline representing a phase of the CI/CD process, such as build, test, or deploy. Stages run sequentially by default.

Job

A collection of steps executed on a single agent. Jobs within a stage can run in parallel or sequentially.

Step

An individual task within a job, such as running a script, invoking a build tool, or deploying an artifact.

Artifact

Build outputs (compiled binaries, packages, or files) that are published during the pipeline and consumed by later stages.

Pipeline Architecture

graph TD
    A["Source Code
Commit"] -->B["Pipeline Trigger"]
    B -->C["Stage 1: Build"]
    C -->D["Job: Compile"]
    D -->E["Step 1: Restore packages"]
    D -->F["Step 2: Build code"]
    D -->G["Step 3: Run tests"]
    C -->H["Publish Artifacts"]
    H -->I["Stage 2: Deploy"]
    I -->J["Job: Deploy to staging"]
    I -->K["Job: Deploy to production"]
    
    style A fill:#e1f5ff
    style C fill:#fff3e0
    style I fill:#e8f5e9

Creating a Pipeline

Via Azure DevOps UI

1. Navigate to Azure DevOps project
2. Select Pipelines > Create Pipeline
3. Choose repository source (Azure Repos, GitHub, Bitbucket)
4. Select pipeline template or start with empty pipeline
5. Configure triggers and variables
6. Save and run

Using YAML

# azure-pipelines.yml
trigger:
  branches:
    include:
      - main

pool:
  vmImage: 'ubuntu-latest'

steps:
  - task: UseDotNet@2
    inputs:
      version: '7.x'
  
  - script: dotnet build
    displayName: 'Build application'
  
  - script: dotnet test
    displayName: 'Run tests'

Agent Pools

Microsoft-Hosted Agents

Azure provides pre-configured agents with common tools installed.

pool:
  vmImage: 'ubuntu-latest'  # Ubuntu Linux
  # vmImage: 'windows-latest'  # Windows Server
  # vmImage: 'macOS-latest'    # macOS

Available images:

• ubuntu-latest, ubuntu-22.04, ubuntu-20.04
• windows-latest, windows-2022, windows-2019
• macOS-latest, macOS-12, macOS-11

Self-Hosted Agents

Custom agents installed on specific infrastructure for specialized requirements.

pool:
  name: 'MyAgentPool'

Use cases:

• Access to internal networks
• Specific hardware requirements
• Pre-installed proprietary tools
• Regulatory compliance requirements

Agent Selection Flow

sequenceDiagram
    participant Pipeline as Pipeline
    participant Pool as Agent Pool
    participant Agent as Available Agent
    participant Job as Job Execution
    
    Pipeline->>Pool: Request agent
    Pool->>Agent: Allocate agent
    Agent->>Job: Execute steps
    Job->>Job: Run tasks
    Job->>Agent: Complete
    Agent->>Pool: Return to pool

Pipeline Triggers

Continuous Integration (CI) Trigger

trigger:
  branches:
    include:
      - main
      - develop
    exclude:
      - feature/*
  paths:
    include:
      - src/**
    exclude:
      - docs/**

Pull Request (PR) Trigger

pr:
  branches:
    include:
      - main
  paths:
    exclude:
      - README.md

Scheduled Trigger

schedules:
  - cron: "0 2 * * *"
    displayName: 'Nightly build'
    branches:
      include:
        - main
    always: true

Variables

Pipeline Variables

variables:
  buildConfiguration: 'Release'
  dotnetVersion: '7.x'

steps:
  - script: dotnet build --configuration $(buildConfiguration)

Variable Groups

Shared variables across multiple pipelines, managed centrally.

variables:
  - group: 'production-variables'
  - name: localVariable
    value: 'local-value'

Runtime Variables

steps:
  - script: |
      echo "##vso[task.setvariable variable=myVar]Hello"
  
  - script: echo $(myVar)

Simple CI Pipeline Example

# Build and test a .NET application
trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
  - task: UseDotNet@2
    displayName: 'Install .NET SDK'
    inputs:
      version: '7.x'
  
  - task: DotNetCoreCLI@2
    displayName: 'Restore dependencies'
    inputs:
      command: 'restore'
      projects: '**/*.csproj'
  
  - task: DotNetCoreCLI@2
    displayName: 'Build project'
    inputs:
      command: 'build'
      projects: '**/*.csproj'
      arguments: '--configuration $(buildConfiguration)'
  
  - task: DotNetCoreCLI@2
    displayName: 'Run tests'
    inputs:
      command: 'test'
      projects: '**/*Tests.csproj'
      arguments: '--configuration $(buildConfiguration) --collect:"XPlat Code Coverage"'
  
  - task: PublishCodeCoverageResults@1
    displayName: 'Publish code coverage'
    inputs:
      codeCoverageTool: 'Cobertura'
      summaryFileLocation: '$(Agent.TempDirectory)/**/*.cobertura.xml'

Publishing Artifacts

steps:
  - task: DotNetCoreCLI@2
    displayName: 'Publish application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/*.csproj'
      arguments: '--configuration Release --output $(Build.ArtifactStagingDirectory)'
  
  - task: PublishBuildArtifacts@1
    displayName: 'Upload artifacts'
    inputs:
      pathToPublish: '$(Build.ArtifactStagingDirectory)'
      artifactName: 'drop'
      publishLocation: 'Container'

Predefined Variables

Azure Pipelines provides built-in variables for common values.

Task Types

Built-in Tasks

# Copy files
- task: CopyFiles@2
  inputs:
    SourceFolder: '$(Build.SourcesDirectory)'
    Contents: '**/*.dll'
    TargetFolder: '$(Build.ArtifactStagingDirectory)'

# Run script
- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      Write-Host "Running PowerShell script"
      Get-ChildItem

# Archive files
- task: ArchiveFiles@2
  inputs:
    rootFolderOrFile: '$(Build.ArtifactStagingDirectory)'
    includeRootFolder: false
    archiveType: 'zip'
    archiveFile: '$(Build.ArtifactStagingDirectory)/app.zip'

Script Tasks

# Bash script
- script: |
    echo "Running bash commands"
    npm install
    npm test
  displayName: 'Run bash script'

# PowerShell script
- powershell: |
    Write-Host "Running PowerShell"
    dotnet --version
  displayName: 'Run PowerShell'

Pipeline Execution Flow

sequenceDiagram
    participant Dev as Developer
    participant Repo as Repository
    participant Pipeline as Azure Pipeline
    participant Agent as Agent
    participant Artifact as Artifact Storage
    
    Dev->>Repo: Push commit
    Repo->>Pipeline: Trigger pipeline
    Pipeline->>Agent: Allocate agent
    Agent->>Agent: Checkout code
    Agent->>Agent: Execute steps
    Agent->>Artifact: Publish artifacts
    Artifact->>Pipeline: Store artifacts
    Pipeline->>Dev: Notification (success/failure)

Classic vs YAML Pipelines

Classic Pipelines (Visual Designer)

graph LR
    A["Visual Designer"] -->B["Drag & Drop Tasks"]
    B -->C["Configure Settings"]
    C -->D["Stored in Azure DevOps"]
    
    style A fill:#ffcccc
    style D fill:#ffcccc

Pros:

• Visual interface, easier for beginners
• No YAML knowledge required

Cons:

• Not stored in source control
• Harder to version and review
• Limited to Azure DevOps

YAML Pipelines

graph LR
    A["YAML File"] -->B["Stored in Repository"]
    B -->C["Version Controlled"]
    C -->D["Code Review Process"]
    
    style A fill:#ccffcc
    style D fill:#ccffcc

Pros:

• Version controlled with code
• Code review via pull requests
• Reusable and templated
• Portable across projects

Cons:

• Requires YAML syntax knowledge
• Steeper learning curve

Service Connections

Connect Azure Pipelines to external services.

Azure Resource Manager Connection

steps:
  - task: AzureWebApp@1
    inputs:
      azureSubscription: 'MyAzureConnection'
      appName: 'my-web-app'
      package: '$(Build.ArtifactStagingDirectory)/**/*.zip'

Docker Registry Connection

steps:
  - task: Docker@2
    inputs:
      containerRegistry: 'MyDockerRegistry'
      repository: 'myapp'
      command: 'buildAndPush'
      Dockerfile: '**/Dockerfile'

Conditions and Dependencies

jobs:
  - job: Build
    steps:
      - script: echo "Building"
  
  - job: Test
    dependsOn: Build
    condition: succeeded()
    steps:
      - script: echo "Testing"
  
  - job: Deploy
    dependsOn: Test
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    steps:
      - script: echo "Deploying"

Pipeline Status and Monitoring

Pipeline Runs View

Track pipeline execution history, success rates, and durations in Azure DevOps UI.

Build Badge

![Build Status](https://dev.azure.com/org/project/_apis/build/status/pipeline-name)

Notifications

Configure email, Slack, or Microsoft Teams notifications for pipeline events.

Real-World Example: Node.js Application

trigger:
  branches:
    include:
      - main
      - develop

pool:
  vmImage: 'ubuntu-latest'

variables:
  nodeVersion: '18.x'

stages:
  - stage: Build
    displayName: 'Build and Test'
    jobs:
      - job: BuildJob
        displayName: 'Build Node.js App'
        steps:
          - task: NodeTool@0
            displayName: 'Install Node.js'
            inputs:
              versionSpec: '$(nodeVersion)'
          
          - script: npm ci
            displayName: 'Install dependencies'
          
          - script: npm run lint
            displayName: 'Run linter'
          
          - script: npm test
            displayName: 'Run tests'
          
          - script: npm run build
            displayName: 'Build application'
          
          - task: CopyFiles@2
            displayName: 'Copy build files'
            inputs:
              SourceFolder: '$(System.DefaultWorkingDirectory)/dist'
              Contents: '**'
              TargetFolder: '$(Build.ArtifactStagingDirectory)'
          
          - task: PublishBuildArtifacts@1
            displayName: 'Publish artifacts'
            inputs:
              pathToPublish: '$(Build.ArtifactStagingDirectory)'
              artifactName: 'webapp'

Key Takeaways

• Azure Pipelines automates CI/CD workflows for multiple platforms and languages
• Pipelines consist of stages, jobs, and steps executed on agents
• Microsoft-hosted agents provide pre-configured environments for common scenarios
• Self-hosted agents support specialized requirements and internal network access
• YAML pipelines are stored in source control, enabling version tracking and code review
• Triggers include CI, pull requests, and schedules
• Variables enable configuration reuse across pipeline definitions
• Artifacts preserve build outputs and pass files between stages
• Service connections authenticate with external services like Azure and Docker registries
• Conditions control when jobs execute based on previous results and branch context

Next Steps: Create a basic YAML pipeline for an existing project, configure triggers, and publish build artifacts for deployment.