Testing Infrastructure as Code

Testing infrastructure code prevents costly production failures, validates configurations before deployment, and enables confident changes. Comprehensive testing strategies combine static analysis, unit tests, integration tests, and compliance validation.

Why Test Infrastructure Code

Infrastructure failures disrupt services and impact customers. Testing catches issues before production deployment.

Cost of Failures

Infrastructure mistakes have serious consequences:

# Dangerous change without testing
resource "aws_security_group" "api" {
  ingress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  # Opens all ports to internet!
  }
}

Without testing, this security group exposes all ports publicly, creating major security vulnerability.

Confidence in Changes

Testing enables safe refactoring and improvements:

# Before refactoring
resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  
  user_data = <<-EOF
    #!/bin/bash
    apt-get update
    apt-get install -y nginx
    systemctl start nginx
  EOF
}

# After refactoring with modules
module "web_server" {
  source = "./modules/web-server"
  
  ami_id        = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  install_nginx = true
}

Tests verify the refactored module behaves identically to the original configuration.

graph TD
    A[Write Infrastructure Code] --> B[Run Tests]
    B --> C{Tests Pass?}
    C -->|No| D[Fix Issues]
    D --> A
    C -->|Yes| E[Deploy to Staging]
    E --> F{Staging Validation}
    F -->|Issues Found| D
    F -->|Success| G[Deploy to Production]
    
    style C fill:#fff4e6
    style G fill:#d4f1d4

Static Analysis

Static analysis examines code without executing it, catching syntax errors, security issues, and best practice violations.

Syntax Validation

Verify code syntax correctness:

# Terraform syntax validation
terraform fmt -check
terraform validate

# Example output on success:
# Success! The configuration is valid.

# Example output on error:
# Error: Unsupported argument
#   on main.tf line 15:
#   instance_typ = "t3.medium"
# An argument named "instance_typ" is not expected here.
# Did you mean "instance_type"?

Syntax validation catches typos and structural errors immediately.

Security Scanning

Identify security vulnerabilities in configuration:

# tfsec - Terraform security scanner
tfsec infrastructure/

# Example findings:
# Result #1 CRITICAL Security group rule allows ingress from 0.0.0.0/0
#   infrastructure/security.tf:10-15
#   
#   10 | resource "aws_security_group_rule" "allow_all" {
#   11 |   type              = "ingress"
#   12 |   from_port         = 0
#   13 |   to_port           = 65535
#   14 |   cidr_blocks       = ["0.0.0.0/0"]
#   15 | }
#   
#   Impact: Unrestricted network access increases attack surface
#   Resolution: Set explicit CIDR blocks for source addresses

Security scanners detect common misconfigurations:

# Checkov - Multi-cloud security scanner
checkov -d infrastructure/

# Example output:
# Check: CKV_AWS_23: "Ensure security group has description"
# FAILED for resource: aws_security_group.api
# File: /security.tf:5-10
# Guide: https://docs.bridgecrew.io/docs/networking_1

Policy as Code

Enforce organizational standards:

# Sentinel policy (HashiCorp)
import "tfplan/v2" as tfplan

# Require all EC2 instances to have specific tags
required_tags = ["Environment", "Owner", "CostCenter"]

main = rule {
  all tfplan.resource_changes as _, rc {
    rc.type is "aws_instance" implies
      all required_tags as tag {
        rc.change.after.tags contains tag
      }
  }
}

Policy enforcement prevents non-compliant infrastructure:

# This would fail policy
resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  # Missing required tags!
}

# This passes policy
resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  
  tags = {
    Environment = "production"
    Owner       = "platform-team"
    CostCenter  = "engineering"
  }
}

Open Policy Agent (OPA) provides flexible policy language:

# OPA policy for Kubernetes
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Pod"
  not input.request.object.spec.containers[_].resources.limits.memory
  msg := "Containers must have memory limits"
}

deny[msg] {
  input.request.kind.kind == "Service"
  input.request.object.spec.type == "LoadBalancer"
  not input.request.object.metadata.annotations["service.beta.kubernetes.io/aws-load-balancer-internal"]
  msg := "LoadBalancer services must be internal"
}

Unit Testing

Unit tests validate individual components in isolation.

Testing Terraform Modules

Test modules with Terratest (Go):

// test/vpc_test.go
package test

import (
  "testing"
  "github.com/gruntwork-io/terratest/modules/terraform"
  "github.com/stretchr/testify/assert"
)

func TestVPCModule(t *testing.T) {
  terraformOptions := &terraform.Options{
    TerraformDir: "../modules/vpc",
    
    Vars: map[string]interface{}{
      "cidr_block": "10.0.0.0/16",
      "environment": "test",
    },
  }
  
  defer terraform.Destroy(t, terraformOptions)
  terraform.InitAndApply(t, terraformOptions)
  
  // Verify VPC CIDR
  vpcCidr := terraform.Output(t, terraformOptions, "vpc_cidr")
  assert.Equal(t, "10.0.0.0/16", vpcCidr)
  
  // Verify subnet count
  subnetIds := terraform.OutputList(t, terraformOptions, "subnet_ids")
  assert.Equal(t, 3, len(subnetIds))
}

Run tests:

cd test
go test -v -timeout 30m

Testing Kubernetes Manifests

Validate Kubernetes resources:

# test_deployment.py
import pytest
import yaml

def load_manifest(filename):
    with open(filename) as f:
        return list(yaml.safe_load_all(f))

def test_deployment_has_replicas():
    manifests = load_manifest("../k8s/deployment.yaml")
    deployment = [m for m in manifests if m['kind'] == 'Deployment'][0]
    
    assert deployment['spec']['replicas'] >= 3, \
        "Production deployments must have at least 3 replicas"

def test_deployment_has_resource_limits():
    manifests = load_manifest("../k8s/deployment.yaml")
    deployment = [m for m in manifests if m['kind'] == 'Deployment'][0]
    
    container = deployment['spec']['template']['spec']['containers'][0]
    assert 'resources' in container
    assert 'limits' in container['resources']
    assert 'cpu' in container['resources']['limits']
    assert 'memory' in container['resources']['limits']

def test_service_type_is_clusterip():
    manifests = load_manifest("../k8s/deployment.yaml")
    service = [m for m in manifests if m['kind'] == 'Service'][0]
    
    assert service['spec']['type'] == 'ClusterIP', \
        "Services should not be directly exposed"

Run Kubernetes tests:

pytest test_deployment.py -v

Helm Chart Testing

Test Helm charts with helm unittest:

# charts/api-service/tests/deployment_test.yaml
suite: test deployment
templates:
  - deployment.yaml
tests:
  - it: should set replica count from values
    set:
      replicaCount: 5
    asserts:
      - equal:
          path: spec.replicas
          value: 5
  
  - it: should set resource limits
    asserts:
      - exists:
          path: spec.template.spec.containers[0].resources.limits
      - exists:
          path: spec.template.spec.containers[0].resources.requests
  
  - it: should use correct image
    set:
      image:
        repository: myapp
        tag: v2.0.0
    asserts:
      - equal:
          path: spec.template.spec.containers[0].image
          value: myapp:v2.0.0

Run Helm tests:

helm unittest charts/api-service

Integration Testing

Integration tests validate infrastructure in real environments.

End-to-End Testing

Test complete infrastructure stacks:

// test/integration_test.go
package test

import (
  "testing"
  "time"
  "github.com/gruntwork-io/terratest/modules/terraform"
  "github.com/gruntwork-io/terratest/modules/http-helper"
  "github.com/stretchr/testify/assert"
)

func TestCompleteInfrastructure(t *testing.T) {
  terraformOptions := &terraform.Options{
    TerraformDir: "../infrastructure/staging",
  }
  
  defer terraform.Destroy(t, terraformOptions)
  terraform.InitAndApply(t, terraformOptions)
  
  // Get load balancer URL
  lbUrl := terraform.Output(t, terraformOptions, "load_balancer_url")
  
  // Test HTTP endpoint responds
  http_helper.HttpGetWithRetry(
    t,
    lbUrl,
    nil,
    200,
    "OK",
    30,
    10*time.Second,
  )
  
  // Test database is accessible
  dbEndpoint := terraform.Output(t, terraformOptions, "database_endpoint")
  assert.NotEmpty(t, dbEndpoint)
  
  // Could add database connection test here
}

Contract Testing

Verify infrastructure meets application requirements:

# infrastructure-contract.yaml
version: 1.0

requirements:
  networking:
    - name: VPC exists
      resource: aws_vpc.main
      assertions:
        - cidr_block_size >= 16
        - enable_dns_hostnames == true
    
    - name: Public subnets available
      resource: aws_subnet.public
      assertions:
        - count >= 2
        - map_public_ip_on_launch == true
  
  database:
    - name: Database encrypted
      resource: aws_db_instance.main
      assertions:
        - storage_encrypted == true
        - multi_az == true
    
    - name: Automated backups enabled
      resource: aws_db_instance.main
      assertions:
        - backup_retention_period >= 7
        - backup_window != null

Validate contracts:

# validate_contracts.py
import yaml
import json

def validate_terraform_output(contract_file, terraform_state):
    with open(contract_file) as f:
        contract = yaml.safe_load(f)
    
    with open(terraform_state) as f:
        state = json.load(f)
    
    failures = []
    
    for category, requirements in contract['requirements'].items():
        for req in requirements:
            resource_name = req['resource']
            # Find resource in state
            resource = find_resource(state, resource_name)
            
            for assertion in req['assertions']:
                if not evaluate_assertion(resource, assertion):
                    failures.append(f"{req['name']}: {assertion}")
    
    return failures

Compliance Testing

Verify infrastructure meets regulatory requirements.

Compliance Frameworks

Test against standards:

# InSpec - Compliance testing framework
inspec exec aws-compliance-profile \
  --target aws://us-east-1 \
  --reporter cli json:compliance-report.json

# Example InSpec test
describe aws_s3_bucket('production-data') do
  it { should exist }
  it { should have_default_encryption_enabled }
  it { should_not be_public }
  its('bucket_policy') { should include 'AES256' }
end

describe aws_rds_instance('production-database') do
  it { should be_encrypted }
  it { should have_multi_az_enabled }
  its('backup_retention_period') { should be >= 30 }
end

Custom Compliance Rules

Enforce organization-specific requirements:

# compliance_tests.py
import boto3
import pytest

@pytest.fixture
def aws_resources():
    return {
        's3': boto3.client('s3'),
        'rds': boto3.client('rds'),
        'ec2': boto3.client('ec2')
    }

def test_all_s3_buckets_encrypted(aws_resources):
    """All S3 buckets must have encryption enabled"""
    s3 = aws_resources['s3']
    buckets = s3.list_buckets()['Buckets']
    
    for bucket in buckets:
        bucket_name = bucket['Name']
        encryption = s3.get_bucket_encryption(Bucket=bucket_name)
        
        assert 'ServerSideEncryptionConfiguration' in encryption, \
            f"Bucket {bucket_name} lacks encryption"

def test_rds_automated_backups(aws_resources):
    """RDS instances must have 30-day backup retention"""
    rds = aws_resources['rds']
    instances = rds.describe_db_instances()['DBInstances']
    
    for instance in instances:
        retention = instance['BackupRetentionPeriod']
        assert retention >= 30, \
            f"Instance {instance['DBInstanceIdentifier']} has insufficient backup retention: {retention} days"

def test_ec2_instances_tagged(aws_resources):
    """All EC2 instances must have required tags"""
    ec2 = aws_resources['ec2']
    instances = ec2.describe_instances()
    
    required_tags = ['Environment', 'Owner', 'CostCenter']
    
    for reservation in instances['Reservations']:
        for instance in reservation['Instances']:
            tags = {tag['Key']: tag['Value'] for tag in instance.get('Tags', [])}
            
            for required_tag in required_tags:
                assert required_tag in tags, \
                    f"Instance {instance['InstanceId']} missing tag: {required_tag}"

Run compliance tests:

pytest compliance_tests.py -v --tb=short

Testing in CI/CD Pipeline

Integrate testing into continuous integration workflows.

Complete Pipeline

# .github/workflows/infrastructure-test.yaml
name: Infrastructure Testing
on:
  pull_request:
    paths:
    - 'infrastructure/**'

jobs:
  static-analysis:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v3
    
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2
    
    - name: Terraform Format Check
      run: terraform fmt -check -recursive
      working-directory: infrastructure
    
    - name: Terraform Validate
      run: |
        terraform init -backend=false
        terraform validate
      working-directory: infrastructure
    
    - name: Run tfsec
      uses: aquasecurity/tfsec-action@v1.0.0
      with:
        working_directory: infrastructure
    
    - name: Run Checkov
      uses: bridgecrewio/checkov-action@master
      with:
        directory: infrastructure
        framework: terraform
  
  unit-tests:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v3
    
    - name: Setup Go
      uses: actions/setup-go@v4
      with:
        go-version: '1.21'
    
    - name: Run Terratest
      run: |
        cd test
        go test -v -timeout 30m
      env:
        AWS_DEFAULT_REGION: us-east-1
  
  integration-tests:
    runs-on: ubuntu-latest
    needs: [static-analysis, unit-tests]
    steps:
    - name: Checkout
      uses: actions/checkout@v3
    
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2
    
    - name: Deploy to Test Environment
      run: |
        terraform init
        terraform apply -auto-approve
      working-directory: infrastructure/test
      env:
        AWS_DEFAULT_REGION: us-east-1
    
    - name: Run Integration Tests
      run: |
        cd test
        go test -v -run Integration
      env:
        AWS_DEFAULT_REGION: us-east-1
    
    - name: Cleanup Test Environment
      if: always()
      run: terraform destroy -auto-approve
      working-directory: infrastructure/test
  
  compliance-check:
    runs-on: ubuntu-latest
    needs: integration-tests
    steps:
    - name: Checkout
      uses: actions/checkout@v3
    
    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
    
    - name: Install Dependencies
      run: |
        pip install pytest boto3 pyyaml
    
    - name: Run Compliance Tests
      run: pytest compliance_tests.py -v
      env:
        AWS_DEFAULT_REGION: us-east-1

graph TB
    A[Pull Request] --> B[Static Analysis]
    B --> C{Pass?}
    C -->|No| D[Report Failures]
    C -->|Yes| E[Unit Tests]
    E --> F{Pass?}
    F -->|No| D
    F -->|Yes| G[Integration Tests]
    G --> H{Pass?}
    H -->|No| D
    H -->|Yes| I[Compliance Check]
    I --> J{Pass?}
    J -->|No| D
    J -->|Yes| K[Approve PR]
    
    style K fill:#d4f1d4
    style D fill:#ffe6e6

Testing Best Practices

Effective infrastructure testing requires discipline and structure.

Test Pyramid

Balance different test types:

         /\
        /  \  E2E Tests (Few)
       /____\
      /      \  Integration Tests (Some)
     /________\
    /          \  Unit Tests (Many)
   /__Static____\
  /   Analysis   \  (Most)

Fail Fast

Order tests from fastest to slowest:

# Run quick static analysis first
stages:
  - lint
  - unit-test
  - integration-test
  - compliance

# Static analysis completes in seconds
lint:
  stage: lint
  script:
    - terraform fmt -check
    - terraform validate
    - tfsec

# Unit tests complete in minutes
unit-test:
  stage: unit-test
  script:
    - go test -v -short

# Integration tests take 10-30 minutes
integration-test:
  stage: integration-test
  script:
    - go test -v -run Integration

Isolated Test Environments

Use dedicated environments for testing:

# test/main.tf
terraform {
  backend "s3" {
    bucket = "terraform-state-test"
    key    = "test/infrastructure-${var.test_id}.tfstate"
    region = "us-east-1"
  }
}

resource "random_id" "test_suffix" {
  byte_length = 4
}

module "test_infrastructure" {
  source = "../modules/complete-stack"
  
  environment = "test-${random_id.test_suffix.hex}"
  # Use small, cheap resources for testing
  instance_type = "t3.small"
  database_size = "db.t3.small"
}

Common Pitfalls

Testing Only Happy Path: Tests must cover failure scenarios. Test what happens when resources don't exist, permissions are denied, or dependencies fail.

No Cleanup: Failed tests leave orphaned resources. Always use defer terraform.Destroy() in Go tests or cleanup blocks in other frameworks.

Slow Test Suites: Hour-long test runs discourage running tests. Optimize by parallelizing tests and using smaller resource sizes.

Ignoring Test Failures: Treating test failures as acceptable defeats the purpose. Fix failures immediately or delete unreliable tests.

Key Takeaways