Advanced ArgoCD Features and Patterns

Beyond basic GitOps deployments, ArgoCD provides advanced capabilities for progressive delivery, multi-tenancy, and sophisticated deployment patterns. These features enable complex production scenarios while maintaining the GitOps philosophy.

Progressive Delivery with Argo Rollouts

Argo Rollouts extends Kubernetes deployments with advanced deployment strategies, integrating with ArgoCD for automated progressive delivery.

Canary Deployments

Gradually shift traffic to new versions:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-service
  namespace: production
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      containers:
      - name: api
        image: registry.example.com/api-service:v2.0.0
        ports:
        - containerPort: 8080
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 5m}
      - analysis:
          templates:
          - templateName: success-rate
      - setWeight: 50
      - pause: {duration: 10m}
      - analysis:
          templates:
          - templateName: success-rate
          - templateName: latency
      - setWeight: 80
      - pause: {duration: 10m}
  services:
    stable: api-service-stable
    canary: api-service-canary

The rollout increases traffic to the new version in stages, pausing between steps to analyze metrics. If analysis fails, automatic rollback occurs.

Analysis Templates

Define metric-based analysis for automated decisions:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: production
spec:
  metrics:
  - name: success-rate
    interval: 60s
    count: 5
    successCondition: result >= 0.95
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(http_requests_total{job="api-service",status=~"2.."}[5m]))
          /
          sum(rate(http_requests_total{job="api-service"}[5m]))
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency
  namespace: production
spec:
  metrics:
  - name: p95-latency
    interval: 60s
    count: 5
    successCondition: result <= 500
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket{job="api-service"}[5m])) by (le)
          ) * 1000

Analysis templates query Prometheus for metrics, evaluating success conditions. Failed analysis triggers automatic rollback.

sequenceDiagram
    participant Git
    participant ArgoCD
    participant Rollout
    participant Analysis
    participant Prometheus
    
    Git->>ArgoCD: New image version
    ArgoCD->>Rollout: Update Rollout spec
    Rollout->>Rollout: Deploy canary (20%)
    Rollout->>Analysis: Start analysis
    
    loop Every 60 seconds
        Analysis->>Prometheus: Query success rate
        Prometheus->>Analysis: Return metric
        Analysis->>Analysis: Evaluate condition
    end
    
    alt Analysis passes
        Rollout->>Rollout: Increase weight (50%)
    else Analysis fails
        Rollout->>Rollout: Rollback to stable
    end

Blue-Green Deployments

Deploy new version alongside old, then switch traffic:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-service
spec:
  replicas: 5
  strategy:
    blueGreen:
      activeService: api-service-active
      previewService: api-service-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 300
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      containers:
      - name: api
        image: registry.example.com/api-service:v2.0.0

Blue-green deploys the new version to separate pods, allowing validation before switching the active service. Manual or automatic promotion controls the cutover.

# Promote blue-green deployment
kubectl argo rollouts promote api-service

# Abort rollout (switch back to previous version)
kubectl argo rollouts abort api-service

Resource Hooks

Execute actions at specific points in the sync lifecycle using resource hooks.

Pre-Sync Hooks

Run tasks before applying manifests:

apiVersion: batch/v1
kind: Job
metadata:
  name: database-migration
  namespace: production
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: registry.example.com/migrations:v2.0.0
        command:
        - /bin/sh
        - -c
        - |
          echo "Running database migrations..."
          /migrations/migrate up
      restartPolicy: Never
  backoffLimit: 2

Pre-sync hooks run database migrations, configuration validation, or backups before deploying application changes.

Post-Sync Hooks

Execute tasks after successful sync:

apiVersion: batch/v1
kind: Job
metadata:
  name: smoke-tests
  namespace: production
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
      - name: test
        image: registry.example.com/smoke-tests:latest
        command:
        - /bin/sh
        - -c
        - |
          echo "Running smoke tests..."
          /tests/smoke-test.sh
      restartPolicy: Never
  backoffLimit: 1

Post-sync hooks validate deployments, send notifications, or trigger downstream processes after successful application sync.

Sync Waves

Control resource creation order:

# Create namespace first (wave 0)
apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    argocd.argoproj.io/sync-wave: "0"
---
# Create secrets next (wave 1)
apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "1"
---
# Deploy database (wave 2)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "2"
---
# Deploy application last (wave 3)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "3"

Sync waves ensure resources deploy in dependency order, preventing failures from missing prerequisites.

graph TD
    A[Wave 0: Namespace] --> B[Wave 1: Secrets]
    B --> C[Wave 2: Database]
    C --> D[Wave 3: Application]
    
    style A fill:#e1f5ff
    style B fill:#e8f4f8
    style C fill:#f0f7fa
    style D fill:#f8fbfc

Multi-Tenancy Patterns

Support multiple teams or customers with isolated ArgoCD environments.

Project-Based Isolation

Restrict resources and repositories per project:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-frontend
  namespace: argocd
spec:
  description: Frontend team applications
  
  # Only these repositories
  sourceRepos:
  - https://github.com/organization/frontend-*
  
  # Only these namespaces
  destinations:
  - namespace: frontend-dev
    server: https://kubernetes.default.svc
  - namespace: frontend-staging
    server: https://kubernetes.default.svc
  - namespace: frontend-prod
    server: https://kubernetes.default.svc
  
  # Denied cluster-scoped resources
  clusterResourceBlacklist:
  - group: ''
    kind: Namespace
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
  
  # Allowed namespace-scoped resources
  namespaceResourceWhitelist:
  - group: apps
    kind: Deployment
  - group: ''
    kind: Service
  - group: networking.k8s.io
    kind: Ingress
  
  # RBAC roles
  roles:
  - name: developer
    description: Frontend developers
    policies:
    - p, proj:team-frontend:developer, applications, get, team-frontend/*, allow
    - p, proj:team-frontend:developer, applications, sync, team-frontend/*, allow
    - p, proj:team-frontend:developer, applications, create, team-frontend/*, allow
    groups:
    - frontend-developers
  
  - name: admin
    description: Frontend team leads
    policies:
    - p, proj:team-frontend:admin, applications, *, team-frontend/*, allow
    groups:
    - frontend-leads

Projects enforce boundaries, ensuring teams only access their repositories, namespaces, and resource types.

ApplicationSet for Multi-Tenancy

Generate applications per team:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-applications
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: https://github.com/organization/team-configs
      revision: main
      files:
      - path: "teams/*/config.json"
  template:
    metadata:
      name: '{{team}}-app'
    spec:
      project: '{{team}}'
      source:
        repoURL: '{{repoURL}}'
        targetRevision: main
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{team}}-{{environment}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Team configuration files define parameters:

{
  "team": "frontend",
  "repoURL": "https://github.com/organization/frontend-manifests",
  "path": "kubernetes",
  "environment": "production"
}

ApplicationSet dynamically creates applications as teams add configuration files, automating onboarding.

Secrets Management Integration

Integrate with external secret management systems for secure credential handling.

External Secrets Operator

Fetch secrets from external systems:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "argocd-app"
          serviceAccountRef:
            name: argocd-application-controller
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: api-credentials
    creationPolicy: Owner
  data:
  - secretKey: api-key
    remoteRef:
      key: production/api-service
      property: api-key
  - secretKey: database-password
    remoteRef:
      key: production/database
      property: password

External Secrets Operator fetches credentials from HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault, syncing them into Kubernetes secrets automatically.

Sealed Secrets Integration

Encrypt secrets before storing in Git:

# Install kubeseal
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/kubeseal-linux-amd64
sudo install -m 755 kubeseal-linux-amd64 /usr/local/bin/kubeseal

# Create secret manifest
kubectl create secret generic api-credentials \
  --from-literal=api-key=secret-value \
  --dry-run=client -o yaml > secret.yaml

# Seal the secret
kubeseal -f secret.yaml -w sealed-secret.yaml

# Commit sealed secret to Git
git add sealed-secret.yaml
git commit -m "Add API credentials"
git push

The sealed secret decrypts only in the target cluster, allowing safe storage in Git repositories.

Notification Integrations

Configure notifications for deployment events and status changes.

Slack Notifications

Send sync status to Slack channels:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token
  
  trigger.on-deployed: |
    - when: app.status.operationState.phase in ['Succeeded']
      send: [app-deployed]
  
  trigger.on-health-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [app-health-degraded]
  
  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed]
  
  template.app-deployed: |
    message: |
      Application {{.app.metadata.name}} deployed successfully!
      Environment: {{.app.spec.destination.namespace}}
      Revision: {{.app.status.sync.revision}}
      Author: {{(call .repo.GetCommitMetadata .app.status.sync.revision).Author}}
    slack:
      attachments: |
        [{
          "title": "{{.app.metadata.name}}",
          "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
          "color": "good",
          "fields": [
            {"title": "Sync Status", "value": "{{.app.status.sync.status}}", "short": true},
            {"title": "Health", "value": "{{.app.status.health.status}}", "short": true}
          ]
        }]
  
  template.app-health-degraded: |
    message: |
      Application {{.app.metadata.name}} health degraded!
      Environment: {{.app.spec.destination.namespace}}
    slack:
      attachments: |
        [{
          "title": "{{.app.metadata.name}}",
          "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
          "color": "danger",
          "fields": [
            {"title": "Health Status", "value": "{{.app.status.health.status}}", "short": true}
          ]
        }]
  
  template.app-sync-failed: |
    message: |
      Failed to sync application {{.app.metadata.name}}
      Error: {{.app.status.operationState.message}}
    slack:
      attachments: |
        [{
          "title": "{{.app.metadata.name}}",
          "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
          "color": "danger"
        }]

Configure secret with Slack token:

kubectl create secret generic argocd-notifications-secret \
  -n argocd \
  --from-literal=slack-token=xoxb-your-slack-token

Subscribe applications to notifications:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service
  annotations:
    notifications.argoproj.io/subscribe.on-deployed.slack: team-deployments
    notifications.argoproj.io/subscribe.on-health-degraded.slack: team-alerts
    notifications.argoproj.io/subscribe.on-sync-failed.slack: team-alerts

Webhook Notifications

Trigger external systems on deployment events:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.webhook.deployment-tracker: |
    url: https://deployment-tracker.example.com/api/webhooks
    headers:
    - name: Authorization
      value: Bearer $webhook-token
  
  trigger.on-deployed: |
    - when: app.status.operationState.phase in ['Succeeded']
      send: [track-deployment]
  
  template.track-deployment: |
    webhook:
      deployment-tracker:
        method: POST
        body: |
          {
            "application": "{{.app.metadata.name}}",
            "environment": "{{.app.spec.destination.namespace}}",
            "revision": "{{.app.status.sync.revision}}",
            "timestamp": "{{.app.status.operationState.finishedAt}}",
            "author": "{{(call .repo.GetCommitMetadata .app.status.sync.revision).Author}}"
          }

Webhooks enable integration with deployment tracking, incident management, or analytics platforms.

ApplicationSet Advanced Generators

Use sophisticated generators for dynamic application creation.

Git Files Generator

Generate applications from files in Git:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: microservices
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: https://github.com/organization/microservices
      revision: main
      files:
      - path: "services/*/service.yaml"
  template:
    metadata:
      name: '{{service.name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/organization/microservices
        targetRevision: main
        path: 'services/{{service.name}}/manifests'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{service.namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Service definition files:

# services/api-service/service.yaml
service:
  name: api-service
  namespace: production
  replicas: 5

Pull Request Generator

Create preview environments for pull requests:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: pr-previews
  namespace: argocd
spec:
  generators:
  - pullRequest:
      github:
        owner: organization
        repo: frontend-app
        tokenRef:
          secretName: github-token
          key: token
      requeueAfterSeconds: 60
  template:
    metadata:
      name: 'frontend-pr-{{number}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/organization/frontend-app
        targetRevision: '{{head_sha}}'
        path: kubernetes
      destination:
        server: https://kubernetes.default.svc
        namespace: 'pr-{{number}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

ArgoCD automatically creates applications for open pull requests and deletes them when PRs close.

sequenceDiagram
    participant Dev as Developer
    participant GitHub
    participant AppSet as ApplicationSet
    participant ArgoCD
    participant K8s
    
    Dev->>GitHub: Create Pull Request
    GitHub->>GitHub: PR #123 opened
    
    loop Every 60 seconds
        AppSet->>GitHub: List open PRs
        GitHub->>AppSet: PR #123
        AppSet->>ArgoCD: Create Application (pr-123)
    end
    
    ArgoCD->>K8s: Deploy to namespace pr-123
    
    Dev->>GitHub: Close Pull Request
    
    loop Every 60 seconds
        AppSet->>GitHub: List open PRs
        GitHub->>AppSet: (no PR #123)
        AppSet->>ArgoCD: Delete Application (pr-123)
    end
    
    ArgoCD->>K8s: Delete namespace pr-123

Common Pitfalls

Rollout Without Analysis: Using Argo Rollouts without analysis templates risks promoting bad deployments. Always configure metric-based analysis for automated decisions.

Hook Resource Leaks: Hooks without proper delete policies accumulate over time. Use BeforeHookCreation or HookSucceeded to clean up hook resources.

Over-Permissive Projects: Granting broad cluster-scoped permissions defeats multi-tenancy isolation. Carefully whitelist resource types and namespaces.

Sealed Secrets Key Loss: Losing the sealing key prevents decryption of all sealed secrets. Back up the sealing key to secure storage immediately after installation.

Key Takeaways