GitOps Workflow with ArgoCD

GitOps transforms infrastructure and application management by using Git as the single source of truth. ArgoCD implements this pattern by continuously monitoring Git repositories and synchronizing desired state with live Kubernetes clusters.

GitOps Principles

GitOps establishes four core principles that guide operational practices.

Declarative Configuration

All system configuration exists as declarative files in Git:

# application-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
        version: v2.1.0
    spec:
      containers:
      - name: api
        image: registry.example.com/api-service:v2.1.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi

Declarative manifests describe the desired system state without specifying how to achieve it. ArgoCD reconciles differences between Git and the cluster automatically.

Versioned and Immutable

Git provides version control for all infrastructure changes:

# View deployment history
git log --oneline -- production/api-service/

# Compare versions
git diff v2.0.0 v2.1.0 -- production/api-service/

# Rollback to previous version
git revert HEAD
git push origin main

Every change creates a commit with authorship, timestamp, and description. This audit trail enables precise rollback to any previous state.

Pulled Automatically

ArgoCD pulls changes from Git rather than accepting pushed updates:

sequenceDiagram
    participant Dev as Developer
    participant Git as Git Repository
    participant Argo as ArgoCD
    participant K8s as Kubernetes Cluster
    
    Dev->>Git: Push manifest changes
    Git->>Git: Store new commit
    
    loop Every 3 minutes
        Argo->>Git: Poll for changes
        Git->>Argo: Return latest commit
        Argo->>Argo: Compare with cluster state
        alt Changes detected
            Argo->>K8s: Apply manifests
            K8s->>K8s: Reconcile resources
            Argo->>Argo: Update sync status
        end
    end

The pull model ensures cluster access credentials never leave the cluster, enhancing security by eliminating the need to distribute cluster credentials to CI/CD systems.

Continuously Reconciled

ArgoCD continuously compares Git with cluster state and corrects drift:

# ArgoCD Application with auto-sync
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/organization/k8s-manifests
    path: production/api-service
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true      # Delete resources removed from Git
      selfHeal: true   # Revert manual changes
    syncOptions:
    - CreateNamespace=true

When someone manually modifies a resource in the cluster, self-healing reverts it to match Git within minutes, maintaining consistency.

Repository Structure

Organizing manifests efficiently supports scalable GitOps workflows.

Environment-Based Structure

Separate directories for each environment:

k8s-manifests/
├── dev/
│   ├── api-service/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   └── ingress.yaml
│   └── database/
│       ├── statefulset.yaml
│       └── service.yaml
├── staging/
│   ├── api-service/
│   └── database/
└── production/
    ├── api-service/
    └── database/

Each environment has independent manifests, enabling different configurations without affecting other environments. Changes promotion follows dev -> staging -> production.

Application-Based Structure

Group by application with environment overlays:

k8s-manifests/
├── api-service/
│   ├── base/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   └── kustomization.yaml
│   ├── overlays/
│   │   ├── dev/
│   │   │   └── kustomization.yaml
│   │   ├── staging/
│   │   │   └── kustomization.yaml
│   │   └── production/
│   │       └── kustomization.yaml
└── database/
    ├── base/
    └── overlays/

Base manifests define common configuration, while overlays customize for each environment. Kustomize manages variations without duplicating YAML.

Helm-Based Structure

Use Helm charts with environment-specific values:

k8s-manifests/
├── charts/
│   └── api-service/
│       ├── Chart.yaml
│       ├── values.yaml
│       └── templates/
└── environments/
    ├── dev-values.yaml
    ├── staging-values.yaml
    └── production-values.yaml

ArgoCD applications reference the chart with different values files:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service-production
spec:
  source:
    repoURL: https://github.com/organization/k8s-manifests
    path: charts/api-service
    targetRevision: main
    helm:
      valueFiles:
      - ../../environments/production-values.yaml

Deployment Workflows

GitOps enables various deployment patterns through Git operations.

Direct Deployment

Push changes directly to the main branch:

# Update image version
sed -i 's/v2.1.0/v2.2.0/g' production/api-service/deployment.yaml

# Commit and push
git add production/api-service/deployment.yaml
git commit -m "Deploy api-service v2.2.0 to production"
git push origin main

# ArgoCD detects and applies within 3 minutes
argocd app get api-service --refresh

Direct deployment works well for automated updates from CI pipelines and non-critical changes with low risk.

Pull Request Workflow

Review changes before deployment:

# Create feature branch
git checkout -b update-api-v2.2.0

# Make changes
sed -i 's/v2.1.0/v2.2.0/g' production/api-service/deployment.yaml

# Commit and push
git add production/api-service/deployment.yaml
git commit -m "Update api-service to v2.2.0"
git push origin update-api-v2.2.0

# Create pull request in GitHub/GitLab
# Team reviews changes
# After approval, merge to main

Pull requests enable team review, automated testing, and approval gates before production changes take effect.

graph TD
    A[Developer Creates Branch] --> B[Update Manifests]
    B --> C[Push to Feature Branch]
    C --> D[Create Pull Request]
    D --> E{Automated Checks}
    E -->|Pass| F[Team Review]
    E -->|Fail| G[Fix Issues]
    G --> B
    F -->|Approved| H[Merge to Main]
    F -->|Changes Requested| G
    H --> I[ArgoCD Detects Change]
    I --> J[Sync to Cluster]
    
    style H fill:#e1f5ff
    style J fill:#e1f5ff

Progressive Delivery

Deploy changes gradually with traffic shifting:

# Argo Rollouts for canary deployment
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-service
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 5m}
      - setWeight: 50
      - pause: {duration: 5m}
      - setWeight: 80
      - pause: {duration: 5m}
  template:
    spec:
      containers:
      - name: api
        image: registry.example.com/api-service:v2.2.0

Progressive delivery reduces risk by gradually shifting traffic to new versions while monitoring metrics. Automatic rollback triggers if errors increase.

Image Update Automation

Automatically update image tags when new versions are built.

Image Updater

ArgoCD Image Updater watches container registries and updates manifests:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service
  annotations:
    argocd-image-updater.argoproj.io/image-list: api=registry.example.com/api-service
    argocd-image-updater.argoproj.io/api.update-strategy: semver
    argocd-image-updater.argoproj.io/api.allow-tags: regexp:^v[0-9]+\.[0-9]+\.[0-9]+$
    argocd-image-updater.argoproj.io/write-back-method: git
spec:
  source:
    repoURL: https://github.com/organization/k8s-manifests
    path: production/api-service

Image Updater polls the registry, detects new semantic versions, updates the manifest in Git, and commits the change. ArgoCD then syncs the updated manifest.

CI Pipeline Integration

CI pipelines can update manifests after building images:

# .github/workflows/deploy.yaml
name: Build and Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v3
    
    - name: Build and push image
      run: |
        docker build -t registry.example.com/api-service:${{ github.sha }} .
        docker push registry.example.com/api-service:${{ github.sha }}
    
    - name: Update manifest
      run: |
        git clone https://github.com/organization/k8s-manifests
        cd k8s-manifests
        sed -i "s|image: registry.example.com/api-service:.*|image: registry.example.com/api-service:${{ github.sha }}|" production/api-service/deployment.yaml
        git commit -am "Update api-service to ${{ github.sha }}"
        git push

The CI pipeline commits the new image tag to the manifest repository after a successful build, triggering ArgoCD to deploy the update.

Secrets Management

Handle sensitive data securely within GitOps workflows.

Sealed Secrets

Encrypt secrets before committing to Git:

# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

# Create regular secret
kubectl create secret generic api-credentials \
  --from-literal=api-key=secret-value \
  --dry-run=client -o yaml > secret.yaml

# Seal the secret
kubeseal -f secret.yaml -w sealed-secret.yaml

# Commit sealed secret to Git
git add sealed-secret.yaml
git commit -m "Add API credentials"
git push

The sealed secret can only be decrypted by the controller running in the cluster. Committing sealed secrets to Git is safe.

# sealed-secret.yaml (safe to commit)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: api-credentials
  namespace: production
spec:
  encryptedData:
    api-key: AgBh9...encrypted...data...

External Secrets Operator

Reference secrets from external secret managers:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: api-credentials
    creationPolicy: Owner
  data:
  - secretKey: api-key
    remoteRef:
      key: production/api-service
      property: api-key

External Secrets Operator fetches secrets from HashiCorp Vault, AWS Secrets Manager, or other backends, creating Kubernetes secrets dynamically without storing sensitive values in Git.

Multi-Cluster Deployments

Manage applications across multiple Kubernetes clusters.

Cluster List

Register all target clusters:

# Register production cluster
argocd cluster add prod-cluster --name production

# Register staging cluster  
argocd cluster add staging-cluster --name staging

# List registered clusters
argocd cluster list

Each cluster registration stores connection credentials, enabling ArgoCD to deploy applications to any registered cluster.

Application Per Cluster

Create separate applications for each environment:

# api-service-production.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service-production
spec:
  project: default
  source:
    repoURL: https://github.com/organization/k8s-manifests
    path: production/api-service
    targetRevision: main
  destination:
    name: production
    namespace: production
---
# api-service-staging.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service-staging
spec:
  project: default
  source:
    repoURL: https://github.com/organization/k8s-manifests
    path: staging/api-service
    targetRevision: main
  destination:
    name: staging
    namespace: staging

Each application points to a different cluster and manifest path, enabling independent deployment schedules and configurations.

ApplicationSet

Generate applications dynamically for multiple clusters:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: api-service
spec:
  generators:
  - list:
      elements:
      - cluster: production
        url: https://prod-cluster.example.com
      - cluster: staging
        url: https://staging-cluster.example.com
  template:
    metadata:
      name: 'api-service-{{cluster}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/organization/k8s-manifests
        path: '{{cluster}}/api-service'
        targetRevision: main
      destination:
        server: '{{url}}'
        namespace: '{{cluster}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

ApplicationSet reduces duplication by templating applications across clusters, simplifying multi-cluster management.

graph TB
    subgraph GitRepo[Git Repository]
        P[production/api-service/]
        S[staging/api-service/]
    end
    
    subgraph ArgoCD
        AS[ApplicationSet]
        A1[Application: production]
        A2[Application: staging]
    end
    
    subgraph Clusters
        PC[Production Cluster]
        SC[Staging Cluster]
    end
    
    AS --> A1
    AS --> A2
    
    P --> A1
    S --> A2
    
    A1 --> PC
    A2 --> SC
    
    style AS fill:#fff4e6

Monitoring and Notifications

Track deployment status and notify teams of changes.

Status Monitoring

Check application health and sync status:

# Get application status
argocd app get api-service

# Watch application sync progress
argocd app sync api-service --watch

# View application history
argocd app history api-service

ArgoCD tracks sync status, health status, and deployment history for each application, providing visibility into the current state.

Notifications

Configure notifications for sync events:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token
  
  trigger.on-sync-succeeded: |
    - when: app.status.operationState.phase in ['Succeeded']
      send: [app-sync-succeeded]
  
  template.app-sync-succeeded: |
    message: |
      Application {{.app.metadata.name}} synced successfully.
      Revision: {{.app.status.sync.revision}}
      Author: {{.app.status.operationState.operation.initiatedBy.username}}
    slack:
      attachments: |
        [{
          "title": "{{.app.metadata.name}}",
          "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
          "color": "good"
        }]

Notifications keep teams informed of deployments, failures, and sync status through Slack, email, or other channels.

Common Pitfalls

Manual Cluster Changes: Editing resources directly in the cluster causes drift. ArgoCD reverts manual changes when self-healing is enabled, frustrating operators who make emergency fixes.

Large Manifests in Single Repo: Storing all applications in one repository causes long sync times. Separate repositories by team or application group.

Ignoring Diff Detection: Some fields change frequently (timestamps, generated values). Configure diff settings to ignore these fields, preventing constant out-of-sync status.

No Rollback Plan: Git revert provides rollback, but teams must practice it. Test rollback procedures regularly to ensure confidence during incidents.

Key Takeaways