StatefulSets & DaemonSets

Beyond Deployments

While Deployments manage stateless applications, Kubernetes provides specialized workload types for stateful applications and infrastructure-level concerns. StatefulSets and DaemonSets extend Kubernetes' capabilities for specific use cases.

StatefulSets

StatefulSets manage stateful applications requiring stable, unique pod identities and persistent storage (databases, message queues, cache clusters).

Why StatefulSets matter: Stateless applications (web servers) are interchangeable—any replica can serve requests. Stateful applications (databases) have unique identities and data tied to specific instances. StatefulSets preserve these identities across pod restarts and scaling operations.

graph TD
    A["Application Type"] -->B{Stateless?}
    B -->|Yes| C["Use Deployment
Any pod replaces any"]
    B -->|No| D["Use StatefulSet
Each pod has identity"]
    
    C -->C1["Web servers
API servers
Workers"]
    D -->D1["Databases
Cache clusters
Message queues"]
    
    style C fill:#90EE90
    style D fill:#FFB6C6

StatefulSet Characteristics

• Stable pod names: postgres-0, postgres-1, postgres-2 (not random)
• Stable DNS: Each pod gets DNS name: postgres-0.postgres-headless.default.svc.cluster.local
• Persistent storage: Each pod retains its PersistentVolume across restarts
• Ordered scaling: Pods scale down in reverse order (respects dependencies)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-headless
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15-alpine
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: postgres-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

Why Headless Service: StatefulSets require a Headless Service for stable DNS. Regular services provide a single IP; headless services return individual pod IPs.

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  clusterIP: None  # Headless
  selector:
    app: postgres
  ports:
  - port: 5432
    targetPort: 5432

DaemonSets

DaemonSets ensure all (or some) nodes run a copy of a pod. Used for infrastructure services that should run on every node.

Why DaemonSets matter: Some services (monitoring agents, log collectors, network plugins) must run on every node to function properly. Manually ensuring this is error-prone and doesn't scale.

graph LR
    A["Kubernetes Cluster"] -->|Node 1| B["DaemonSet Pod
Monitoring Agent"]
    A -->|Node 2| C["DaemonSet Pod
Monitoring Agent"]
    A -->|Node 3| D["DaemonSet Pod
Monitoring Agent"]
    
    style B fill:#FFD700
    style C fill:#FFD700
    style D fill:#FFD700

Common DaemonSet Use Cases

• Monitoring: Node exporter, Prometheus agent
• Logging: Fluent Bit, Filebeat (collect logs from every node)
• Networking: CNI plugins (Calico, Weave)
• Security: Intrusion detection agents

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true  # Access host network
      hostPID: true      # Access host processes
      containers:
      - name: node-exporter
        image: prom/node-exporter:latest
        ports:
        - containerPort: 9100
        volumeMounts:
        - name: proc
          mountPath: /host/proc
        - name: sys
          mountPath: /host/sys
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys

Node Selectors

Run DaemonSet only on specific nodes:

spec:
  template:
    spec:
      nodeSelector:
        disktype: ssd  # Only runs on nodes with this label

StatefulSet vs DaemonSet vs Deployment

Real-World Example: Stateful Database Cluster

apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
spec:
  clusterIP: None
  ports:
  - port: 3306
  selector:
    app: mysql

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql-headless
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: root-password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-storage
          mountPath: /var/lib/mysql
        livenessProbe:
          exec:
            command:
            - mysqladmin
            - ping
          initialDelaySeconds: 30
          periodSeconds: 10
  volumeClaimTemplates:
  - metadata:
      name: mysql-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 20Gi

Pod IPs remain stable:

• mysql-0.mysql-headless
• mysql-1.mysql-headless
• mysql-2.mysql-headless

Each pod maintains its own persistent volume, preserving data across restarts.

Common Pitfalls

• Using Deployment for stateful apps - Data loss on pod recreation
• Forgetting headless service - StatefulSet DNS doesn't work without it
• Not setting resource limits - DaemonSet pods consuming node resources
• Ignoring node selectors - DaemonSet runs on all nodes, including control plane (usually unwanted)

Key Takeaways

• StatefulSets provide stable pod identities and persistent storage for stateful applications
• Each StatefulSet pod maintains its own PersistentVolume, preserving data across restarts
• DaemonSets ensure infrastructure services run on every (or selected) cluster node
• Headless Services required for StatefulSets to enable stable DNS resolution
• StatefulSets scale orderly, respecting pod dependencies and termination sequences
• Node selectors control which nodes run DaemonSet pods
• Use Deployments for stateless applications, StatefulSets for databases and caches

Next Steps: Deploy a StatefulSet database, verify persistent storage behavior, implement DaemonSet for monitoring, test pod identity preservation.