Module 6 of 8

Storage & Volumes

Container dies. Data gone. Let's fix that.

CKA 10% CKAD 10%

Civica Kubernetes Training

The Problem

Why Storage Matters

"Container dies. Data gone. That is the problem. Every container has a writable filesystem, but it is as permanent as writing on a whiteboard. The moment someone erases it -- restart, crash, reschedule -- everything you wrote is lost."

The Challenges

  • Container filesystems are ephemeral by design
  • Data lost on container restart or Pod reschedule
  • Containers in the same Pod cannot share files by default
  • Stateful apps (databases, uploads) need durable storage

What We Will Solve

  • Share data between containers in a Pod
  • Persist data across Pod restarts
  • Dynamically provision cloud storage
  • Manage configuration and secrets safely
Overview

What We Will Cover

Core Storage

  • Volumes and volume types
  • emptyDir and hostPath
  • Persistent Volumes (PV)
  • Persistent Volume Claims (PVC)
  • StorageClasses and dynamic provisioning

Configuration & Advanced

  • ConfigMaps -- app configuration
  • Secrets -- sensitive data
  • CSI (Container Storage Interface)
  • Azure-specific storage options
  • Volume snapshots and best practices
Fundamentals

Volumes -- Attaching Storage to Pods

A Volume is a directory accessible to containers in a Pod. It is defined in the Pod spec and mounted into one or more containers.

  • Volumes have a lifecycle tied to the Pod (not the container)
  • If a container crashes, the volume data survives the restart
  • Multiple containers can share the same volume
  • Volume type determines where data lives and its durability
Key distinction: A Volume outlives a container but dies with the Pod (unless backed by persistent storage).

Volume Lifecycle

Pod
Container A
Container B
mount
Shared Volume

Both containers read/write to the same volume

emptyDir

emptyDir -- Scratch Paper That Lives with the Pod

"emptyDir is like scratch paper on your desk. It appears when you sit down (Pod starts) and gets thrown away when you leave (Pod deleted). But if you just take a coffee break (container restart), the paper stays."

  • Created empty when Pod is assigned to a node
  • Persists across container restarts (within the same Pod)
  • Deleted permanently when Pod is removed
  • Stored on node's disk or in memory (medium: Memory)
Use cases: Temporary cache, scratch space for sorting, sharing files between sidecar containers (e.g., log collector reading app logs).
apiVersion: v1
kind: Pod
metadata:
  name: shared-data
spec:
  containers:
    - name: writer
      image: busybox
      command: ["sh","-c",
        "echo hello > /data/greeting; sleep 3600"]
      volumeMounts:
        - name: scratch
          mountPath: /data
    - name: reader
      image: busybox
      command: ["sh","-c",
        "cat /data/greeting; sleep 3600"]
      volumeMounts:
        - name: scratch
          mountPath: /data
  volumes:
    - name: scratch
      emptyDir: {}
hostPath

hostPath -- Mounting Node Storage

hostPath mounts a file or directory from the host node's filesystem into a Pod. Useful for system-level workloads but dangerous for regular applications.

  • Data persists on that specific node
  • Pod must be scheduled on the same node to see the data
  • Security risk: exposes host filesystem to containers
  • Types: Directory, File, DirectoryOrCreate, FileOrCreate
Warning: Never use hostPath for regular application data. It breaks portability (tied to one node) and is a security risk. It is mainly for DaemonSets that need host access (monitoring agents, log collectors).
apiVersion: v1
kind: Pod
metadata:
  name: log-reader
spec:
  containers:
    - name: reader
      image: busybox
      command: ["tail","-f",
        "/var/log/syslog"]
      volumeMounts:
        - name: host-logs
          mountPath: /var/log
          readOnly: true
  volumes:
    - name: host-logs
      hostPath:
        path: /var/log
        type: Directory
Comparison

Volume Types at a Glance

TypeLifecycleShared Across PodsUse Case
emptyDirDies with PodNoScratch space, sidecar sharing
hostPathPersists on nodeOnly on same nodeSystem daemons, node-level access
PersistentVolumeIndependent of PodYes (with access modes)Databases, file uploads, stateful apps
configMapCluster lifetimeYes (read-only)Configuration files
secretCluster lifetimeYes (read-only)Credentials, TLS certs
projectedVariesNoCombine multiple sources into one mount
Rule of thumb: If your data needs to survive Pod deletion, you need a PersistentVolume. Everything else is temporary.
Quiz Time

Check Your Understanding: Volume Basics

1. What happens to an emptyDir volume when the Pod is deleted?

A) It is preserved on the node for the next Pod
B) It is permanently deleted along with all its data
C) It is moved to a PersistentVolume automatically
D) It remains until the node is restarted
An emptyDir volume is created when a Pod is assigned to a node and is permanently deleted when that Pod is removed from the node. The data does not survive Pod deletion.

2. Why is hostPath generally discouraged for regular application workloads?

A) It is slower than other volume types
B) It does not support read-write access
C) It ties the Pod to a specific node and poses security risks
D) It requires a CSI driver to function
hostPath mounts the host node's filesystem, which ties the Pod to a specific node (breaking portability) and exposes the host filesystem to containers (a security risk). It should only be used for system-level DaemonSets.

3. An emptyDir with medium: Memory stores data where?

A) In a cloud storage bucket
B) On the node's SSD
C) In a tmpfs (RAM-backed filesystem)
D) In the container's overlay filesystem
Setting medium: Memory on an emptyDir creates a tmpfs (RAM-backed filesystem). It is very fast but counts against the container's memory limit, and data is lost on Pod deletion.
Transition

We Need Something That Outlives the Pod

"emptyDir is scratch paper. hostPath ties you to one desk. But what if you need a real filing cabinet -- one that stays put even when you change offices? That is what Persistent Volumes are for."

The analogy: Think of Persistent Volumes as storage lockers at the gym. The gym (cluster) owns the lockers (PVs). You get a claim ticket (PVC) to reserve one. Even if you leave and come back (Pod restart), your stuff is still in the locker.
PV

Persistent Volumes (PV) -- The Storage Locker

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically by a StorageClass.

  • Cluster-scoped -- not tied to any namespace
  • Has a lifecycle independent of any Pod
  • Defines capacity, access modes, and storage backend
  • Can be pre-provisioned (static) or dynamically created
Access Modes:
ReadWriteOnce (RWO) -- one node read/write
ReadOnlyMany (ROX) -- many nodes read-only
ReadWriteMany (RWX) -- many nodes read/write
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-azure-disk
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: managed-premium
  azureDisk:
    diskName: my-data-disk
    diskURI: /subscriptions/.../my-data-disk
    kind: Managed
PVC

Persistent Volume Claims (PVC) -- Your Claim Ticket

"You do not ask the gym owner to physically hand you locker number 47. You fill out a claim form: 'I need a medium locker, with a lock.' The gym matches you to an available one. That is a PVC."

  • Namespace-scoped request for storage
  • Specifies desired size, access mode, and StorageClass
  • Kubernetes binds the PVC to a matching PV
  • Pods reference the PVC, not the PV directly
  • This abstraction separates storage consumers from storage providers
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: managed-premium
# Check binding status
$ kubectl get pvc data-claim
NAME         STATUS   VOLUME        CAPACITY
data-claim   Bound    pv-azure-01   20Gi
PVC

Using a PVC in a Pod

Once a PVC is bound to a PV, any Pod in the same namespace can mount it. The Pod references the PVC name -- it never needs to know about the underlying PV or cloud storage details.

  • Pod spec defines volumes referencing the PVC
  • Container spec defines volumeMounts with mount path
  • Data persists across Pod restarts and rescheduling
  • PVC prevents PV from being deleted while in use
apiVersion: v1
kind: Pod
metadata:
  name: database
spec:
  containers:
    - name: postgres
      image: postgres:15
      volumeMounts:
        - name: db-storage
          mountPath: /var/lib/postgresql/data
      env:
        - name: POSTGRES_PASSWORD
          value: "secret"
  volumes:
    - name: db-storage
      persistentVolumeClaim:
        claimName: data-claim
Lifecycle

PV/PVC Lifecycle & Reclaim Policies

PV/PVC Lifecycle

Available
Bound
Released
Reclaimed

PVC binds to PV → PVC deleted → PV enters Released state

Reclaim Policies

  • Retain -- PV kept after PVC deletion. Admin must manually reclaim. Data safe.
  • Delete -- PV and underlying storage deleted when PVC is deleted. Default for dynamic provisioning.
  • Recycle -- (Deprecated) Basic rm -rf and make available again.
Production tip: Use Retain for production databases. With Delete, deleting the PVC also deletes the Azure Disk -- data gone forever.
StorageClass

StorageClasses -- Choosing Your Locker Tier

"Not all lockers are created equal. Some are small and free (Standard HDD). Some are large and climate-controlled (Premium SSD). A StorageClass defines the tier -- you pick which tier you want in your PVC."

  • Defines the provisioner (who creates the storage)
  • Defines parameters (disk type, replication, encryption)
  • Defines reclaim policy and binding mode
  • Enables dynamic provisioning -- no pre-created PVs needed
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: premium-ssd
provisioner: disk.csi.azure.com
parameters:
  skuName: Premium_LRS
  kind: Managed
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
Azure

Azure Storage Classes on AKS

AKS comes with several built-in StorageClasses. You can also create custom ones.

StorageClassProvisionerDisk TypeAccess ModeUse Case
manageddisk.csi.azure.comStandard SSD (StandardSSD_LRS)RWOGeneral workloads
managed-premiumdisk.csi.azure.comPremium SSD (Premium_LRS)RWOProduction databases
managed-csidisk.csi.azure.comStandard SSD (CSI)RWODefault in newer AKS
managed-csi-premiumdisk.csi.azure.comPremium SSD (CSI)RWOHigh-performance
azurefilefile.csi.azure.comAzure Files (Standard)RWXShared file access
azurefile-premiumfile.csi.azure.comAzure Files (Premium)RWXHigh-perf shared files
Key difference: Azure Disk = RWO (one node). Azure Files = RWX (many nodes). Choose based on whether multiple Pods on different nodes need simultaneous access.
Dynamic

Dynamic Provisioning -- Automatic Locker Assignment

"With dynamic provisioning, you do not need an admin to create storage in advance. You submit your claim, and the system automatically provisions a new disk in Azure, creates a PV, and binds it to your PVC. It is like walking into a hotel and a room is ready before you even check in."

  • PVC references a StorageClass
  • StorageClass provisioner creates storage on-demand
  • PV is automatically created and bound
  • No admin intervention needed
# Just create a PVC -- that's it!
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: managed-csi-premium
  resources:
    requests:
      storage: 100Gi
# Azure automatically creates:
# 1. A Premium SSD Managed Disk
# 2. A PV object in Kubernetes
# 3. Binds PVC to PV
$ kubectl get pvc,pv
NAME       STATUS   VOLUME       CAPACITY
my-data    Bound    pvc-a1b2c3   100Gi
Binding Mode

WaitForFirstConsumer

By default, PVCs are provisioned immediately. But this can cause problems in multi-zone clusters -- the disk might be created in a different zone than the Pod.

  • Immediate -- PV provisioned when PVC is created
  • WaitForFirstConsumer -- PV provisioned when a Pod first uses the PVC
  • WaitForFirstConsumer ensures the disk is in the same zone as the Pod
AKS best practice: Always use WaitForFirstConsumer in multi-zone AKS clusters. Otherwise, you may get a disk in Zone 1 and a Pod in Zone 2, causing a mount failure.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: zone-aware-ssd
provisioner: disk.csi.azure.com
parameters:
  skuName: Premium_LRS
volumeBindingMode: WaitForFirstConsumer

WaitForFirstConsumer

PVC Created
Pending...
Pod Scheduled
Disk in Same Zone
Expansion

Volume Expansion

Running out of space? You can expand a PVC without downtime (if the StorageClass allows it).

  • StorageClass must have allowVolumeExpansion: true
  • Edit the PVC to increase spec.resources.requests.storage
  • Azure Disk: requires Pod restart for filesystem resize
  • Azure Files: online expansion (no restart needed)
  • You can only increase size, never decrease
# Expand PVC from 20Gi to 50Gi
$ kubectl patch pvc data-claim -p \
  '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

# Check status
$ kubectl get pvc data-claim
NAME         STATUS   CAPACITY
data-claim   Bound    50Gi

# For Azure Disk, restart the Pod:
$ kubectl delete pod database
# Pod recreated by Deployment/StatefulSet
# Filesystem resized on mount
Quiz Time

Check Your Understanding: PV & PVC

1. What is the scope difference between a PV and a PVC?

A) PV is cluster-scoped; PVC is namespace-scoped
B) Both are namespace-scoped
C) PV is namespace-scoped; PVC is cluster-scoped
D) Both are cluster-scoped
PersistentVolumes are cluster-scoped resources (not in any namespace). PersistentVolumeClaims are namespace-scoped, meaning a PVC in namespace "dev" cannot be used by Pods in namespace "prod."

2. What reclaim policy should you use for production databases?

A) Delete
B) Retain
C) Recycle
D) Archive
Retain keeps the PV and its underlying storage after the PVC is deleted. This prevents accidental data loss. With Delete, deleting the PVC also deletes the Azure Disk -- your data is permanently gone.

3. Why is WaitForFirstConsumer important in multi-zone AKS clusters?

A) It makes provisioning faster
B) It reduces storage costs
C) It ensures the disk is created in the same zone as the Pod
D) It enables volume expansion
With WaitForFirstConsumer, the PV is not created until a Pod needs it. This lets Kubernetes provision the disk in the same availability zone as the scheduled Pod, preventing cross-zone mount failures.
Transition

Beyond Storage: Configuration & Secrets

"We have solved the data persistence problem. But applications need more than just disk space. They need configuration (database URLs, feature flags) and secrets (passwords, API keys). Kubernetes has dedicated objects for both."

ConfigMaps

The bulletin board for app configuration. Non-sensitive key-value pairs and files.

Secrets

The vault for sensitive data. Base64-encoded, with access controls.

ConfigMap

ConfigMaps -- The Bulletin Board

"A ConfigMap is like the bulletin board in your office. You pin up configuration that anyone can read: the Wi-Fi password for guests, the lunch menu, office hours. It is not secret -- just information your app needs."

  • Stores non-sensitive configuration data
  • Key-value pairs or entire files
  • Consumed as environment variables or mounted files
  • Changes to mounted ConfigMaps auto-update in Pods (with delay)
  • Max size: 1 MiB
# Create from literal values
$ kubectl create configmap app-config \
    --from-literal=DB_HOST=postgres \
    --from-literal=LOG_LEVEL=info

# Create from a file
$ kubectl create configmap nginx-conf \
    --from-file=nginx.conf
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DB_HOST: postgres.default.svc.cluster.local
  LOG_LEVEL: info
  MAX_CONNECTIONS: "100"
ConfigMap

Using ConfigMaps in Pods

As Environment Variables

containers:
  - name: app
    image: myapp:latest
    envFrom:
      - configMapRef:
          name: app-config
    # Or specific keys:
    env:
      - name: DATABASE_HOST
        valueFrom:
          configMapKeyRef:
            name: app-config
            key: DB_HOST

As Mounted Files

containers:
  - name: nginx
    image: nginx:latest
    volumeMounts:
      - name: config
        mountPath: /etc/nginx/conf.d
volumes:
  - name: config
    configMap:
      name: nginx-conf
      # Each key becomes a file
      # Key = filename, Value = content
Auto-update: ConfigMaps mounted as volumes update automatically (within ~60 seconds). Environment variables do not update -- the Pod must be restarted.
Secrets

Secrets -- The Vault for Sensitive Data

"If ConfigMaps are the bulletin board, Secrets are the locked safe in the manager's office. Passwords, API keys, TLS certificates -- anything you would not want posted on the wall goes here."

  • Stored as Base64-encoded data (not encrypted by default!)
  • Types: Opaque, kubernetes.io/tls, kubernetes.io/dockerconfigjson
  • Consumed the same way as ConfigMaps (env vars or volumes)
  • Mounted Secrets are stored in tmpfs (never written to disk on the node)
  • Max size: 1 MiB
Important: Base64 is encoding, not encryption. Anyone with API access can read Secrets. Enable encryption at rest and use RBAC to restrict access.
# Create from literal
$ kubectl create secret generic db-creds \
    --from-literal=username=admin \
    --from-literal=password=S3cur3P@ss

# Create TLS secret
$ kubectl create secret tls my-tls \
    --cert=tls.crt --key=tls.key
apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
data:
  username: YWRtaW4=       # base64
  password: UzNjdXIzUEBzcw== # base64
Secrets

Using Secrets in Pods

As Environment Variables

containers:
  - name: app
    image: myapp:latest
    env:
      - name: DB_USER
        valueFrom:
          secretKeyRef:
            name: db-creds
            key: username
      - name: DB_PASS
        valueFrom:
          secretKeyRef:
            name: db-creds
            key: password

As Mounted Files

containers:
  - name: app
    image: myapp:latest
    volumeMounts:
      - name: creds
        mountPath: /etc/secrets
        readOnly: true
volumes:
  - name: creds
    secret:
      secretName: db-creds
      # Files: /etc/secrets/username
      #        /etc/secrets/password
      # Values are auto-decoded from base64
Best practice: Mount Secrets as files rather than env vars. Environment variables can leak in logs, error reports, and child processes. Mounted files in tmpfs are more secure.
Security

Securing Secrets -- Beyond Base64

Kubernetes-Native

  • Encryption at rest: Configure EncryptionConfiguration to encrypt Secrets in etcd
  • RBAC: Restrict who can read Secrets
  • Audit logging: Track Secret access
  • Namespace isolation: Secrets are namespace-scoped

Azure Integration

  • Azure Key Vault Provider: Mount Key Vault secrets directly into Pods via CSI driver
  • Managed Identity: No credentials needed to access Key Vault
  • Auto-rotation: Secrets update automatically from Key Vault
  • External Secrets Operator: Sync Key Vault to K8s Secrets
# Azure Key Vault CSI SecretProviderClass
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kv
spec:
  provider: azure
  parameters:
    keyvaultName: my-keyvault
    objects: |
      array:
        - |
          objectName: db-password
          objectType: secret
    tenantId: "your-tenant-id"
Quiz Time

Check Your Understanding: Secrets & ConfigMaps

1. How are Kubernetes Secrets stored by default?

A) AES-256 encrypted in etcd
B) Base64-encoded (not encrypted) in etcd
C) Encrypted with the cluster's TLS certificate
D) Stored in a separate secure database
By default, Kubernetes Secrets are stored as Base64-encoded data in etcd, which is encoding, not encryption. To actually encrypt Secrets at rest, you must configure an EncryptionConfiguration on the API server.

2. A ConfigMap mounted as a volume is updated. What happens to Pods using it?

A) Pods restart automatically
B) The mounted files update automatically (with a delay of up to ~60 seconds)
C) Nothing -- the Pod must be deleted and recreated
D) Only new Pods see the updated ConfigMap
ConfigMaps mounted as volumes are automatically updated in running Pods (typically within 60 seconds). However, ConfigMaps consumed as environment variables are NOT updated -- the Pod must be restarted for env vars to reflect changes.

3. Why is mounting Secrets as files preferred over environment variables?

A) Files are faster to read
B) Environment variables have a size limit
C) Environment variables can leak in logs, error reports, and child processes
D) Mounted files support encryption while env vars do not
Environment variables are often included in crash dumps, debug output, and are inherited by child processes, making them more likely to leak. Mounted Secret files are stored in tmpfs (RAM) and are only accessible to the specific container.
Transition

The Plugin Architecture: CSI

"Kubernetes started with built-in storage drivers for every cloud provider. That quickly became unsustainable. Enter CSI -- the Container Storage Interface. It is a plugin system that lets any storage vendor write a driver that works with any container orchestrator."

Analogy: CSI is like USB for storage. Just as USB lets any peripheral work with any computer, CSI lets any storage system work with any container orchestrator. Plug in the driver, and it just works.
CSI

Container Storage Interface (CSI)

  • Industry standard for storage in container orchestrators
  • Storage vendors ship their own drivers (out-of-tree)
  • No need to modify Kubernetes core code
  • Supports: provisioning, attaching, mounting, snapshots, cloning, expansion

Azure CSI Drivers

  • disk.csi.azure.com -- Azure Managed Disks
  • file.csi.azure.com -- Azure Files (SMB/NFS)
  • blob.csi.azure.com -- Azure Blob Storage (NFS/FUSE)
  • secrets-store.csi.k8s.io -- Azure Key Vault

CSI Architecture

Pod
kubelet
CSI Driver (DaemonSet)
Cloud Storage API
Azure Disk
Azure Files
Azure Blob
Snapshots

Volume Snapshots

Volume Snapshots are point-in-time copies of a PV. Like taking a photo of your data at a specific moment -- you can restore from it later.

  • Requires a VolumeSnapshotClass (like StorageClass for snapshots)
  • Create a VolumeSnapshot from an existing PVC
  • Restore by creating a new PVC from the snapshot
  • Azure Disk supports incremental snapshots
Use case: Take a snapshot before a database migration. If something goes wrong, restore from the snapshot.
# Create a snapshot
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: db-snapshot
spec:
  volumeSnapshotClassName: azure-disk-snapshot
  source:
    persistentVolumeClaimName: data-claim
# Restore from snapshot
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restored-data
spec:
  storageClassName: managed-csi-premium
  dataSource:
    name: db-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
StatefulSets

Storage & StatefulSets

StatefulSets use volumeClaimTemplates to automatically create a unique PVC for each replica. This is how databases like PostgreSQL and MongoDB get per-replica storage.

  • Each replica gets its own PVC: data-postgres-0, data-postgres-1, etc.
  • PVCs survive Pod deletion and rescheduling
  • When scaling down, PVCs are not deleted (data preserved)
  • Combined with Headless Services for stable network identity
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    spec:
      containers:
        - name: postgres
          image: postgres:15
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: managed-csi-premium
        resources:
          requests:
            storage: 50Gi
Projected

Projected Volumes

A Projected Volume combines multiple volume sources into a single mount point. Think of it as a folder that pulls files from different places.

  • Combines: secret, configMap, downwardAPI, serviceAccountToken
  • All sources appear as files in one directory
  • Useful for apps that read config from a single directory
volumes:
  - name: app-config
    projected:
      sources:
        - configMap:
            name: app-settings
        - secret:
            name: app-secrets
        - downwardAPI:
            items:
              - path: "labels"
                fieldRef:
                  fieldPath: metadata.labels
Best Practices

Storage Best Practices

Storage Architecture

  • Use dynamic provisioning -- avoid pre-creating PVs manually
  • Use WaitForFirstConsumer in multi-zone clusters
  • Set Retain reclaim policy for production data
  • Use Azure Files when multiple Pods need RWX access
  • Use Azure Disk (Premium SSD) for database workloads
  • Enable volume expansion on StorageClasses

Secrets & Configuration

  • Mount Secrets as files, not env vars
  • Enable encryption at rest for Secrets in etcd
  • Use Azure Key Vault CSI for production secrets
  • Keep ConfigMaps and Secrets small (under 1 MiB)
  • Use immutable ConfigMaps/Secrets when content won't change
  • Never store secrets in ConfigMaps
Troubleshooting

Storage Troubleshooting

SymptomLikely CauseFix
PVC stuck in PendingNo matching PV or StorageClassCheck kubectl describe pvc for events
Pod stuck in ContainerCreatingVolume cannot attach/mountCheck kubectl describe pod and node events
Multi-attach errorAzure Disk (RWO) used by multiple Pods on different nodesUse Azure Files (RWX) or ensure single-node access
Permission denied on mountContainer runs as non-root but volume is root-ownedUse securityContext.fsGroup in Pod spec
Zone mismatchDisk in zone 1, Pod in zone 2Use WaitForFirstConsumer binding mode
Secret not foundSecret in wrong namespace or typo in nameVerify with kubectl get secret -n <ns>
# Useful debugging commands
$ kubectl describe pvc my-claim         # Check PVC events
$ kubectl get events --sort-by=.lastTimestamp  # Recent events
$ kubectl describe pod my-pod           # Check mount errors
$ kubectl get sc                        # List StorageClasses
Hands-On

Workshop: Storage in Action

Exercise 1: Volumes

  1. Create a Pod with two containers sharing an emptyDir
  2. Write data in one container, read from the other
  3. Delete and recreate the Pod -- verify data is gone

Exercise 2: PV/PVC

  1. Create a PVC requesting 5Gi of Premium SSD
  2. Deploy a Pod that mounts the PVC
  3. Write data, delete the Pod, recreate it
  4. Verify data persists

Exercise 3: ConfigMaps

  1. Create a ConfigMap with app settings
  2. Mount it as a volume in a Pod
  3. Update the ConfigMap and watch it auto-update

Exercise 4: Secrets

  1. Create a Secret with database credentials
  2. Mount as files and env vars in a Pod
  3. Verify values are decoded correctly
Recap

Module 6 Summary

Volumes

  • emptyDir -- scratch paper, dies with Pod
  • hostPath -- node storage, use with caution
  • PV/PVC -- persistent, cloud-backed storage
  • Dynamic provisioning via StorageClasses

Configuration

  • ConfigMaps for non-sensitive config
  • Secrets for sensitive data (base64, not encrypted)
  • Azure Key Vault CSI for production
  • Mount as files, not env vars

Azure Storage

  • Azure Disk (RWO) -- databases
  • Azure Files (RWX) -- shared access
  • WaitForFirstConsumer for zones
  • CSI drivers for all Azure storage
Key takeaway: Volumes solve the ephemeral problem. PV/PVC decouple storage consumers from providers. StorageClasses automate provisioning. ConfigMaps and Secrets inject configuration safely. CSI makes it all pluggable.
Preview

Coming Up Next

In Module 7, we will build on everything we have learned:

Observability

Monitoring, logging, and metrics with Prometheus and Grafana

Health Checks

Liveness, readiness, and startup probes in depth

Debugging

Advanced troubleshooting techniques for production clusters

Final Quiz

Module 6 -- Final Check

1. Which Azure storage option should you choose when multiple Pods on different nodes need read/write access to the same volume?

A) Azure Managed Disk (Premium SSD)
B) Azure Files
C) Azure Blob via CSI
D) emptyDir with medium: Memory
Azure Files supports ReadWriteMany (RWX) access mode, allowing multiple Pods on different nodes to read and write simultaneously. Azure Managed Disks only support ReadWriteOnce (RWO) -- one node at a time.

2. In a StatefulSet with volumeClaimTemplates, what happens to PVCs when you scale down?

A) PVCs are deleted immediately
B) PVCs are archived to cold storage
C) PVCs are preserved and can be reused when scaling back up
D) PVCs are moved to a different namespace
When a StatefulSet scales down, the PVCs created by volumeClaimTemplates are NOT deleted. They remain in the namespace so data is preserved. If you scale back up, the Pods reattach to their original PVCs.

3. Which command helps diagnose why a PVC is stuck in Pending status?

A) kubectl logs pvc/my-claim
B) kubectl describe pvc my-claim
C) kubectl get pv --all-namespaces
D) kubectl top pvc my-claim
kubectl describe pvc my-claim shows the Events section, which contains detailed messages about why the PVC cannot be bound -- such as no matching StorageClass, insufficient capacity, or provisioner errors.
End of Module

Thank You!

Module 6: Storage & Volumes -- Complete

Questions? Let's discuss before moving to Module 7.

Civica Kubernetes Training

← Back