Our Journey Today

Logging: Reading the Black Box Recorder

In aviation, when something goes wrong, investigators look at the black box. In Kubernetes, that black box is your container logs. Everything your application writes to stdout/stderr is captured by the kubelet.

kubectl logs — Your First Debugging Tool

# Basic: logs from a single pod
kubectl logs my-pod

# Specific container in a multi-container pod
kubectl logs my-pod -c sidecar

# Follow logs in real-time (like tail -f)
kubectl logs my-pod -f

# Last 100 lines only
kubectl logs my-pod --tail=100

# Logs from the last hour
kubectl logs my-pod --since=1h

# Logs from a PREVIOUS crashed container (critical for CrashLoopBackOff!)
kubectl logs my-pod --previous

# Logs from all pods matching a label
kubectl logs -l app=api --all-containers

# Logs from a specific deployment's pods
kubectl logs deployment/api-server --tail=50

Cluster-Level Log Aggregation

Node-level logs are ephemeral. For production, you need centralized log aggregation that survives pod restarts and node failures.

Monitoring: The Dashboard That Tells You Everything

Logs tell you what happened. Monitoring tells you what's happening right now. It's the difference between reading a crash report and watching the speedometer.

# Node resource usage
kubectl top nodes

# Pod resource usage
kubectl top pods -n production
kubectl top pods --sort-by=memory

Azure Monitor for AKS

Azure provides managed observability for AKS clusters without the overhead of running your own Prometheus/Grafana stack.

# Enable Container Insights on an existing AKS cluster
az aks enable-addons -a monitoring -n myCluster -g myResourceGroup

# Query container logs via KQL
ContainerLog | where LogEntry contains "error" | project TimeGenerated, LogEntry | top 50

Step 1: Get the Status

Always start with kubectl get to understand the current state. The STATUS column tells you what phase the pod is in.

# See all pods and their status
kubectl get pods -n production
NAME          READY   STATUS             RESTARTS   AGE
api-server    0/1     CrashLoopBackOff   5          3m
web-frontend  1/1     Running            0          1h
db-worker     0/1     Pending            0          10m

Step 2: Describe the Pod

kubectl describe is your detective report. It shows events, conditions, and the full pod specification. Focus on the Events section at the bottom.

kubectl describe pod api-server -n production

# Key sections to examine:

# 1. Status & Conditions
Status:       Running
Conditions:
  Ready:      False    # <-- readiness probe failing

# 2. Container State
State:        Waiting
  Reason:     CrashLoopBackOff
Last State:   Terminated
  Reason:     Error
  Exit Code:  1        # <-- non-zero = crash

# 3. Events (most important!)
Events:
  Warning  BackOff   kubelet  Back-off restarting failed container
  Normal   Pulled    kubelet  Successfully pulled image
  Warning  Unhealthy kubelet  Readiness probe failed: connection refused

Step 3: Read the Logs

Events tell you what Kubernetes sees. Logs tell you what the application sees. For a crashing container, use --previous to see the last run's output.

# Logs from the crashed container
kubectl logs api-server -n production --previous

# Example output that tells us the problem:
Starting API server on port 8080...
Connecting to database at postgres.db.svc:5432...
ERROR: FATAL: password authentication failed for user "admin"
ERROR: Cannot connect to database. Exiting.

The answer is clear: the database password is wrong. The fix might be updating a Secret or ConfigMap.

Step 4: Get Inside the Container

Sometimes you need to poke around inside a running container or test connectivity from within the cluster.

# Exec into a running container
kubectl exec -it api-server -n production -- /bin/sh

# Run a specific command without interactive shell
kubectl exec api-server -- cat /etc/config/settings.conf

# For crashed/minimal containers, use ephemeral debug containers (K8s 1.25+)
kubectl debug -it api-server --image=busybox --target=api-server

# Spin up a temporary debug pod in the same namespace
kubectl run debug --image=busybox --rm -it --restart=Never -- sh

# Test DNS resolution from inside the cluster
kubectl run dns-test --image=busybox --rm -it --restart=Never -- \
  nslookup api-service.production.svc.cluster.local

# Test connectivity to a service
kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- \
  curl -v http://api-service.production:8080/health

Debugging Services: Where's the Break in the Chain?

Users can't reach your app. The chain is: User → Ingress → Service → Endpoints → Pod. Check each link.

# 1. Does the Service exist and have the right selector?
kubectl get svc api-service -n production -o wide

# 2. Does the Service have endpoints? (Most common issue!)
kubectl get endpoints api-service -n production
# If ENDPOINTS is <none>, the selector doesn't match any running pods

# 3. Check that pod labels match the service selector
kubectl get pods -n production --show-labels
kubectl get svc api-service -n production -o jsonpath='{.spec.selector}'

# 4. Is the pod actually listening on the right port?
kubectl exec api-server -- netstat -tlnp
# or
kubectl exec api-server -- ss -tlnp

# 5. Test from within the cluster
kubectl run test --rm -it --image=curlimages/curl --restart=Never -- \
  curl http://api-service.production:8080

Scenario 1: Pod Stuck in Pending

A Pending pod means the scheduler cannot find a suitable node. Here's the investigation flow:

kubectl describe pod stuck-pod -n production

# Common events you'll see:
Events:
  Warning  FailedScheduling  0/3 nodes are available:
    1 Insufficient cpu           # Node doesn't have enough CPU
    2 node(s) had taints that the pod didn't tolerate  # Missing toleration

# Check node capacity vs usage
kubectl top nodes
kubectl describe node node-1

# Check for taints
kubectl get nodes -o json | \
  jq '.items[].spec.taints'

# Check PVC status
kubectl get pvc -n production

Scenario 2: CrashLoopBackOff

The container starts, crashes, and Kubernetes keeps restarting it with exponential back-off (10s, 20s, 40s... up to 5 minutes).

Scenario 3: ImagePullBackOff

Kubernetes cannot pull the container image. This is often one of the simplest issues to fix, but it can be confusing.

# 1. Check the exact image reference
kubectl describe pod my-pod | grep Image

# 2. Can you pull it manually?
docker pull myregistry.azurecr.io/app:v2

# 3. Is imagePullSecret configured?
kubectl get pod my-pod -o jsonpath=\
'{.spec.imagePullSecrets}'

# 4. Create/fix pull secret
kubectl create secret docker-registry \
  acr-creds \
  --docker-server=myregistry.azurecr.io \
  --docker-username=user \
  --docker-password=pass

The Systematic Debugging Flow

When something is broken, follow this order. It covers 95% of issues:

Debugging Node Issues

If a node shows NotReady, the kubelet may be down, or the node may have resource pressure.

# Check node status
kubectl get nodes
NAME     STATUS     ROLES          AGE   VERSION
node-1   Ready      control-plane  30d   v1.29.0
node-2   NotReady   <none>         30d   v1.29.0

# Detailed node conditions
kubectl describe node node-2

# Key conditions to check:
Conditions:
  MemoryPressure   True   # Node running out of memory
  DiskPressure     True   # Node running out of disk
  PIDPressure      False
  Ready            False  # Kubelet not reporting or unhealthy

# On the node itself (SSH or debug container):
systemctl status kubelet
journalctl -u kubelet -f
systemctl status containerd

Helm: The Package Manager for Kubernetes

Think of Helm like apt or npm for Kubernetes. Instead of managing dozens of YAML files, you install a "chart" that contains everything your application needs — deployments, services, configmaps, RBAC, all templated and versioned.

Essential Helm Commands

# Add a repository
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# Search for charts
helm search repo nginx
helm search hub wordpress        # search Artifact Hub

# Install a chart (creates a release)
helm install my-nginx bitnami/nginx -n web --create-namespace

# Install with custom values
helm install my-app ./my-chart -f production-values.yaml

# Override specific values on the command line
helm install my-app ./my-chart --set replicaCount=3,image.tag=v2

# List releases
helm list -n web

# Upgrade a release
helm upgrade my-nginx bitnami/nginx --set replicaCount=3

# Rollback to a previous revision
helm rollback my-nginx 1

# Uninstall a release
helm uninstall my-nginx -n web

# Show what would be installed (dry-run)
helm template my-app ./my-chart -f values.yaml

Helm Chart Structure

my-chart/
  Chart.yaml          # metadata (name, version)
  values.yaml         # default config values
  templates/
    deployment.yaml   # templated manifests
    service.yaml
    ingress.yaml
    _helpers.tpl      # template helpers
    NOTES.txt         # post-install message
  charts/             # sub-chart dependencies

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
      - name: app
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        ports:
        - containerPort: {{ .Values.service.port }}

Custom Resource Definitions: Extending Kubernetes

Kubernetes ships with resources like Pods, Services, and Deployments. CRDs let you define your own resource types, making the API server understand application-specific concepts.

# List all CRDs in the cluster
kubectl get crds

# Get custom resources
kubectl get certificates -n production
kubectl describe certificate my-tls -n production

Operators: Automating Human Knowledge

An Operator is a CRD + a custom controller that encodes operational knowledge. Think of it as a robot sysadmin that watches your custom resources and takes action.

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: my-db
spec:
  postgresVersion: 15
  instances:
  - replicas: 3
    dataVolumeClaimSpec:
      accessModes: [ReadWriteOnce]
      resources:
        requests:
          storage: 10Gi

CKA: Certified Kubernetes Administrator

CKA: Strategy & Time Management

CKA: Practice Scenarios

# etcd backup
ETCDCTL_API=3 etcdctl snapshot save \
  /opt/etcd-backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Common kubelet fix
ssh node-2
systemctl status kubelet
# Check for config errors
journalctl -u kubelet | tail -50
# Fix config and restart
systemctl restart kubelet

CKAD: Certified Kubernetes Application Developer

CKAD: Speed Tricks & Strategy

The CKAD is about speed. You need to create resources fast without writing YAML from scratch.

# Generate pod YAML
kubectl run nginx --image=nginx \
  --dry-run=client -o yaml > pod.yaml

# Generate deployment YAML
kubectl create deploy app \
  --image=app:v1 --replicas=3 \
  --dry-run=client -o yaml > deploy.yaml

# Generate service YAML
kubectl expose deploy app \
  --port=80 --target-port=8080 \
  --dry-run=client -o yaml > svc.yaml

# Generate job YAML
kubectl create job backup \
  --image=busybox \
  --dry-run=client -o yaml > job.yaml

kubectl Aliases & Speed Shortcuts

Set these up at the start of your exam. They save minutes over the 2-hour session.

# Essential aliases (add to ~/.bashrc at exam start)
alias k=kubectl
alias kgp='kubectl get pods'
alias kgs='kubectl get svc'
alias kgd='kubectl get deploy'
alias kgn='kubectl get nodes'
alias kd='kubectl describe'
alias kdp='kubectl describe pod'
alias kaf='kubectl apply -f'
alias kdf='kubectl delete -f'
alias kex='kubectl exec -it'
alias klo='kubectl logs'
alias klof='kubectl logs -f'

# Enable auto-completion (usually pre-configured)
source <(kubectl completion bash)
complete -o default -F __start_kubectl k

# Quick context switch
alias kcc='kubectl config current-context'
alias kuc='kubectl config use-context'

# The most important shortcut: generate YAML
export do='--dry-run=client -o yaml'
# Usage: k run nginx --image=nginx $do > pod.yaml

kubectl Power Moves for the Exam

# Pod with command
k run busybox --image=busybox \
  --restart=Never -- sleep 3600

# Service for existing deployment
k expose deploy app --port=80 --type=ClusterIP

# ConfigMap from literal
k create cm my-config \
  --from-literal=key=value

# Secret from literal
k create secret generic my-secret \
  --from-literal=pass=s3cret

# Role + RoleBinding
k create role pod-reader \
  --verb=get,list --resource=pods -n dev
k create rolebinding read-pods \
  --role=pod-reader --user=jane -n dev

# Force delete a stuck pod
k delete pod stuck --grace-period=0 --force

# Get all events sorted by time
k get events --sort-by=.lastTimestamp

# Get pod IPs
k get pods -o wide

# JSON path for specific field
k get pod my-pod \
  -o jsonpath='{.status.podIP}'

# All images in namespace
k get pods -o jsonpath=\
'{.items[*].spec.containers[*].image}'

# Watch pods in real-time
k get pods -w

# Replace a resource quickly
k get pod my-pod -o yaml > tmp.yaml
# edit tmp.yaml
k replace -f tmp.yaml --force

Study Resources & Practice Labs

Bookmark These Docs Pages

During the exam, you can access kubernetes.io/docs. Knowing where to find things saves critical time.

Key Takeaways