Kafka Integration

Knative offers three levels of Kafka integration:

• KafkaSource -- Consume from Kafka topics into Knative
• KafkaChannel -- Use Kafka as the backing store for Channels
• Kafka Broker -- Native Kafka-backed Broker (best performance)

KafkaSource in Detail

apiVersion: sources.knative.dev/v1beta1
kind: KafkaSource
metadata:
  name: payment-events
spec:
  consumerGroup: knative-payments
  bootstrapServers:
    - kafka-cluster.kafka.svc.cluster.local:9092
  topics:
    - payments.completed
    - payments.failed
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: default
  # Optional: initial offset
  initialOffset: latest     # or "earliest"
  

• Each Kafka message becomes a CloudEvent
• Kafka headers are mapped to CloudEvent extensions
• Consumer group manages offsets automatically

KafkaChannel: Durable Event Delivery

# Install Kafka Channel
kubectl apply -f https://github.com/knative-extensions/eventing-kafka-broker/releases/download/knative-v1.14.0/eventing-kafka-channel-install.yaml

# Set KafkaChannel as the default channel type
apiVersion: v1
kind: ConfigMap
metadata:
  name: default-ch-webhook
  namespace: knative-eventing
data:
  default-ch-config: |
    clusterDefault:
      apiVersion: messaging.knative.dev/v1beta1
      kind: KafkaChannel
      spec:
        numPartitions: 3
        replicationFactor: 3
  

Key benefit: Events survive pod restarts and broker failures. In-Memory channels do NOT provide this guarantee.

Kafka Broker: Native Integration

# Install Kafka Broker
kubectl apply -f https://github.com/knative-extensions/eventing-kafka-broker/releases/download/knative-v1.14.0/eventing-kafka-broker.yaml

# Create a Kafka Broker
apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
  name: kafka-broker
  annotations:
    eventing.knative.dev/broker.class: Kafka
spec:
  config:
    apiVersion: v1
    kind: ConfigMap
    name: kafka-broker-config
    namespace: knative-eventing
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-broker-config
  namespace: knative-eventing
data:
  default.topic.partitions: "6"
  default.topic.replication.factor: "3"
  bootstrap.servers: "kafka-cluster:9092"
  

Kafka Integration Comparison

Recommendation: Use Kafka Broker for new deployments. Use KafkaSource when integrating with existing Kafka topics.

Knowledge Check

Custom Event Sources

Two approaches to create your own event sources:

ContainerSource Example

apiVersion: sources.knative.dev/v1
kind: ContainerSource
metadata:
  name: azure-blob-watcher
spec:
  template:
    spec:
      containers:
        - image: myregistry.azurecr.io/blob-watcher:v1
          env:
            - name: STORAGE_ACCOUNT
              value: "mystorageaccount"
            - name: CONTAINER_NAME
              value: "uploads"
            - name: CONNECTION_STRING
              valueFrom:
                secretKeyRef:
                  name: storage-secret
                  key: connection-string
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: default
  

Your container polls Azure Blob Storage and emits CloudEvents to $K_SINK when new blobs appear.

Autoscaler Tuning: config-autoscaler

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-autoscaler
  namespace: knative-serving
data:
  # Scale-to-zero settings
  enable-scale-to-zero: "true"
  scale-to-zero-grace-period: "30s"
  scale-to-zero-pod-retention-period: "0s"

  # Scaling windows
  stable-window: "60s"           # Window for stable mode decisions
  panic-window-percentage: "10"  # % of stable window for panic mode
  panic-threshold-percentage: "200" # Trigger panic if 2x target

  # Scale bounds
  max-scale-up-rate: "1000"      # Max ratio of scale-up per tick
  max-scale-down-rate: "2"       # Max ratio of scale-down per tick

  # Target utilization
  target-burst-capacity: "200"   # Extra capacity for bursts
  activator-capacity: "100"      # Requests activator can buffer
  

Panic Mode: Handling Traffic Spikes

  Normal (Stable Mode)              Panic Mode
  ------------------               ----------
  60-second window                  6-second window (10% of 60s)
  Gradual scaling                   Aggressive scaling

  Panic triggers when:
  observed_concurrency > 2x target (panic-threshold: 200%)

  Returns to stable when:
  traffic stays below target for full stable window
    

Observability Stack

Three pillars of observability for Knative:

Logging: Structured JSON logs from all Knative components, configurable via config-logging ConfigMap.

Metrics Configuration

# config-observability ConfigMap (knative-serving namespace)
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: knative-serving
data:
  # Enable Prometheus metrics
  metrics.backend-destination: prometheus

  # Request metrics reporting period
  metrics.reporting-period-seconds: "5"

  # Enable request tracing
  enable-tracing: "true"

  # Tracing backend
  zipkin-endpoint: "http://zipkin.observability.svc.cluster.local:9411/api/v2/spans"

  # Sample rate (1.0 = trace everything, 0.1 = 10%)
  sample-rate: "0.1"
  

Key Metrics to Monitor

Knowledge Check

Logging Configuration

# config-logging ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-logging
  namespace: knative-serving
data:
  # Log level for Knative components
  loglevel.controller: "info"
  loglevel.autoscaler: "info"    # Set to "debug" for troubleshooting
  loglevel.activator: "info"
  loglevel.webhook: "info"
  loglevel.queueproxy: "info"

  # Structured logging format
  zap-logger-config: |
    {
      "level": "info",
      "development": false,
      "outputPaths": ["stdout"],
      "errorOutputPaths": ["stderr"],
      "encoding": "json",
      "encoderConfig": {
        "timeKey": "ts",
        "levelKey": "level",
        "nameKey": "logger",
        "callerKey": "caller",
        "messageKey": "msg"
      }
    }
  

Knative with Istio

Istio provides advanced networking features beyond basic Kourier:

• mTLS -- Automatic mutual TLS between all services
• Authorization policies -- Fine-grained access control
• Traffic mirroring -- Shadow traffic for testing
• Circuit breaking -- Prevent cascade failures
• Observability -- Built-in metrics, tracing, access logs

# Install Knative with Istio networking
kubectl apply -f https://github.com/knative/net-istio/releases/download/knative-v1.14.0/net-istio.yaml

# Configure Knative to use Istio
kubectl patch configmap/config-network \
  --namespace knative-serving \
  --type merge \
  --patch '{"data":{"ingress-class":"istio.ingress.networking.knative.dev"}}'
  

Kourier vs. Istio: When to Choose

Recommendation: Start with Kourier. Move to Istio when you need mTLS, authorization, or are already using Istio for other workloads.

Multi-Tenant Knative

• Namespace isolation -- Each team gets their own namespace
• ResourceQuotas -- Limit total pods, CPU, memory per namespace
• LimitRanges -- Default resource requests/limits for containers
• Network Policies -- Restrict cross-namespace communication
• RBAC -- Control who can create/modify Knative resources

# Example ResourceQuota for a team namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-quota
  namespace: team-alpha
spec:
  hard:
    pods: "50"
    requests.cpu: "20"
    requests.memory: "40Gi"
    limits.cpu: "40"
    limits.memory: "80Gi"
  

Performance: Optimizing Cold Starts

# Pre-pull images with a DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: image-prepull
spec:
  template:
    spec:
      initContainers:
        - name: prepull
          image: myregistry.azurecr.io/my-api:v1
          command: ["sh", "-c", "exit 0"]
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
  

Concurrency Tuning Guide

annotations:
  autoscaling.knative.dev/target: "100"
spec:
  containerConcurrency: 0  # unlimited
      

annotations:
  autoscaling.knative.dev/target: "1"
spec:
  containerConcurrency: 1  # one at a time
      

Tip: Load test to find the right target. Start with target = 70% of what your container can handle at acceptable latency.

Knowledge Check

Troubleshooting: Service Not Ready

# Step 1: Check service status
kn service describe my-api
# Look at Conditions:
#   Ready: False
#   ConfigurationsReady: False
#   RoutesReady: True

# Step 2: Check the latest revision
kn revision describe my-api-00003
# Look for:
#   ContainerHealthy: False
#   ResourcesAvailable: False

# Step 3: Check pods
kubectl get pods -l serving.knative.dev/service=my-api
kubectl describe pod my-api-00003-deployment-xxx

# Common causes:
# - Image pull errors (wrong image name, missing credentials)
# - Container crash (check logs: kubectl logs ...)
# - Readiness probe failing
# - Resource limits too low (OOMKilled)
# - Missing ConfigMaps or Secrets
  

Troubleshooting: Revision Failures

# Revision stuck in "not ready"
kubectl get revisions
NAME               CONFIG NAME   READY   REASON
my-api-00003       my-api        False   ContainerMissing

# Check revision details
kubectl get revision my-api-00003 -o yaml | grep -A 10 "conditions"

# Common REASON values and fixes:
  

Troubleshooting: Eventing Issues

# Events not being delivered? Systematic check:

# 1. Check Broker status
kubectl get broker default -o yaml
# Is READY: True?

# 2. Check Trigger status
kubectl get triggers -o wide
# Are all triggers READY: True?
# Is the subscriber URL correct?

# 3. Check Source status
kubectl get pingsource,kafkasource -o wide

# 4. Deploy event-display to see what's arriving
kn service create debug-display \
  --image gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
kn trigger create catch-all --broker default --sink ksvc:debug-display
kubectl logs -l serving.knative.dev/service=debug-display -f

# 5. Check eventing controller logs
kubectl logs -n knative-eventing -l app=eventing-controller --tail=50

# 6. Check dead letter sink for failed deliveries
kubectl logs -l serving.knative.dev/service=dead-letter-handler
  

Knative on AKS Best Practices

• Use ACR integration -- Attach ACR to AKS for seamless image pulls
• Node pools -- Dedicated node pool for Knative system components
• Cluster autoscaler -- Enable AKS cluster autoscaler alongside Knative autoscaling
• Azure Monitor -- Integrate with Container Insights for centralized observability
• Azure Key Vault -- Use CSI driver for secrets instead of K8s Secrets
• Private endpoints -- Use private AKS clusters with internal load balancers
• Availability Zones -- Spread nodes across AZs for resilience

# AKS-specific: Internal load balancer for Kourier
kubectl annotate svc kourier -n kourier-system \
  service.beta.kubernetes.io/azure-load-balancer-internal="true"
  

AKS Network Configuration for Knative

# Create AKS cluster optimized for Knative
az aks create \
  --resource-group myRG \
  --name knative-cluster \
  --node-count 3 \
  --node-vm-size Standard_D4s_v3 \
  --network-plugin azure \
  --network-policy azure \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 20 \
  --zones 1 2 3 \
  --attach-acr myACR \
  --enable-managed-identity

# Configure DNS for Knative
# Option 1: Use Azure DNS with external-dns
# Option 2: Use nip.io for development
# Option 3: Configure config-domain with your domain
kubectl patch configmap/config-domain \
  --namespace knative-serving \
  --type merge \
  --patch '{"data":{"mycompany.com":""}}'
  

When to Use Knative vs. Regular Deployments

Decision Flowchart

  Is it an HTTP workload?
     /          \
   Yes           No --> Regular Deployment
    |
  Is it stateless?
     /          \
   Yes           No --> Regular Deployment (StatefulSet)
    |
  Does it have variable/bursty traffic?
     /          \
   Yes           No (steady) --> Either works, Knative adds convenience
    |
  Do you want scale-to-zero?
     /          \
   Yes           No
    |              |
  Knative!      Do you want built-in traffic splitting?
                   /          \
                 Yes           No --> Regular Deployment is fine
                  |
               Knative!
    

Migrating to Knative

• Phase 1: Install Knative alongside existing workloads
• Phase 2: Deploy new services as Knative Services
• Phase 3: Migrate stateless HTTP services one at a time
• Phase 4: Add Knative Eventing for event-driven patterns

# Knative can coexist with regular Deployments!
# Same cluster, same namespace, no conflicts.

# Convert a Deployment to Knative Service:
# 1. Take your container image
# 2. Create a Knative Service YAML
# 3. Move env vars, secrets, configmaps
# 4. Add autoscaling annotations
# 5. Deploy and test
# 6. Switch DNS / traffic
  

Knative Security Checklist

• Network Enable TLS everywhere (auto-tls + cert-manager)
• Network Use cluster-local for internal services
• Network Consider Istio for mTLS between services
• Auth Use RBAC to restrict Knative resource access
• Auth Service accounts with least privilege
• Secrets Use Azure Key Vault CSI driver, not plain K8s Secrets
• Secrets Never put secrets in container env vars directly
• Images Use private registries with image pull secrets
• Images Scan images for vulnerabilities
• Runtime Set resource limits to prevent noisy neighbors

Course Summary

Key Takeaways

• Knative brings serverless to Kubernetes -- scale-to-zero, auto-scaling, traffic management
• Kafka integration provides durable, high-throughput event backbone
• Tune the autoscaler for your workload -- concurrency, burst capacity, panic mode
• Observability is critical -- metrics, tracing, and structured logging
• Choose Kourier for simplicity, Istio for enterprise features
• Use Knative for stateless HTTP workloads with variable traffic
• Start small -- install alongside existing workloads, migrate gradually

Your containers now scale to zero, spring back to life, split traffic, and react to events. Welcome to serverless on Kubernetes.