Module 04

Verification & Advanced Patterns

Trust but verify. Then push the boundaries.

"You promoted to staging. The ArgoCD sync succeeded. Pods are running.

But is the application actually working? Is the API responding? Are error rates normal?

Promotion without verification is just a faster way to break things."

What is Verification?

Automated Health Checks After Promotion

Verification runs after a Stage is promoted to confirm the application is healthy. Only verified Freight can proceed to the next Stage.

Uses Argo Rollouts AnalysisTemplates under the hood
Supports Prometheus queries, HTTP checks, custom jobs
Runs automatically after promotion completes
Results: Successful Failed Inconclusive

Why Verification Matters

Without Verification:
Promote to dev → Promote to staging → Promote to prod
(Hope for the best)

With Verification:
Promote to dev → Verify → Promote to staging → Verify → Promote to prod
(Confidence at every step)

AnalysisTemplates

Kargo uses Argo Rollouts AnalysisTemplate CRDs for verification.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: http-health-check
  namespace: my-web-app
spec:
  metrics:
  - name: webcheck
    interval: 10s
    count: 3
    successCondition: result == "200"
    failureLimit: 1
    provider:
      web:
        url: http://my-web-app.dev.svc.cluster.local/healthz
        method: GET
        jsonPath: "{$.statusCode}"

Prometheus Metrics Check

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-check
  namespace: my-web-app
spec:
  metrics:
  - name: error-rate
    interval: 30s
    count: 5
    successCondition: result[0] < 0.05
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc:9090
        query: |
          sum(rate(http_requests_total{
            service="my-web-app",
            status=~"5.."
          }[5m])) /
          sum(rate(http_requests_total{
            service="my-web-app"
          }[5m]))

Fails if the 5xx error rate exceeds 5% over the measurement window.

Adding Verification to a Stage

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: dev
  namespace: my-web-app
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: my-warehouse
    sources:
      direct: true
  promotionTemplate:
    steps:
    # ... git-clone, git-update, git-commit, git-push ...
  verification:
    analysisTemplates:
    - name: http-health-check
    - name: error-rate-check
    analysisRunMetadata:
      labels:
        app: my-web-app
        stage: dev

Verification Flow

1 Promotion completes (Git updated, ArgoCD synced)
2 Kargo creates an AnalysisRun from the template(s)
3 Argo Rollouts controller executes the checks
4 Results flow back to Kargo
5 If Successful: Freight is marked as verified in the Stage
6 Downstream Stages can now promote this Freight

Quiz 1 — Verification Basics

1. What CRD does Kargo use for verification?

a) Argo Rollouts AnalysisTemplate b) Kubernetes Job c) Tekton PipelineRun

2. When does verification run?

a) Before Freight is created b) After a promotion completes in a Stage c) Only when manually triggered

3. What happens if verification fails?

a) Kargo automatically rolls back b) The Freight is NOT marked as verified, blocking promotion to downstream stages c) The entire Project is deleted

Custom Job-Based Verification

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: smoke-test
  namespace: my-web-app
spec:
  metrics:
  - name: smoke-test
    provider:
      job:
        spec:
          template:
            spec:
              containers:
              - name: test
                image: myacr.azurecr.io/smoke-tests:latest
                command: ["/bin/sh", "-c"]
                args:
                - |
                  curl -f http://my-web-app.dev.svc/api/health
                  && echo "Tests passed"
              restartPolicy: Never
          backoffLimit: 1

Re-verification

Running Verification Again

Sometimes you need to re-run verification — maybe a flaky test, or transient infrastructure issue.

CLI: kargo verify --project my-web-app --stage dev
UI: Click "Reverify" on the Stage
Creates a new AnalysisRun
Previous results are preserved in history

Advanced Patterns

Beyond the basics: parallel stages, control flow, multi-warehouse, and more.

Parallel Stages

Deploy to Multiple Environments Simultaneously

Not all pipelines are linear. You might want to promote to multiple stages in parallel after dev verification.

dev (verified)
↓          ↓
staging-us    staging-eu
↓          ↓
prod-us       prod-eu

Parallel Stages Configuration

# staging-us: requests from dev
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: staging-us
  namespace: my-web-app
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: my-warehouse
    sources:
      stages: [dev]
  # ... promotion template for US region ...

---
# staging-eu: ALSO requests from dev (parallel!)
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: staging-eu
  namespace: my-web-app
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: my-warehouse
    sources:
      stages: [dev]
  # ... promotion template for EU region ...

Converging After Parallel Stages

Require ALL Parallel Stages Before Proceeding

Production can require Freight to be verified in all parallel staging environments before it becomes eligible.

# prod-us: requires both staging-us AND staging-eu
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: prod-us
  namespace: my-web-app
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: my-warehouse
    sources:
      stages:
      - staging-us
      - staging-eu   # Must be verified in BOTH

Multi-Warehouse Pipelines

Different Artifact Lifecycles

Sometimes your application images and infrastructure config change at different cadences.

# Two Warehouses
---
kind: Warehouse
metadata:
  name: app-images
spec:
  subscriptions:
  - image:
      repoURL: myacr.azurecr.io/my-web-app
      imageSelectionStrategy: SemVer
---
kind: Warehouse
metadata:
  name: infra-config
spec:
  subscriptions:
  - git:
      repoURL: https://github.com/my-org/infra-config.git
      branch: main

Quiz 2 — Advanced Patterns

1. How do you create parallel stages in Kargo?

a) Use a "parallel" keyword in the Stage spec b) Multiple stages request Freight from the same upstream stage c) Set concurrency: 2 in the Project spec

2. How can a downstream Stage require Freight to pass through ALL parallel stages?

a) List all parallel stages in the sources.stages array b) Set requireAll: true in the Stage spec c) This is not possible in Kargo

3. When would you use multiple Warehouses?

a) When different artifacts have different lifecycles or release cadences b) Only when you have more than 10 images c) Multiple Warehouses are not supported

Control Flow in Promotion Steps

Conditional Steps

Promotion steps can include conditional logic using the if field to skip steps based on context or previous step outputs.

promotionTemplate:
  steps:
  - uses: git-clone
    config:
      repoURL: https://github.com/my-org/k8s-config.git
      checkout:
      - branch: main
        path: ./src
  - uses: git-update
    config:
      path: ./src
      updates:
      - file: apps/my-web-app/${{ ctx.stage }}/values.yaml
        key: image.tag
        value: ${{ freight.images["myacr.azurecr.io/my-web-app"].tag }}
  # Only run argocd-update for prod
  - uses: argocd-update
    if: ctx.stage == "prod"
    config:
      apps:
      - name: my-web-app-prod

Rollback Scenarios

When Things Go Wrong

Kargo makes rollback straightforward because Freight is immutable and historical.

Re-promote previous Freight — select an older, known-good Freight and promote it to the broken Stage
Via CLI: kargo promote --stage prod --freight previous-good-freight
Via UI: Click on the previous Freight and promote it
Kargo runs the same promotion steps, but with the old artifact versions

Rollback in Practice

Current state: prod has Freight brave-dolphin (v1.6.0) — broken!

Previous good: Freight jolly-penguin (v1.5.0) — was working

Rollback:

# Promote the old, working Freight to prod
kargo promote --project my-web-app \
  --stage prod \
  --freight jolly-penguin

# Kargo runs promotion steps:
# - Clones Git repo
# - Updates values.yaml to image.tag: "1.5.0"
# - Commits and pushes
# - ArgoCD syncs back to v1.5.0

Kargo + ArgoCD Integration

ArgoCD Application per Stage

Each Stage typically has a corresponding ArgoCD Application watching the same Git path/branch.

Health Sync

Kargo can check ArgoCD Application health as part of the promotion to ensure the sync succeeded.

Shared Credentials

Both Kargo and ArgoCD may need access to the same Git repos. Coordinate credentials management.

RBAC Alignment

Ensure ArgoCD project RBAC and Kargo project RBAC are aligned for consistent access control.

ArgoCD Applications for Kargo

# ArgoCD Application for the dev Stage
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-web-app-dev
  namespace: argocd
  annotations:
    # Link to Kargo for UI integration
    kargo.akuity.io/authorized-stage: my-web-app:dev
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/k8s-config.git
    targetRevision: main
    path: apps/my-web-app/dev
  destination:
    server: https://kubernetes.default.svc
    namespace: my-web-app-dev
  syncPolicy:
    automated:
      selfHeal: true

Quiz 3 — Integration & Rollback

1. How do you roll back a Stage to a previous version in Kargo?

a) Delete the current Freight and Kargo auto-reverts b) Promote a previously verified Freight to the Stage c) Run kubectl rollout undo

2. What annotation links an ArgoCD Application to a Kargo Stage?

a) kargo.akuity.io/authorized-stage b) argocd.io/kargo-stage c) No annotation is needed

3. What makes rollback easy in Kargo?

a) Kargo stores full deployment snapshots b) Freight is immutable and historical, so you can re-promote any previous Freight c) Kargo automatically rolls back on any error

Troubleshooting Kargo

Common Issues & Solutions

Promotion stuck in Pending: Check if Freight is approved for the Stage. Check RBAC.
Git push fails: Verify Git credentials Secret (labels + annotations). Check branch protection rules.
Verification fails: kubectl get analysisrun -n my-web-app for detailed metrics results.
Freight not created: Check Warehouse status and controller logs. Verify image tags exist in registry.
ArgoCD not syncing: Ensure ArgoCD Application targetRevision matches the branch Kargo pushes to.

Useful Debug Commands

# Kargo controller logs
kubectl logs -n kargo deploy/kargo-controller -f

# Warehouse status
kubectl describe warehouse my-warehouse -n my-web-app

# Stage status and current Freight
kubectl describe stage dev -n my-web-app

# Promotion details
kubectl describe promotion promote-xyz -n my-web-app

# Verification (AnalysisRun) results
kubectl get analysisrun -n my-web-app
kubectl describe analysisrun <name> -n my-web-app

# Freight details
kargo get freight --project my-web-app -o yaml

Best Practices

Start Simple

Begin with a linear dev → staging → prod pipeline. Add complexity only when needed.

Always Verify

Add verification to every Stage, even dev. Catch issues early. Use health checks and smoke tests.

Use SemVer

Semantic versioning gives you the most control. Use constraints to limit major version jumps.

RBAC Matters

Restrict who can promote to production. Use Kargo RBAC roles for approvals.

Monitor Freight

Set up alerts for Freight stuck in non-verified state. Track promotion success rates.

Document Pipelines

Name your Stages and Warehouses clearly. Use the Kargo UI to visualize and share pipeline status.

Security Best Practices

Least privilege credentials — Git tokens should only have write access to the specific config repo
Rotate secrets regularly — Use Azure Key Vault or external-secrets-operator
Workload Identity — Prefer managed identity over static credentials for ACR access
Branch protection — Ensure Kargo's Git push goes through the same branch protection as human PRs (or use a dedicated service account)
Network policies — Restrict Kargo controller's network access to only required endpoints
Audit everything — Kargo Promotion resources serve as an audit log. Export to your SIEM.

The Complete Picture

CI builds & pushes image v1.5.0 to ACR
↓
Warehouse detects new image, creates Freight
↓ auto-promote
Stage: dev → Promotion (git-clone, git-update, git-commit, git-push)
↓
ArgoCD syncs dev cluster → Verify (health + error rate)
↓ manual promote
Stage: staging → Promotion → ArgoCD syncs → Verify
↓ approval required
Stage: prod → Approved → Promotion → ArgoCD syncs → Verify

Course Recap

Module 01: Kargo is a Kubernetes-native promotion engine that complements ArgoCD
Module 02: Warehouses watch for artifacts, Freight carries versions through the pipeline
Module 03: Stages are checkpoints with Promotion Templates (git-clone → update → commit → push)
Module 04: Verification gates promotions with health checks. Advanced patterns include parallel stages and rollback.

You now have the tools to build safe, automated, auditable promotion pipelines.

Resources & Next Steps

Kargo Docs: https://docs.kargo.io
Kargo GitHub: https://github.com/akuity/kargo
Argo Rollouts Docs: https://argoproj.github.io/argo-rollouts
Akuity Platform: https://akuity.io (managed ArgoCD + Kargo)
CNCF Slack: #kargo channel

Thank you!