Module 05

Advanced Patterns & Operations

Production-ready ArgoCD -- patterns, integrations, and operational excellence

Use arrow keys or click sides to navigate

The Journey

From Beginner to Production-Ready

You have learned the fundamentals: Applications, Projects, sync strategies, hooks, and ApplicationSets. Now imagine it is six months later. Your team manages 50+ services across multiple clusters. You need patterns that scale, monitoring that alerts before users notice, and disaster recovery that works at 3 AM. This module covers the advanced patterns that separate a demo from a production setup.

Pattern

App of Apps Pattern

An ArgoCD Application that manages other ArgoCD Applications.

Root App (App of Apps)

↓ manages

App: frontend

App: backend

App: database

App: monitoring

Why: Instead of manually creating each Application, you store Application manifests in Git. A "root" Application watches that directory and creates/manages all child Applications via ArgoCD.

App of Apps: Structure

# Git repo structure:
# cluster-config/
#   root-app.yaml          <-- The root Application
#   apps/
#     frontend.yaml         <-- Application for frontend
#     backend.yaml          <-- Application for backend
#     monitoring.yaml       <-- Application for monitoring
#     ingress.yaml          <-- Application for ingress controller

# root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/cluster-config.git
    targetRevision: main
    path: apps                   # Directory containing App YAMLs
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd            # Apps are created in argocd ns
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

App of Apps: Child App

# apps/frontend.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: frontend
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: frontend-team
  source:
    repoURL: https://github.com/org/frontend.git
    targetRevision: main
    path: deploy/helm
    helm:
      valueFiles:
        - values-prod.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: frontend
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

App of Apps vs ApplicationSets

App of Apps

Each child app is a full, unique YAML
Maximum flexibility per app
Different sources, projects, policies
Best for heterogeneous apps
More YAML, more control

ApplicationSets

One template generates many apps
Less YAML, less duplication
Best for homogeneous apps
Dynamic: auto-adds/removes apps
Less flexibility per individual app

Common pattern: Use App of Apps at the top level to manage ApplicationSets, which in turn generate individual Applications. Best of both worlds.

Quiz Time

App of Apps

1. In the App of Apps pattern, what does the root Application's source path contain?

Correct! The root app's source path contains ArgoCD Application YAML files. ArgoCD applies them to the argocd namespace, which creates the child Applications.

2. When should you choose App of Apps over ApplicationSets?

Correct! App of Apps excels when each child application is unique. ApplicationSets are better when applications follow a common pattern.

3. Can App of Apps and ApplicationSets be combined?

Correct! A common production pattern uses App of Apps to manage ApplicationSets, which generate the actual Applications. This provides both flexibility and scalability.

Multi-Source

Multi-Source Applications

A single Application that pulls from multiple sources -- combine a Helm chart with Git-stored values.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  sources:                          # Note: "sources" (plural)
    - repoURL: https://charts.bitnami.com/bitnami
      chart: nginx
      targetRevision: 15.3.5
      helm:
        valueFiles:
          - $values/envs/prod/values.yaml  # Reference the other source
    - repoURL: https://github.com/org/config.git
      targetRevision: main
      ref: values                   # Name this source "$values"
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app

Multi-Source: Why?

Your team uses a community Helm chart (e.g., nginx-ingress from a Helm repo) but stores environment-specific values files in your own Git repo. Before multi-source, you had to either fork the Helm chart or use ugly workarounds. Now you simply reference both sources in one Application.

Separate chart from config: Helm chart repo + Git values repo
Multiple config repos: Common base config + team-specific overrides
CRDs + Application: Deploy CRDs from one source, the app from another

Note: Multi-source Applications use sources (plural) instead of source (singular). The ref field creates a named reference accessible via $refname.

Image Updater

ArgoCD Image Updater

Automatically updates container image tags in your ArgoCD Applications when new images are pushed to a registry.

Container Registry (ACR)

→

Image Updater

→

Updates Git / Overrides

The workflow: CI builds and pushes myapp:v1.2.4 to ACR. Image Updater detects the new tag, updates the Application (either by writing back to Git or by setting parameter overrides), and ArgoCD syncs the change.

Image Updater: Configuration

# Annotate your Application to enable Image Updater
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
  annotations:
    # Which images to track
    argocd-image-updater.argoproj.io/image-list: >
      myapp=myregistry.azurecr.io/myapp
    # Update strategy: semver, latest, digest, name
    argocd-image-updater.argoproj.io/myapp.update-strategy: semver
    # Semver constraint
    argocd-image-updater.argoproj.io/myapp.allow-tags: "regexp:^v\\d+\\.\\d+\\.\\d+$"
    # Write-back method: git or argocd (parameter override)
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/write-back-target: kustomization

Image Updater: Update Strategies

semver

Follows semantic versioning. Picks the highest tag matching a constraint like ^1.x.

latest

Picks the most recently pushed tag (by creation date), optionally filtered by regex.

digest

Tracks a mutable tag (like latest) and updates when the digest changes.

name

Sorts tags alphabetically and picks the last one. Useful for date-based tags like 20240315.

Quiz Time

Multi-Source & Image Updater

1. In a multi-source Application, how do you reference a values file from a second source?

Correct! You set ref: values on the second source, then reference it as $values/path/to/file.yaml in the Helm valueFiles of the first source.

2. What does the ArgoCD Image Updater "semver" strategy do?

Correct! The semver strategy parses tags as semantic versions and picks the highest one matching your constraint (e.g., ^1.x picks the latest 1.x.x).

3. What are the two write-back methods for Image Updater?

Correct! "git" write-back commits the updated tag to your Git repo (true GitOps). "argocd" write-back sets a parameter override on the Application resource directly.

Notifications

ArgoCD Notifications

Send alerts when sync succeeds, fails, or health changes. Built into ArgoCD as the Notification Controller.

# argocd-notifications-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  # Define a Slack service
  service.slack: |
    token: $slack-token
  # Define a trigger
  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed]
  # Define a template
  template.app-sync-failed: |
    slack:
      attachments: |
        [{
          "color": "#E8601A",
          "title": "{{.app.metadata.name}} sync failed",
          "text": "Application {{.app.metadata.name}} sync failed.\nRevision: {{.app.status.sync.revision}}"
        }]

Notifications: Microsoft Teams

# Teams webhook integration
data:
  service.webhook.teams: |
    url: https://outlook.office.com/webhook/your-webhook-url
    headers:
      - name: Content-Type
        value: application/json

  template.app-sync-status: |
    webhook:
      teams:
        method: POST
        body: |
          {
            "@type": "MessageCard",
            "themeColor": "{{if eq .app.status.sync.status \"Synced\"}}28a745{{else}}E8601A{{end}}",
            "title": "ArgoCD: {{.app.metadata.name}}",
            "text": "Sync status: {{.app.status.sync.status}}\nHealth: {{.app.status.health.status}}"
          }

  trigger.on-sync-status-unknown: |
    - when: app.status.sync.status == 'Unknown'
      send: [app-sync-status]

Subscribing to Notifications

# Annotate Applications to subscribe to notifications
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: frontend-prod
  annotations:
    # Subscribe to Slack notifications
    notifications.argoproj.io/subscribe.on-sync-failed.slack: alerts-channel
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: deploys-channel

    # Subscribe to Teams webhook
    notifications.argoproj.io/subscribe.on-sync-failed.teams: ""

    # Subscribe to multiple triggers
    notifications.argoproj.io/subscribe.on-health-degraded.slack: alerts-channel

Pattern: notifications.argoproj.io/subscribe.<trigger>.<service>: <recipient>

Custom Health

Custom Health Checks (Lua)

ArgoCD has built-in health for standard K8s resources but you can add custom health checks for CRDs using Lua scripts.

# argocd-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.certmanager.io_Certificate: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message
            return hs
          end
          if condition.type == "Ready" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = "Certificate is valid"
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for certificate"
    return hs

Custom Health: Common CRDs

cert-manager Certificates: Check if certificate is Ready and not expired
Istio VirtualService: Always healthy (no status field) -- use custom "Healthy" return
Sealed Secrets: Check if the secret has been unsealed successfully
Azure resources (CAPZ): Check provisioning state for AzureManagedCluster
Crossplane claims: Check if the claim is bound and synced

Key: Custom health checks return one of: Healthy, Progressing, Degraded, Suspended, or Missing. They are written in Lua and configured in the argocd-cm ConfigMap.

Quiz Time

Notifications & Health

1. How does an Application subscribe to ArgoCD notifications?

Correct! Applications opt-in to notifications via annotations following the pattern: notifications.argoproj.io/subscribe.<trigger>.<service>: <recipient>

2. What language are ArgoCD custom health checks written in?

Correct! ArgoCD uses Lua for custom health checks and custom actions. Lua is lightweight, sandboxed, and fast -- ideal for embedding in a controller.

3. Where are custom health checks configured?

Correct! Custom health checks are defined in the argocd-cm ConfigMap under resource.customizations.health.<group_kind>.

Disaster Recovery

Backup & Restore

ArgoCD stores all state as Kubernetes resources. Backup is straightforward.

# Export all ArgoCD Applications, AppProjects, and settings
argocd admin export > argocd-backup.yaml

# This exports:
# - All Application resources
# - All AppProject resources
# - Repository credentials (encrypted)
# - Cluster credentials (encrypted)
# - ArgoCD settings (ConfigMaps)

# Restore to a new cluster
argocd admin import - < argocd-backup.yaml

# Alternative: Use Velero for namespace-level backup
velero backup create argocd-backup \
  --include-namespaces argocd \
  --storage-location azure-blob

DR: Git is Your Backup

Here is the beauty of GitOps: your Git repository IS your disaster recovery plan. If your ArgoCD installation or entire cluster is destroyed, you reinstall ArgoCD, point it at the same Git repos, and everything rebuilds itself. The only additional state you need to back up is repository credentials, cluster credentials, and ArgoCD configuration (SSO, RBAC, notifications).

DR checklist: (1) ArgoCD manifests in Git (App of Apps). (2) Repo credentials backed up or in a vault. (3) Cluster credentials can be re-generated. (4) SSO config in Git (argocd-cm, argocd-rbac-cm).

SSO with Azure AD

# argocd-cm ConfigMap -- OIDC configuration
data:
  url: https://argocd.yourcompany.com
  oidc.config: |
    name: Azure AD
    issuer: https://login.microsoftonline.com/YOUR_TENANT_ID/v2.0
    clientID: YOUR_APP_CLIENT_ID
    clientSecret: $oidc.azure.clientSecret   # From argocd-secret
    requestedScopes:
      - openid
      - profile
      - email
    requestedIDTokenClaims:
      groups:
        essential: true

Setup: Register an App Registration in Azure AD, configure redirect URI to https://argocd.yourcompany.com/auth/callback, and add group claims to the token configuration.

Azure AD: RBAC Mapping

# argocd-rbac-cm ConfigMap
data:
  policy.csv: |
    # Map Azure AD group to ArgoCD admin role
    g, "a]d-group-id-for-platform-team", role:admin

    # Map Azure AD group to read-only role
    g, "ad-group-id-for-developers", role:readonly

    # Custom role: can sync but not delete
    p, role:deployer, applications, get, */*, allow
    p, role:deployer, applications, sync, */*, allow
    p, role:deployer, applications, action/*, */*, allow
    g, "ad-group-id-for-deployers", role:deployer

  # Default policy for authenticated users
  policy.default: role:readonly
  # Use Azure AD groups for RBAC
  scopes: '[groups, email]'

Monitoring

Monitoring ArgoCD

ArgoCD exposes Prometheus metrics on port 8082 for all major components.

# Key metrics to monitor:
# Application Controller
argocd_app_info                        # App sync/health status
argocd_app_sync_total                  # Sync operation count
argocd_app_reconcile_count             # Reconciliation count
argocd_app_reconcile_bucket            # Reconciliation duration

# API Server
argocd_api_server_request_total        # API request count
grpc_server_handled_total              # gRPC request count

# Repo Server
argocd_git_request_total               # Git operations count
argocd_git_request_duration_seconds    # Git operation duration

# Create ServiceMonitor for Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-metrics
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: argocd

Troubleshooting Guide

App stuck "Progressing": Check pod logs for CrashLoopBackOff or pending PVCs. Use argocd app get to see resource-level health.
OutOfSync but nothing changed: Check ignoreDifferences for controller-managed fields (HPA replicas, mutating webhooks).
Sync fails with "already exists": Resource was created outside ArgoCD. Use argocd app sync --force or add the ArgoCD tracking label.
Slow reconciliation: Check Redis memory, increase repo-server replicas, enable ApplyOutOfSyncOnly.
Repository not accessible: Verify credentials with argocd repo list, check network policies, TLS certificates.

Essential Debug Commands

# Check ArgoCD component logs
kubectl logs -n argocd deployment/argocd-application-controller
kubectl logs -n argocd deployment/argocd-repo-server
kubectl logs -n argocd deployment/argocd-server

# Application-level debugging
argocd app get my-app --show-operation    # Last sync details
argocd app diff my-app                    # What would change
argocd app resources my-app               # List all resources
argocd app logs my-app                    # Application pod logs

# Force a refresh from Git
argocd app get my-app --refresh

# Hard refresh (invalidate cache)
argocd app get my-app --hard-refresh

# Check repo connectivity
argocd repo list
argocd repo validate https://github.com/org/repo.git

Quiz Time

Operations Mastery

1. In a GitOps disaster recovery scenario, what is the primary backup?

Correct! In GitOps, Git IS your backup. Reinstall ArgoCD, point it at the same repos, and everything rebuilds. You only need to separately back up credentials and ArgoCD-specific configuration.

2. How do you map Azure AD groups to ArgoCD roles?

Correct! The argocd-rbac-cm ConfigMap's policy.csv field maps Azure AD group IDs to ArgoCD roles using the g, "group-id", role:rolename syntax.

3. What does `argocd app get my-app --hard-refresh` do?

Correct! --hard-refresh invalidates the manifest cache, forcing ArgoCD to re-clone the repository and regenerate manifests from scratch. Useful when you suspect a cache issue.

Best Practices

Production Best Practices

Separate app code from config: Application source in one repo, K8s manifests in another
Use AppProjects: Never let teams use the default project in production
Enable self-heal and prune: True GitOps means the cluster always matches Git
Configure webhooks: Do not rely solely on 3-minute polling
Set resource limits: Add CPU/memory limits to ArgoCD components
Use SSO: Integrate with Azure AD -- no shared local accounts
Monitor metrics: Alert on sync failures, reconciliation duration, degraded apps

More Best Practices

Pin chart versions: Always use specific targetRevision for Helm charts, not *
Use ignoreDifferences: For HPA-managed replicas, mutating webhook fields
ApplicationSets for scale: Do not copy-paste Application YAML
Protect PVCs: Annotate with Prune=false to prevent accidental deletion
Test in lower environments first: Use progressive rollout strategies
Git branch strategy: Use main for prod, feature branches for dev
Limit auto-sync in prod: Consider manual sync for production with sync windows
Back up credentials: Repo and cluster credentials are the only state outside Git

ArgoCD Ecosystem

Argo Rollouts

Progressive delivery with canary and blue-green deployments. Integrates natively with ArgoCD for advanced deployment strategies.

Argo Workflows

Kubernetes-native workflow engine for CI/CD pipelines. Chain complex tasks as DAGs or sequential steps.

Argo Events

Event-driven automation. Trigger workflows or ArgoCD syncs from webhooks, message queues, or cloud events.

External Secrets Operator

Sync secrets from Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault into Kubernetes Secrets.

Summary

Module 05 Recap

App of Apps: Manage Applications with Applications for maximum flexibility
Multi-source: Combine Helm chart repos with Git values repos
Image Updater: Auto-promote image tags with semver, latest, or digest strategies
Notifications: Slack, Teams, webhooks via triggers, templates, and annotations
Custom health checks: Lua scripts for CRDs in argocd-cm ConfigMap
Disaster recovery: Git IS your backup; only back up credentials separately
SSO with Azure AD: OIDC config + RBAC group mappings
Monitoring: Prometheus metrics on all components

Course Complete

The Full ArgoCD Journey

01: Intro & GitOps

→

02: Apps & Projects

→

03: Sync & Hooks

04: AppSets & Multi-Cluster

→

05: Advanced Patterns

You now have the knowledge to build a production-grade GitOps platform with ArgoCD on AKS.

Training Complete

You are now equipped to run ArgoCD in production

Go forth and practice GitOps -- your clusters will thank you

Advanced Patterns & Operations

From Beginner to Production-Ready

App of Apps Pattern

App of Apps: Structure

App of Apps: Child App

App of Apps vs ApplicationSets

App of Apps

ApplicationSets

App of Apps

1. In the App of Apps pattern, what does the root Application's source path contain?

2. When should you choose App of Apps over ApplicationSets?

3. Can App of Apps and ApplicationSets be combined?

Multi-Source Applications

Multi-Source: Why?

ArgoCD Image Updater

Image Updater: Configuration

Image Updater: Update Strategies

semver

latest

digest

name

Multi-Source & Image Updater

1. In a multi-source Application, how do you reference a values file from a second source?

2. What does the ArgoCD Image Updater "semver" strategy do?

3. What are the two write-back methods for Image Updater?

ArgoCD Notifications

Notifications: Microsoft Teams

Subscribing to Notifications

Custom Health Checks (Lua)

Custom Health: Common CRDs

Notifications & Health

1. How does an Application subscribe to ArgoCD notifications?

2. What language are ArgoCD custom health checks written in?

3. Where are custom health checks configured?

Backup & Restore

DR: Git is Your Backup

SSO with Azure AD

Azure AD: RBAC Mapping

Monitoring ArgoCD

Troubleshooting Guide

Essential Debug Commands

Operations Mastery

1. In a GitOps disaster recovery scenario, what is the primary backup?

2. How do you map Azure AD groups to ArgoCD roles?

3. What does argocd app get my-app --hard-refresh do?

Production Best Practices

More Best Practices

ArgoCD Ecosystem

Argo Rollouts

Argo Workflows

Argo Events

External Secrets Operator

Module 05 Recap

The Full ArgoCD Journey

Training Complete

3. What does `argocd app get my-app --hard-refresh` do?