ArgoCD

GitOps Continuous Delivery for Kubernetes — Deep Study

GitOps Kubernetes CD Pipeline Declarative Delivery Multi-Cluster CNCF Graduated

What is ArgoCD?

ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. It watches Git repositories and automatically syncs the desired state defined in Git to the actual state running in your cluster. It is a CNCF Graduated project and one of the most widely adopted CD tools in the cloud-native ecosystem.

Core Idea: Git is the single source of truth. Everything — configs, Helm values, Kustomize overlays — lives in Git. ArgoCD continuously reconciles the cluster state to match Git.

GitOps Principles (OpenGitOps)

📁

Declarative

Desired state described declaratively in YAML/JSON in Git

🔢

Versioned

Git history is the audit trail; rollback = git revert

🤖

Automated

Software agents apply desired state automatically

🔄

Continuously Reconciled

Agents detect and correct drift from desired state

🎯

Push vs Pull

ArgoCD is Pull-based

Traditional CI/CD pushes to clusters. ArgoCD agents pull from Git inside the cluster — no external access needed, no exposed cluster credentials.

👁️

Continuous Reconciliation

3-minute default polling + webhooks

ArgoCD polls Git every 3 minutes by default. GitHub/GitLab webhooks trigger immediate syncs. Drift is detected and corrected automatically.

🌐

Multi-Cluster

One control plane, many targets

A single ArgoCD instance can deploy to dozens of clusters. Clusters are registered by kubeconfig credentials stored as Kubernetes Secrets.

📦

Source Types

Helm, Kustomize, plain YAML, Jsonnet
  • Helm charts (v2/v3)
  • Kustomize overlays
  • Plain directory of YAML
  • Jsonnet
  • Config Management Plugins (CMP)

Architecture

  ┌─────────────────────────────────────────────────────────────────────┐
  │                        ArgoCD Control Plane                         │
  │                                                                     │
  │   ┌─────────────┐   ┌─────────────┐   ┌───────────────────────┐   │
  │   │ API Server  │   │ Repo Server │   │ Application Controller │   │
  │   │ gRPC / REST │   │ Git clone   │   │ Kubernetes controller  │   │
  │   │ Web UI      │   │ render tmpl │   │ reconciliation loop    │   │
  │   │ CLI gateway │   │ Helm/Kust   │   │ sync / health check    │   │
  │   └──────┬──────┘   └──────┬──────┘   └──────────┬────────────┘   │
  │          │                 │                       │                │
  │   ┌──────▼──────────────────────────────────────────────────────┐  │
  │   │                     Redis Cache                             │  │
  │   │          (app state, repo cache, cluster cache)             │  │
  │   └─────────────────────────────────────────────────────────────┘  │
  │                                                                     │
  │   ┌─────────────┐   ┌─────────────┐   ┌───────────────────────┐   │
  │   │ Dex         │   │ Notification│   │ ApplicationSet Ctrl   │   │
  │   │ OIDC/SSO    │   │ Controller  │   │ generates Applications│   │
  │   └─────────────┘   └─────────────┘   └───────────────────────┘   │
  └─────────────────────────────────────────────────────────────────────┘
           │                                          │
           ▼ kubectl / kubeconfig                     ▼ watches Git repos
  ┌─────────────────────┐                   ┌────────────────────┐
  │  Target Cluster(s)   │                   │  Git Repository     │
  │  Deployments, Svcs  │                   │  Helm / Kustomize  │
  │  CRDs, ConfigMaps   │                   │  plain YAML        │
  └─────────────────────┘                   └────────────────────┘

Core Components

🌐

API Server

  • Exposes gRPC + REST API
  • Serves the Web UI
  • CLI (argocd) gateway
  • Auth/RBAC enforcement
  • Webhook receiver
📂

Repo Server

  • Clones Git repos (caches locally)
  • Renders Helm templates
  • Runs Kustomize builds
  • Runs Config Management Plugins
  • Stateless — horizontally scalable
🔄

Application Controller

  • Core reconciliation loop
  • Compares live vs desired state
  • Triggers syncs
  • Runs health checks
  • Sharded for scale
🔐

Dex

  • Built-in OIDC provider
  • SSO bridge (GitHub, LDAP, SAML)
  • Optional — can use external OIDC
⚡

Redis

Caches app state, repo content, cluster state. Required for HA. All components share it.

📋

ApplicationSet Controller

  • Generates Application CRDs from templates
  • Multi-cluster, multi-env automation
  • Replaces manual App creation

Core Concepts

Application CRD — The Core Unit

Every deployment is an Application custom resource. It binds a Git source to a Kubernetes destination.

apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app namespace: argocd # always in ArgoCD namespace spec: project: default # AppProject for RBAC scoping source: repoURL: https://github.com/org/repo targetRevision: HEAD # branch, tag, or commit SHA path: k8s/overlays/production # path in repo (Kustomize or plain YAML) # OR for Helm: # chart: my-chart # helm: # releaseName: my-app # valueFiles: [values-prod.yaml] destination: server: https://kubernetes.default.svc # in-cluster namespace: production syncPolicy: automated: prune: true # delete resources removed from Git selfHeal: true # revert manual changes to cluster syncOptions: - CreateNamespace=true - PrunePropagationPolicy=foreground - ApplyOutOfSyncOnly=true retry: limit: 5 backoff: duration: 5s factor: 2 maxDuration: 3m

AppProject — Isolation & RBAC Scoping

AppProject controls which Git repos, clusters, and namespaces a set of Applications can use. Essential for multi-team setups.

apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: team-payments namespace: argocd spec: description: Payments team apps sourceRepos: # allowed Git repos - https://github.com/org/payments-* destinations: # allowed clusters + namespaces - server: https://prod-cluster namespace: payments-* clusterResourceWhitelist: # can deploy cluster-scoped resources - group: '*' kind: Namespace namespaceResourceBlacklist: # cannot create these - group: '' kind: ResourceQuota roles: - name: developer policies: - p, proj:team-payments:developer, applications, sync, team-payments/*, allow - p, proj:team-payments:developer, applications, get, team-payments/*, allow

Sync & Health Status

Every Application has two independent status dimensions. Both must be understood together.

Sync Status

Does the live cluster state match the desired Git state?

StatusMeaning
SyncedLive matches Git exactly
OutOfSyncDrift detected — needs sync
UnknownCannot determine (permission, network)

Health Status

Is the deployed application actually healthy?

StatusMeaning
HealthyAll resources ready and serving
ProgressingDeployment rolling out
DegradedPod CrashLoopBackOff, etc.
SuspendedCronJob, HPA paused
MissingResource not found in cluster
Key insight: An app can be Synced but Degraded (Git matches cluster but pods are crashing). Or OutOfSync but Healthy (manual hotfix applied). You need both green.

Sync Strategies

👆

Manual Sync

Default — operator-triggered

ArgoCD detects drift but does not sync automatically. Operator triggers via UI, CLI, or API. Best for production environments requiring human approval.

argocd app sync my-app
🤖

Automated Sync

Continuous reconciliation

ArgoCD automatically syncs when it detects changes. Combine with prune and selfHeal for full GitOps.

automated: prune: true selfHeal: true
✂️

Prune

Delete orphaned resources

When a resource is removed from Git, ArgoCD deletes it from the cluster. Disabled by default — must be explicitly enabled. Use Prune=false annotation to protect specific resources.

🔁

Self Heal

Revert manual changes

When someone manually edits a cluster resource (via kubectl), ArgoCD detects the drift within minutes and reverts it back to the Git state. Enforces true GitOps.

Sync Options (per Application)

OptionEffect
CreateNamespace=trueCreates the destination namespace if it doesn't exist
ApplyOutOfSyncOnly=trueOnly applies resources that have drifted (faster)
PrunePropagationPolicy=foregroundWait for child resources to be deleted before parent
Replace=trueUse kubectl replace instead of apply (for immutable fields)
ServerSideApply=trueUse server-side apply (recommended for large resources)
SkipDryRunOnMissingResource=trueSkip dry-run for CRDs not yet installed
Validate=falseDisable kubectl schema validation
RespectIgnoreDifferences=trueHonor ignoreDifferences during live state comparison

Resource Hooks

Hooks are Kubernetes Jobs annotated to run at specific phases of the sync lifecycle. Use them for database migrations, smoke tests, notifications, and cleanup.

PreSync → Sync begins → Sync → Resources applied → PostSync ↘ on failure → SyncFail | skip resource → Skip
# Database migration Job that runs BEFORE sync apiVersion: batch/v1 kind: Job metadata: name: db-migrate annotations: argocd.argoproj.io/hook: PreSync argocd.argoproj.io/hook-delete-policy: HookSucceeded spec: template: spec: containers: - name: migrate image: myapp:latest command: ["python", "manage.py", "migrate"] restartPolicy: Never

Hook Delete Policies

PolicyWhen Deleted
HookSucceededAfter successful completion
HookFailedAfter failure (for debugging)
BeforeHookCreationBefore next hook run (default)

Common Hook Use Cases

  • PreSync: DB migrations, schema checks
  • Sync: Apply resources in specific order
  • PostSync: Smoke tests, cache warm-up
  • SyncFail: Alert team, rollback notification
  • Skip: Exclude resource from sync

App of Apps Pattern

A "root" Application points to a directory of other Application manifests. This enables bootstrapping an entire environment by syncing one app. The root app creates child Applications, which in turn deploy workloads. 

  Git Repo
  ├── apps/                        ← Root App points here
  │   ├── api-service.yaml         ← Application CRD
  │   ├── web-frontend.yaml        ← Application CRD
  │   ├── postgres.yaml            ← Application CRD
  │   └── redis.yaml               ← Application CRD
  │
  └── services/
      ├── api-service/             ← Actual K8s manifests
      ├── web-frontend/
      ├── postgres/
      └── redis/

  Root App syncs → creates 4 child Applications
  Each child App syncs → deploys its service
# Root Application (the "App of Apps") apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: root-app namespace: argocd spec: source: repoURL: https://github.com/org/gitops-repo targetRevision: HEAD path: apps/production # directory with Application YAMLs destination: server: https://kubernetes.default.svc namespace: argocd # child Applications live here syncPolicy: automated: prune: true selfHeal: true
Tip: App of Apps is great for small-to-medium setups. For large-scale multi-cluster/multi-env deployments, prefer ApplicationSet — it is more DRY and avoids manual YAML duplication.

ApplicationSet

ApplicationSet is a higher-level CRD that automatically generates Application objects from a template and one or more generators. It eliminates repetitive Application YAML for multi-env or multi-cluster deployments.

Generators

List Generator

Static list of key-value pairs. Generates one App per item.

generator: list

Cluster Generator

One App per registered cluster. Filter by cluster labels.

generator: clusters

Git Generator

One App per directory or file found in a Git repo path.

generator: git

Matrix Generator

Cartesian product of two generators (e.g. clusters × environments).

generator: matrix

Merge Generator

Merges multiple generator outputs with override logic.

generator: merge

SCM Provider

One App per repo in a GitHub Org, GitLab Group, or Bitbucket.

generator: scmProvider

Pull Request

One App per open PR — preview environments per PR.

generator: pullRequest

Cluster Decision Resource

Integrates with Open Cluster Management (OCM) for dynamic cluster selection.

generator: clusterDecisionResource

Example: Git Directory Generator

apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: services namespace: argocd spec: generators: - git: repoURL: https://github.com/org/gitops-repo revision: HEAD directories: - path: services/* # one App per directory template: metadata: name: '{{path.basename}}' # e.g. "api-service" spec: project: default source: repoURL: https://github.com/org/gitops-repo targetRevision: HEAD path: '{{path}}' # the matched directory destination: server: https://kubernetes.default.svc namespace: '{{path.basename}}' syncPolicy: automated: prune: true selfHeal: true

Example: Matrix Generator (clusters × environments)

generators: - matrix: generators: - clusters: selector: matchLabels: region: asia - list: elements: - env: staging - env: production # Generates: asia-cluster-1-staging, asia-cluster-1-production, # asia-cluster-2-staging, asia-cluster-2-production, ...

ApplicationSet Policies

syncPolicy.preserveResourcesOnDeletion: When set to true, deleting the ApplicationSet does NOT delete the child Applications or their resources. Safe for production.
goTemplate: Use goTemplate: true to unlock Go templating syntax (range, if, index) instead of the limited {{param}} substitution.

Multi-Cluster Deployment

ArgoCD manages multiple target clusters from a single control plane. Clusters are registered as Kubernetes Secrets in the argocd namespace.

1

Register Cluster via CLI

argocd cluster add my-prod-cluster --name production — stores kubeconfig as Secret with label argocd.argoproj.io/secret-type: cluster

2

In-Cluster vs External

The cluster ArgoCD runs on uses https://kubernetes.default.svc. External clusters use their API server URL. Each gets a ServiceAccount with appropriate RBAC.

3

Target Cluster in Application

destination.server: https://prod-cluster-api:6443 or use destination.name: production (the cluster alias).

4

ApplicationSet + Cluster Generator

Use the cluster generator to automatically create one Application per registered cluster. Label clusters (e.g. env=prod) and filter by label.

Hub-Spoke Model: For very large setups (100+ clusters), consider running a management cluster with ArgoCD as the hub, deploying workload Applications to spoke clusters. Each spoke needs minimal RBAC — no ArgoCD installation required.

RBAC & Multi-Tenancy

Built-in Roles

RolePermissions
role:readonlyRead everything, sync nothing
role:adminFull access to all apps/projects

Custom roles defined in argocd-rbac-cm ConfigMap using Casbin policies.

Policy Format (Casbin)

# p, subject, resource, action, object p, role:staging-dev, applications, sync, staging/*, allow p, role:staging-dev, applications, get, staging/*, allow p, role:staging-dev, applications, create, staging/*, deny # Assign SSO group to role g, github-org:engineering, role:staging-dev

Multi-Tenancy Strategies

🏢

Namespace Isolation

Each team gets their own namespace. AppProject restricts which namespaces a team's apps can deploy to. Simple and effective.

☸️

Cluster Isolation

Each team or environment gets a dedicated cluster. AppProject restricts which cluster(s) are allowed. Maximum isolation — recommended for production vs dev.

🏗️

Separate ArgoCD

Each team runs their own ArgoCD instance. Maximum autonomy. Higher operational cost. Common at large enterprise scale.

Secrets Management

Never store plaintext secrets in Git. ArgoCD itself doesn't encrypt secrets — you need an external solution. The three most common patterns:

🔒

Sealed Secrets

Bitnami — encrypt for Git
  • CLI encrypts Secret → SealedSecret CRD
  • Sealed Secrets controller decrypts in cluster
  • Encrypted blob safe to store in Git
  • Simple, no external dependency
  • Hard to rotate the cluster key
kubeseal --format yaml < secret.yaml > sealed.yaml
🗝️

External Secrets Operator

ESO — sync from vault/cloud
  • Syncs secrets FROM an external store TO Kubernetes Secrets
  • Supports: AWS SM, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, 1Password
  • Secret values never stored in Git
  • Automatic refresh interval
🏦

Vault + ArgoCD Vault Plugin

AVP — inline replacement
  • Placeholder in YAML: <path:secret/data/db#password>
  • ArgoCD Vault Plugin fetches from Vault during sync
  • Works with any Vault-compatible store
  • Requires CMP configuration
🔐

SOPS

Mozilla — file-level encryption
  • Encrypts YAML/JSON files using age, GPG, KMS
  • Encrypted file stored in Git
  • Decrypted at render time via CMP
  • Works well with Helm values files

Notifications

Argo CD Notifications watches Application events and sends messages to services like Slack, Microsoft Teams, PagerDuty, email, and more.

Triggers (when to notify)

  • on-sync-succeeded
  • on-sync-failed
  • on-health-degraded
  • on-deployed
  • on-sync-running
  • Custom triggers with expr conditions

Services (where to send)

Slack MS Teams PagerDuty OpsGenie Email Webhook Telegram GitHub GitLab
# Subscribe an Application to notifications metadata: annotations: notifications.argoproj.io/subscribe.on-sync-failed.slack: deployments-alerts notifications.argoproj.io/subscribe.on-health-degraded.slack: on-call notifications.argoproj.io/subscribe.on-deployed.webhook: ci-webhook

Argo Rollouts — Progressive Delivery

Argo Rollouts is a separate Kubernetes controller that extends Deployments with progressive delivery strategies: canary and blue/green. It integrates tightly with ArgoCD for GitOps-driven rollouts.

🐦

Canary

Gradual traffic shift
  • Route X% of traffic to new version
  • Pause, promote, or abort manually
  • Automated analysis with metrics
  • Integrates with Istio, Nginx, ALB
strategy: canary: steps: - setWeight: 10 - pause: duration: 5m - setWeight: 50 - pause: {} # manual promote - setWeight: 100
🔵

Blue/Green

Instant traffic switch
  • Run old (blue) and new (green) in parallel
  • Switch traffic instantly when ready
  • Instant rollback (switch back)
  • No gradual traffic shifting
strategy: blueGreen: activeService: my-app-active previewService: my-app-preview autoPromotionEnabled: false
📊

Analysis

Automated promotion gate
  • Query Prometheus, Datadog, CloudWatch
  • Pass/fail threshold determines auto-promote or rollback
  • Custom metrics (error rate, latency p99, business KPIs)
analysis: templates: - templateName: error-rate args: - name: service-name value: my-app
Rollout CRD replaces Deployment: Replace your kind: Deployment with kind: Rollout (same spec structure). ArgoCD detects Rollout resources and shows rollout status in the UI.

ArgoCD vs Flux v2

Aspect ArgoCD Flux v2
UI Rich built-in web UI, app graph visualization No built-in UI (use Weave GitOps or Flux UI add-on)
Model Application CRD — centralized Kustomization + HelmRelease — distributed (per-cluster)
Multi-Cluster Hub-spoke: one ArgoCD manages many clusters Each cluster runs its own Flux controllers
Pull Request Envs ApplicationSet PullRequest generator Flux ImageUpdateAutomation + Notification Controller
Secrets Plugin-based (ESO, AVP, Sealed Secrets) Native SOPS support; ESO integration
Progressive Delivery Argo Rollouts (separate) Flagger (separate)
RBAC Casbin policies, SSO/OIDC via Dex Kubernetes-native RBAC only
Helm Support First-class (Helm chart as source) First-class (HelmRelease CRD)
Learning Curve Moderate — rich feature set Moderate — more Kubernetes-native, less GUI
Best For Teams wanting visibility, UI, multi-cluster hub Teams wanting minimal footprint, decentralized gitops

Best Practices

📁

Separate App & Config Repos

Keep application source code in one repo and Kubernetes manifests/Helm values in a separate GitOps repo. CI updates the GitOps repo; ArgoCD watches it.

🌿

Branch per Environment

Use main for production, staging for staging. Or use a single branch with environment-specific directories. Avoid long-lived feature branches in the GitOps repo.

🔄

Enable Self-Heal

Always enable selfHeal: true in production. This ensures any manual kubectl changes are reverted, enforcing Git as the true source of truth.

🏗️

Use AppProject Per Team

Give each team their own AppProject to scope which repos, clusters, and namespaces they can use. Prevents accidental cross-team interference.

🔐

Never Store Secrets in Git

Use Sealed Secrets, ESO, or SOPS. Audit your repo for committed secrets before adopting ArgoCD. Rotate any secrets that were exposed.

📊

Monitor Sync Status

Set up Notifications to alert on sync failures and health degradation. Export ArgoCD metrics to Prometheus and build Grafana dashboards for fleet health visibility.

🔗

Use ApplicationSet

Prefer ApplicationSet over manual App-of-Apps YAML for multi-env/multi-cluster deployments. It is more maintainable, DRY, and supports dynamic cluster registration.

🧪

PostSync Health Gates

Add PostSync hooks for smoke tests. Combine with Argo Rollouts Analysis for automated canary promotion. Block promotion if error rate exceeds threshold.

GitOps Repository Structure (Recommended)

gitops-repo/ ├── apps/ # App-of-Apps or ApplicationSets │ ├── production/ │ │ └── applicationset.yaml # generates prod Applications │ └── staging/ │ └── applicationset.yaml │ ├── services/ # per-service manifests │ ├── api-service/ │ │ ├── base/ # Kustomize base │ │ └── overlays/ │ │ ├── staging/ │ │ └── production/ │ └── web-frontend/ │ ├── Chart.yaml # Helm chart │ ├── values.yaml │ ├── values-staging.yaml │ └── values-production.yaml │ └── infrastructure/ # cluster-wide resources ├── cert-manager/ ├── ingress-nginx/ └── monitoring/