GitOps with ArgoCD: Declarative Kubernetes Deployments

Introduction

ArgoCD is the most widely adopted GitOps tool for Kubernetes, and for good reason. It provides a declarative approach to continuous deployment where your Git repository is the single source of truth for your application's desired state. When you push a change to Git—updating an image tag, modifying a ConfigMap, adjusting resource limits—ArgoCD detects the change and automatically (or manually, your choice) applies it to your cluster. When someone makes an unauthorized change to the cluster through kubectl, ArgoCD detects the drift and restores the declared state.

What sets ArgoCD apart from other GitOps tools is its combination of a powerful sync engine with an intuitive web UI that visualizes application state, deployment history, and resource health. Operations teams can see at a glance which applications are healthy, which are out of sync, and what changed in the last deployment. Developers can self-service their deployments by creating Application resources. Security teams can audit every change through Git history. This shared visibility across teams makes ArgoCD not just a deployment tool but a collaboration platform for Kubernetes operations.

This guide provides a comprehensive, hands-on walkthrough of setting up ArgoCD for production Kubernetes deployments. We cover installation and configuration, defining applications with sync policies, implementing the App of Apps pattern for managing multiple services, configuring RBAC and SSO, managing multi-cluster deployments, and implementing notifications and monitoring. By the end, you will have a production-ready ArgoCD setup that handles your entire Kubernetes deployment lifecycle.

Understanding ArgoCD: Core Concepts

ArgoCD operates on a simple but powerful model: you define Application resources that describe where your manifests live (Git repository and path) and where they should be deployed (cluster and namespace). ArgoCD then synchronizes the actual state with the desired state.

Application Resources

An Application is the fundamental ArgoCD resource. It connects a source (Git repo, Helm repo, or directory) to a destination (cluster and namespace) and defines sync behavior.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-api
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/my-api/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production

Projects for Multi-Tenancy

Projects provide multi-tenancy in ArgoCD. They restrict which repositories, clusters, and namespaces a team can deploy to.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  description: "Team A project"
  sourceRepos:
    - "https://github.com/team-a/*"
    - "https://charts.team-a.io/*"
  destinations:
    - server: https://kubernetes.default.svc
      namespace: "team-a-*"
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace
  namespaceResourceBlacklist:
    - group: ""
      kind: Secret  # Prevent creating secrets via manifests
  roles:
    - name: developer
      description: "Developer access"
      policies:
        - p, proj:team-a:developer, applications, get, team-a/*, allow
        - p, proj:team-a:developer, applications, sync, team-a/*, allow
      groups:
        - team-a-devs

Sync Waves and Phases

Sync waves control the order in which resources are applied during a sync operation. Resources are organized into phases (pre-sync, sync, post-sync) and waves within each phase. This ensures dependencies are created before dependent resources.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"  # Applied before wave 2

Architecture and Design Patterns

The App of Apps Pattern

The App of Apps pattern is ArgoCD's solution for managing multiple applications from a single entry point. A parent Application points to a directory containing child Application manifests. When you add a new Application YAML to that directory, ArgoCD automatically deploys the new application.

# Parent application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-apps
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/infrastructure.git
    targetRevision: main
    path: clusters/production/applications
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The ApplicationSet Pattern

ApplicationSet is a Kubernetes controller that generates ArgoCD Applications dynamically from templates. It supports multiple generators: Git directory, Git file, cluster list, and matrix combinations.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-services
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/infrastructure.git
        revision: main
        directories:
          - path: apps/*
  template:
    metadata:
      name: "{{path.basename}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/infrastructure.git
        targetRevision: main
        path: "{{path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{path.basename}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Step-by-Step Implementation

Production Installation

# Install ArgoCD with HA configuration
kubectl create namespace argocd
 
# Apply the HA installation manifest
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml
 
# Wait for all pods to be ready
kubectl -n argocd rollout status deployment/argocd-server
 
# Configure ArgoCD CLI
argocd login argocd.example.com --sso
 
# Set admin password
argocd account update-password

Configuring SSO

# argocd-cm ConfigMap for OIDC configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
data:
  url: https://argocd.example.com
  oidc.config: |
    name: GitHub
    issuer: https://github.com/login/oauth
    clientID: $argocd-secret:oidc.github.clientID
    clientSecret: $argocd-secret:oidc.github.clientSecret
    requestedScopes:
      - openid
      - profile
      - email

RBAC Configuration

# argocd-rbac-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Admins can do everything
    p, role:admin, applications, *, */*, allow
    p, role:admin, clusters, *, *, allow
    p, role:admin, repositories, *, *, allow
    p, role:admin, projects, *, *, allow
 
    # Developers can sync and view their team's apps
    p, role:dev, applications, get, team-a/*, allow
    p, role:dev, applications, sync, team-a/*, allow
    p, role:dev, applications, action/*, team-a/*, allow
 
    # Map GitHub teams to roles
    g, myorg:platform-team, role:admin
    g, myorg:team-a, role:dev

Notifications Configuration

# Trigger definition for sync failures
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Failed', 'Error']
      send: [slack-notification]
 
  template.slack-notification: |
    slack:
      attachments: |
        [{
          "color": "#E96D76",
          "title": "ArgoCD Sync Failed",
          "fields": [{
            "title": "Application",
            "value": "{{app.metadata.name}}",
            "short": true
          }, {
            "title": "Status",
            "value": "{{app.status.sync.status}}",
            "short": true
          }],
          "text": "Application {{app.metadata.name}} sync failed at {{app.status.operationState.finishedAt}}"
        }]
 
  service.slack: |
    token: $slack-token

Real-World Use Cases

Use Case 1: Microservices Deployment

Managing dozens of microservices with ArgoCD requires automation. The ApplicationSet pattern generates Applications from a directory structure.

# For each directory in apps/, create an Application
# apps/api-gateway/manifests/
# apps/user-service/manifests/
# apps/order-service/manifests/
# apps/payment-service/manifests/
 
# ApplicationSet automatically creates:
# Application: api-gateway
# Application: user-service
# Application: order-service
# Application: payment-service

Use Case 2: Canary Deployment with Argo Rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 5
  strategy:
    canary:
      canaryService: api-server-canary
      stableService: api-server-stable
      trafficRouting:
        nginx:
          stableIngress: api-server-ingress
      steps:
        - setWeight: 10
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: api-server-canary
        - setWeight: 30
        - pause: { duration: 10m }
        - setWeight: 60
        - pause: { duration: 10m }
        - setWeight: 100

Use Case 3: Blue-Green Deployment

For zero-downtime deployments with instant rollback capability.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-server
spec:
  strategy:
    blueGreen:
      activeService: api-server-active
      previewService: api-server-preview
      autoPromotionEnabled: false
      prePromotionAnalysis:
        templates:
          - templateName: smoke-tests

Use Case 4: Secret Management with External Secrets Operator

Managing secrets in GitOps without storing them in Git by integrating with cloud secret managers.

Best Practices for Production

Enable HA mode for production: Deploy ArgoCD in HA mode with multiple replicas of the server, controller, and repo server to ensure availability.
Use Projects for multi-tenancy: Create AppProjects for each team or service group, restricting source repositories and destination namespaces.
Set sync windows for production: Use sync windows to restrict when automated syncs can occur—for example, block production deploys outside business hours.
Enable notifications: Configure Slack, email, or PagerDuty notifications for sync failures, health degradation, and drift detection.
Implement progressive sync policies: Use manual sync for production and automated sync for staging and development environments.
Use resource exclusions for system resources: Exclude frequently-changing resources (like HPA status) from sync to reduce noise.
Monitor ArgoCD itself: Expose ArgoCD metrics to Prometheus and create dashboards for sync latency, application health, and resource usage.
Regular backup of ArgoCD state: While Git is the source of truth, ArgoCD's own state (projects, RBAC, cluster credentials) should be backed up.

Common Pitfalls and Solutions

Pitfall	Impact	Solution
Storing credentials in manifests	Security breach	Use External Secrets Operator or SOPS
Automated sync on production	Risky untested deployments reaching prod	Use manual sync or sync windows for production
No health checks on applications	Deploying broken apps that appear healthy	Define liveness, readiness, and startup probes
Ignoring sync failures	Silent deployment failures	Set up notifications for all failure states
Single repo for all services	Slow sync, merge conflicts	Use ApplicationSet with Git directory generator
Not using resource hooks	Database migrations run before pods start	Use PreSync hooks for migrations

Performance Optimization

# ArgoCD server resources for large installations
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
  namespace: argocd
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: argocd-server
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1000m"
              memory: "1Gi"
 
# Controller settings for large numbers of applications
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
data:
  controller.operation.processors: "25"
  controller.status.processors: "25"
  controller.self.heal.timeout.seconds: "5"
  controller.repo.server.timeout.seconds: "60"

Comparison with Alternatives

Feature	ArgoCD	Flux v2	Jenkins X	Spinnaker
Web UI	Rich built-in	External (Weave GitOps)	Limited	Rich
Sync model	Pull (poll + webhook)	Pull (continuous reconcile)	Push + Pull	Push
Multi-cluster	Native	Namespace-based	Multi-cluster	Multi-cloud
Progressive delivery	Argo Rollouts	Flagger	Built-in	Built-in
RBAC	Projects + policies	K8s RBAC	K8s RBAC	Roles + permissions
SSO	OIDC, SAML, LDAP, GitHub	N/A	OIDC	OIDC, SAML, LDAP
Notification	Notification controller	Alert/Provider CRDs	Webhooks	Echo service
Learning curve	Moderate	Moderate	Moderate	Steep

Advanced Patterns

Resource Hooks for Database Migrations

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  namespace: production
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: ghcr.io/myorg/api:v2.1.0
          command: ["npm", "run", "migrate"]
      restartPolicy: Never
  backoffLimit: 0

Custom Health Checks

# argocd-cm ConfigMap: custom health check for CRDs
resource.customizations.health.argoproj.io_Rollout: |
  hs = {}
  if obj.status ~= nil then
    if obj.status.currentStepIndex == #obj.spec.strategy.canary.steps then
      hs.status = "Healthy"
    else
      hs.status = "Progressing"
    end
  end
  return hs

Ignore Differences for Auto-Computed Fields

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # Ignore HPA-managed replica count
    - group: ""
      kind: Service
      jqPathExpressions:
        - .spec.clusterIP  # Ignore auto-assigned ClusterIP

Testing Strategies

# Validate application manifests
argocd app get my-api --refresh
argocd app diff my-api
 
# Test sync locally before applying
argocd app sync my-api --dry-run
 
# Validate with kustomize
kustomize build overlays/production/ | kubeval --strict
 
# Test ApplicationSet rendering
argocd appset get my-appset -o yaml

Notifications and Monitoring

ArgoCD's notification controller sends alerts when application state changes. Configure it to notify your team via Slack, email, or webhooks:

# argocd-notifications-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  service.slack: |
    token: $slack-token
    signingSecret: $slack-signing-secret
  
  template.app-sync-succeeded: |
    message: |
      ✅ Application {{.app.metadata.name}} sync succeeded.
      Revision: {{.app.status.sync.revision}}
      {{range .app.status.operationState.syncResult.resources}}
      {{.kind}}/{{.name}} - {{.status}}
      {{end}}
  
  template.app-health-degraded: |
    message: |
      🔴 Application {{.app.metadata.name}} health is Degraded!
      {{range .app.status.resources}}
      {{.kind}}/{{.name}} - Health: {{.health.status}}
      {{end}}
  
  trigger.on-sync-succeeded: |
    - when: app.status.operationState.phase in ['Succeeded']
      send: [app-sync-succeeded]
  
  trigger.on-health-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [app-health-degraded]

# Subscribe to notifications via Application annotations
metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: deployments
    notifications.argoproj.io/subscribe.on-health-degraded.slack: alerts

Automated Image Updates

ArgoCD Image Updater watches container registries and automatically updates image tags in your Git repository when new versions are published:

# Install ArgoCD Image Updater
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml

# Configure image update strategy via annotations
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: api=ghcr.io/myorg/api
    argocd-image-updater.argoproj.io/api.update-strategy: semver
    argocd-image-updater.argoproj.io/api.allow-tags: regexp:^[0-9]+\.[0-9]+\.[0-9]+$
    argocd-image-updater.argoproj.io/write-back-method: git:secret:git-creds

This configuration watches for new semver-compatible tags of ghcr.io/myorg/api and automatically updates the image reference in your Git repository. The update triggers ArgoCD's normal sync cycle, providing end-to-end automation from image build to deployment.

Multi-Cluster Deployment

ArgoCD can deploy to multiple clusters from a single control plane. Register remote clusters and target them in Application resources:

# Register a remote cluster
argocd cluster add remote-cluster-context --name production-us-east
 
# List registered clusters
argocd cluster list

# Application targeting a remote cluster
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-us-east
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/api/overlays/us-east
  destination:
    server: https://us-east.k8s.example.com
    namespace: production

Combined with ApplicationSet's cluster generator, you can deploy the same application to all registered clusters with per-cluster configuration overrides:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: global-api
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: production
  template:
    metadata:
      name: "api-{{name}}"
    spec:
      project: production
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: apps/api/overlays/{{metadata.region}}
      destination:
        server: "{{server}}"
        namespace: production

Future Outlook

ArgoCD continues to evolve with improved multi-cluster management, better ApplicationSet generators, enhanced notification capabilities, and deeper integration with progressive delivery tools like Argo Rollouts and Flagger. The Argo project (now a CNCF graduated project) is expanding with Argo Events for event-driven automation and Argo Workflows for pipeline orchestration, creating a comprehensive GitOps-native platform for cloud-native operations. As the ecosystem matures, expect tighter integration with policy engines like OPA/Gatekeeper and improved support for Helm and Kustomize workflows.

ArgoCD's declarative approach to continuous delivery ensures that your cluster state always matches your Git repository, providing auditability and rollback capabilities. The CNCF graduation of the Argo project signals long-term stability and community investment, making it a safe choice for enterprise adoption.

Disaster Recovery with ArgoCD

ArgoCD simplifies disaster recovery because your entire cluster state is declaratively defined in Git. To restore a cluster:

Provision a new cluster
Install ArgoCD on the new cluster
Point ArgoCD at your existing Git repository
ArgoCD synchronizes all applications to the new cluster automatically

This process works because ArgoCD's App of Apps pattern defines every resource declaratively. There is no imperative state to lose. Combined with Velero for persistent volume backups, you can achieve full disaster recovery with minimal downtime and zero manual intervention.

# Backup ArgoCD itself
kubectl get applications -n argocd -o yaml > argocd-apps-backup.yaml
 
# Restore on new cluster
kubectl apply -f argocd-apps-backup.yaml

Conclusion

ArgoCD brings the power of GitOps to Kubernetes deployments with a combination of declarative configuration, continuous reconciliation, and intuitive visualization. By defining your desired state in Git and letting ArgoCD handle the synchronization, you gain automated deployments, drift detection, easy rollbacks, and full auditability. The App of Apps and ApplicationSet patterns scale this approach to hundreds of applications across multiple clusters and environments.

The key takeaways are: start with the App of Apps pattern for managing multiple applications, use Projects for multi-tenancy and RBAC, configure sync waves for ordered deployments, implement notifications for operational awareness, and use resource hooks for pre/post deployment tasks. With ArgoCD, your Git repository becomes your operations team's single source of truth, and every deployment becomes a Git commit that can be reviewed, approved, and rolled back. For organizations with multiple clusters, ApplicationSet combined with the cluster generator provides a scalable approach to global deployments with per-region customization.

Minh Vo

Slaying code & making it lit fr fr 🔥 tagline