As we adopt containers and Kubernetes, we end up running and maintaining multiple Kubernetes clusters. Some are used as production, some as QA while others are used as short-lived dev environments. There are many cases where we want to package and deploy applications to these clusters. The kind of applications we want to deploy may also vary a lot. On one hand, we have third-party off-the-shelf applications like Prometheus, Grafana, Elasticsearch, Kafka, etc. which have their own Helm charts. In many cases, these applications only need to be configured to our use case, pinned to a specific version and they can be deployed. While on the other hand, we have bespoke applications that are updated & deployed multiple times a week. Even multiple times a day in some cases.
In both cases, we have the problem of installing, updating and maintaining these applications on multiple clusters. With a handful of clusters, say three to ten, a CI pipeline (like Jenkins) could be set up to deploy to each of them using a template rendering tool of your choice (Kustomize, Helm, Ship , …). This approach works at a smaller scale. What if we have around a hundred clusters with at least a dozen apps each? In this blog post, we will leverage Gitops to tackle the problem of deploying an application and keeping it updated in one cluster as well as deploying multiple applications to multiple clusters.
GitOps is one of the solutions to this problem. Whatever templates or configs are defined in a git repository should match the resources running in the clusters. It also involves an agent that will fetch any new commits made to the git repository and apply them to the cluster. The simplest form of an agent can be a cronjob that runs every few minutes, clones the git repository, renders the templates and applies them to a target cluster.
There are several advantages of adopting GitOps. Listing a few:
ArgoCD is a declarative GitOps continuous delivery tool for Kubernetes. It periodically renders resource templates from a git repository and compares them against the current state present in the cluster. Synchronisation can be done manually through the GUI/CLI or by setting up a post-commit webhook. It may also be done automatically (every 3 min). A sync will apply the templates to the cluster. It tries to make sure that the current state of the resources in the cluster matches the templates generated by the git repository at the latest commit.
Let’s go ahead and create an ArgoCD application. To keep things simple, we’ll use a Helm chart that deploys nginx using a Deployment and a Service.
Setup the cluster. We’ll be using minikube to create our cluster.
# Start Minikube
minikube start --kubernetes-version="v1.15.7"
# Setup RBAC for helm/tiller
kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller \
--clusterrole cluster-admin \
--serviceaccount=kube-system:tiller
# Initialise helm and install tiller
helm init --service-account tiller
Run the following commands to install ArgoCD in the cluster using Helm
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install --name argocd --namespace argocd argo/argo-cd --wait
Once the pods are up, you can port forward to the argocd-server service to connect using the CLI and GUI. The initial admin password is the name of the argocd-server pod. Use the following command to print and copy it. We’ll need that in the next step to login to ArgoCD.
kubectl get pod -n argocd -l app.kubernetes.io/name=argocd-server
NAME READY STATUS RESTARTS AGE
argocd-server-5b56dcc4f9-xvgbs 1/1 Running 1 21h
After you know the admin password, you can port-forward and login using the “admin” user and the password from the previous step.
kubectl port-forward service/argocd-server -n argocd 8080:443
argocd login localhost:8080 --insecure
You can also access the GUI with the same credentials at localhost:8080 after accepting the self signed certificate. Let’s deploy a sample application using the GUI. Once you’ve logged in, you should see this page.
Click on “Create Application” and fill in these details. You can also click the “Edit as YAML” to look at the Application Custom Resource (CR) spec that will be generated by the GUI. This spec can also be edited directly to perform actions that the GUI does not support through the form.
General:
Name: test
Project: default
SyncPolicy : Manual
Source:
RepositoryURL: https://github.com/kanuahs/argocd-demo.git
Revision: HEAD
Path: nginx-chart
Destination:
Server: 'https://kubernetes.default.svc'
Namespace: default
Helm:
ValueFiles: values.yaml
This page will be visible once the Application is created. The Application health will be “Missing” and status will be “out of sync” as automatic synchronisation is disabled. Opening the Application will let you see a diff for each resource present in the app. Click on sync to deploy the application to the cluster.
Once you see the Application as healthy and synced, you can check if nginx is working by going to the nodeport IP. You will see the default nginx landing page. Use the following command to open the URL in the default browser.
minikube service nginx-service --url
Let’s simulate a commit by switching to another branch. Click on “test” to open the Application and the “App Details” to check the values.
Click on edit and TargetRevision to “nginxV2”. This is a branch in the git repository that has a commit that changes the landing page for nginx.
Click on “Sync” after making the changes. You will see a new ReplicaSet being created. Go to the URL where nginx is exposed and you will see an additional line that says “This was deployed using ArgoCD”. You can open the URL with the following command.
minikube service nginx-service --url
ArgoCD 1.3 and later version support directly fetching the chart from a Helm Chart repository and installing it into a cluster. Creating an Application from a Helm chart registry is similar to the steps listed above. However, let’s use the CLI to speed things up a bit.
argocd app create monitoring \
--repo https://kubernetes-charts.storage.googleapis.com \
--helm-chart prometheus \
--dest-server https://kubernetes.default.svc \
--dest-namespace default \
--revision 9.7.4
argocd app list # To view all applications
argocd app get monitoring # To view details about the monitoring Application we just created.
We will use the above command to create an ArgoCD Application called “monitoring” by using the Prometheus chart. To perform a sync we can use the following command:
argocd app sync monitoring
kubectl get pods -l release=monitoring
Run the following commands to delete the apps.
argocd app delete test
argocd app delete monitoring
So far, we have deployed a single application to a single cluster using ArgoCD. We’ll now use the ‘App of Apps’ pattern to deploy multiple apps to one cluster. However, before we jump into that, we’ll have to look at the Application CR that gets created when we use the GUI or CLI. In the nginx example, the following Application was generated from the form.
kind: Application
metadata:
name: test
namespace: argocd
spec:
destination:
namespace: default
server: https://kubernetes.default.svc
project: default
source:
helm:
valueFiles:
- values.yaml
path: nginx-chart
repoURL: https://github.com/kanuahs/argocd-demo.git
targetRevision: HEAD
We can create these Application CRs directly using Helm or kubectl or another Application. We will create an Application that deploys the Prometheus chart and the nginx sample chart. Use the following command to create an App of Apps:
argocd app create app-of-apps \
--repo https://github.com/kanuahs/argocd-demo.git \
--path app-of-apps-chart \
--dest-server https://kubernetes.default.svc \
--dest-namespace default \
--sync-policy automated \
--self-heal \
--auto-prune
We are also enabling auto sync. ArgoCD will check the state every 3 minutes and perform a sync if the Application is out of sync.
$ argocd app list
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
app-of-apps https://kubernetes.default.svc default default Synced Healthy Auto-Prune <none> https://github.com/kanuahs/argocd-demo.git app-of-apps-chart
minikube-monitoring https://kubernetes.default.svc default default Synced Healthy Auto-Prune <none> https://kubernetes-charts.storage.googleapis.com 9.7.4
nginx https://kubernetes.default.svc default default Synced Healthy Auto-Prune <none> https://github.com/kanuahs/argocd-demo.git nginx-chart HEAD
We’ll need to create another cluster for this demo. If you already have a Kubernetes cluster and a Kubeconfig, you can skip the cluster creation steps.
We’ll be using Google Kubernetes Engine (GKE) as an external cluster. Run the following command to create a cluster
gcloud config set project [PROJECT_ID]
gcloud container clusters create gke-argocd \
--zone us-central1-a \
--machine-type n1-standard-4 \
--num-nodes 1
Once the cluster is created and ready, run the following command to add it to your kubeconfig:
gcloud container clusters get-credentials gke-argocd --zone us-central1-a
We’ll have to setup RBAC and access credentials for ArgoCD to connect to the cluster that we just created. The CLI makes this easy. Run the following command:
$ argocd cluster add
ERRO[0000] Choose a context name from:
CURRENT NAME CLUSTER SERVER
* gke_[PROJECT_NAME]_us-central1-a_gke-argocd gke_[PROJECT_NAME]_us-central1-a_gke-argocd https://[KUBERNETES-MASTER-IP]
minikube minikube https://192.168.99.100:8443
$ argocd cluster add gke_[PROJECT_NAME]_us-central1-a_gke-argocd
INFO[0001] ServiceAccount "argocd-manager" created in namespace "kube-system"
INFO[0001] ClusterRole "argocd-manager-role" created
INFO[0001] ClusterRoleBinding "argocd-manager-role-binding" created, bound "argocd-manager" to "argocd-manager-role"
Cluster 'gke_[PROJECT_NAME]_us-central1-a_gke-argocd' added
This will let ArgoCD deploy apps to the GKE cluster that we just created. Fork this git repository and clone it to your development machine. We’ll have to make a few changes in the values.yaml file of the app-of-apps chart. The file is located at this path: “app-of-apps-chart/values.yaml”
# file: app-of-apps-chart/values.yaml
clusters:
- name: minikube
destinationServer: https://kubernetes.default.svc
apps:
prometheus:
enabled: true
version: "9.7.4"
sampleApp:
enabled: true
branch: HEAD
# - name: gke
# destinationServer: #INSERT MASTERIP HERE
# apps:
# prometheus:
# enabled: true
# version: "9.7.4"
# sampleApp:
# enabled: true
# branch: HEAD
Uncomment the GKE section in values.yaml and fill in the destinationServer field with the GKE cluster’s master IP address. You can use the following command to print it.
$ argocd cluster list
SERVER NAME VERSION STATUS MESSAGE
https://[KUBERNETES_MASTER_IP] gke_[PROJECT_NAME]_us-central1-a_gke-argocd 1.13+ Successful
https://kubernetes.default.svc 1.14 Successful
Once you’ve updated the values.yaml file, commit and push it to your fork. Now, we have to redeploy the Application with your fork as the source. You can run the following command to do that after filling in your GitHub username:
argocd app patch app-of-apps --patch '{"spec": { "source": { "repoURL": "https://github.com/[GITHUB_USERNAME]/argocd-demo.git" } }}' --type merge
ArgoCD should pick up the changes and auto-sync. You can check the status of the Applications with kubectl or with the following command:
$ argocd app list
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
app-of-apps https://kubernetes.default.svc default default Synced Healthy Auto-Prune <none> https://github.com/[GITHUB_USERNAME]/argocd-demo.git app-of-apps-chart
minikube-monitoring https://kubernetes.default.svc default default Synced Healthy Auto-Prune <none> https://kubernetes-charts.storage.googleapis.com 9.7.4
minikube-nginx https://kubernetes.default.svc default default Synced Healthy Auto-Prune <none> https://github.com/kanuahs/argocd-demo.git nginx-chart HEAD
gke-monitoring https://[KUBERNETES_MASTER_IP] default default Synced Healthy Auto-Prune <none> https://kubernetes-charts.storage.googleapis.com 9.7.4
gke-nginx https://[KUBERNETES_MASTER_IP] default default Synced Healthy Auto-Prune <none> https://github.com/kanuahs/argocd-demo.git nginx-chart HEAD
This command will clean up all the Applications and delete the resources created by the Application.
argocd app delete app-of-apps
We hope you found the Multicluster GitOps with ArgoCD blog informative and engaging. For more reads like this one, subscribe to our blogs for up-to-date news, projects, and related content in real-time.
Looking for help with GitOps adoption using ArgoCD? do check our Argo CD consulting capabilities and expertise to know how we can help with your GitOps adoption journey. If you’re looking for managed on-demand ArgoCD support, check our support model.