Have you ever had to manually update your container images, only to forget to do it or make a mistake? Automatic image updates can help you avoid these problems and ensure that your applications are always running the latest and most secure images. Manual image updates can be time-consuming and error-prone, especially if you have a large number of containerized applications.
In this blog post, we’ll explore the advantages of automatic image updates and explain how to implement them in your environment using GitHub Actions. Moreover, we’ll cover the process of temporarily pausing these updates to ensure application stability during incidents or any unforeseen issues.
So, let’s deep dive into automatic image updates to Git using Flux and GitHub Actions!
In the world of application management, GitOps fills a crucial need by simplifying and enhancing how we handle applications. But why do we need it? Managing applications can be complex and prone to errors. To address this, GitOps establishes Git as the central hub for handling configuration and infrastructure changes, allowing for streamlined, efficient, and reliable management. It speeds up deployment, ensures consistent and error-free setups, and reduces operational costs.
Moreover, it’s important to note that GitOps isn’t just about managing applications; it also intersects with the concept of auto-image updates. These updates involve automatically refreshing the underlying components of your application by regularly updating to the latest images pushed to the container registry, ensuring your application stays in sync with the most up-to-date software and dependencies.
GitOps streamlines software deployment by automating build, test, and deployment processes triggered by Git commits. This enhances reliability through a review and approval system, reducing errors and downtime. Collaboration is promoted through Git’s features for change tracking and code reviews, with seamless integration of image updates.
Here are several challenges you may face when dealing with image updates in the absence of GitOps:
GitOps using auto-image updates can help to solve all of these pain points by providing a declarative and automated approach to application deployment and management. With GitOps, the manifest is automatically updated when a new image is released, and a new build is automatically triggered. This ensures that deployments are consistent and timely across all environments.
Here are several advantages of using GitOps for image updates:
In this section, we will explore two primary methods for automating image updates to keep your applications up-to-date and secure.
A script can be configured to run on a schedule using a cron job or when certain events occur, such as when a new commit is pushed to the Git repository.
Here are a few pros and cons of the scripted approach:
Pros of scripted approach:
Cons of scripted approach:
In the previous section, we saw that scripting can be complex and time-consuming. That’s where dedicated tools come in handy. In this section, we will explore how GitOps tools make automating image updates easier. We will specifically look at the two most popular tools, Argo CD and Flux, to understand how they help with automatic image updates.
The Argo CD image updater is a tool that can help you update your Kubernetes workloads to the latest versions of their container images. It does this by setting appropriate application parameters for your Argo CD applications.
To use the image updater, you annotate your Argo CD application resources with a list of images to be considered for update, along with a version constraint to restrict the maximum allowed new version for each image. The image updater will then periodically check for new versions of the images and update them if necessary.
The image updater can be used to update images in a variety of ways, including:
Image automation in Flux can be used to automate the following tasks:
Image automation controllers
In Flux CD, the image automation controllers, which include the Image Reflector Controller
and Image Automation Controller
, are responsible for maintaining the synchronization between the image metadata in Kubernetes and the latest image metadata from the registry. The image reflector controller achieves this by regularly scanning the registry for changes to image metadata. When a change is detected, they update the Kubernetes resources that reference the affected images accordingly.
Moreover, the image automation controllers are capable of triggering image updates based on changes to the image metadata. For instance, you can configure automation to automatically update the image tags in your Kubernetes deployments to the latest stable versions of your images.
For more details about the implementation and design of the image automation controllers in Flux CD, refer to the official documentation on image reflector and automation Controllers.
Let us explore the pros and cons of employing GitOps for automating image updates, gaining insights into its efficiency and potential challenges.
Pros of GitOps approach:
Cons of GitOps approach:
Let’s understand how Flux employs two controllers, i.e. image automation controller and image reflector controller, to achieve image update automation.
These controllers, together, collaborate to update a Git repository whenever new container image tags become available.
It is composed of two custom resources: Image Repository and Image Policy. Let us understand how they operate together.
Image Repository
resource and configure the scanning interval.The image automation controller includes a custom resource called Image Update Automation
.
Image Update
resource clones the Git repository, updates YAML files based on the latest images from the image reflector controller, and commits changes to the specified Git repository.
After cloning, the image automation controller identifies and updates the deployment YAML manifest, using comments to mark fields for updating. The automation process checks the image policy in the comment and updates the field value accordingly.
Once updated, the image update automation controller commits and pushes changes to the specified branch. The source controller pulls the updated manifest, and the Kustomization controller applies the changes. This is how the Flux image automation controller streamlines the process of automating image version updates.
We will explore a live demonstration of a Flux GitOps configuration that seamlessly manages a workflow from staging to production, showcasing the automatic deployment process when making changes to source code. Additionally, we’ll highlight its integration with GitHub Actions to enhance the efficiency of the entire deployment pipeline.
In staging, developers make changes to application code, triggering a new build. The new image tag is then propagated to the Kubernetes manifest and finally deployed to the staging environment without requiring any approval in between.
In the case of production, a new PR is raised with an updated Kubernetes manifest, requiring approval for merge in the config repository. Only after approval is given does it get deployed to the production environment.
Ensure you have following things setup:
Let’s take a look at two Git repositories we will be using.
Beginning with an empty cluster, our initial task is to bootstrap Flux itself. Flux serves as the foundation upon which we’ll bootstrap all other components.
In the following sections, we’ll take a detailed look at each file within the apps directory, one by one.
├── apps
│ ├── git-repo.yaml
│ ├── image-auto-prod.yaml
│ ├── image-auto-staging.yaml
│ ├── image-policy-prod.yaml
│ ├── image-policy-staging.yaml
│ ├── image-repo.yaml
│ ├── kustomization-prod.yaml
│ └── kustomization-staging.yaml
This instructs Flux on how to interact with the Git repository where your application’s source code resides. It contains the following configuration:
ref:
branch: main
secretRef:
name: ssh-credentials
url: ssh://git@github.com/infracloudio/flux-helloenv-app
ref:
Specifies the Git branch to watch for changes, in this case, main.
secretRef:
Refers to a Kubernetes Secret named ssh-credentials, which likely contains SSH keys for secure Git access.
The ssh-credentials
is a secret that needs to be created.
url:
Indicates the URL of the Git repository, Change this to url: ssh://git@github.com/<your_github_username>/flux-helloenv-app
once you fork it.
This file is responsible for scanning the Docker image registry and fetching image tags based on the defined policy. Here’s the configuration:
image: docker.io/shapai/helloenv
interval: 1m0s
image:
Specifies the Docker image repository (for e.g. docker.io/shapai/helloenv) to scan for image tags. Change this to have your relevant Docker image repository, where the CI job will build and push the image. This same registry needs to be updated in your forked CI file, where the CI build will push images.
interval:
Sets the interval at which Flux will scan the image repository (every 1 minute in this case) and fetch image tags according to the defined policy.
Next, we will go through the image policy files image-policy-staging.yaml and image-policy-prod.yaml
This file defines the image tagging policy for the staging environment. Here’s the configuration:
filterTags:
extract: $ts
pattern: ^main-[a-f0-9]+-(?P<ts>[0-9]+)
imageRepositoryRef:
name: helloenv
policy:
numerical:
order: asc
filterTags:
This section specifies how to filter image tags. It extracts the timestamp ($ts) from tags that match the specified pattern. Tags are filtered in ascending order based on this timestamp, ensuring that Flux fetches the latest built image for the staging environment.
This file defines the image tagging policy for the production environment:
imageRepositoryRef:
name: helloenv
policy:
semver:
range: '>=1.0.0'
imageRepositoryRef:
Refers to the image repository named helloenv.
policy:
Defines a Semantic Versioning (SemVer) policy that specifies a range for acceptable image tags (in this case, any version greater than or equal to 1.0.0).
These image policies are crucial in ensuring that Flux deploys the correct images to the respective environments (staging and production) based on the defined criteria.
These YAML files define Kustomization resources for managing Kubernetes resources in both staging and production environments within the flux-system namespace. They are configured to synchronize with a specified Git repository, allowing for automated deployment and management of Kubernetes resources.
Now, let’s go through the ImageUpdateAutomation files.
This YAML file configures an ImageUpdateAutomation object to update the image tags in the ./demo/kustomize/staging
directory of the flux-helloenv-app Git repository. It will scan the repo every 1 minute (specified by the interval field).
The git
section specifies the branch to checkout and the commit message template. The sourceRef section specifies the Git repository containing the Kubernetes manifests to update.
The update
section specifies the path to the Kubernetes manifests to update and the strategy to use.
When this ImageUpdateAutomation object is deployed, Flux will periodically check for new image updates. If it finds any new updates, it will update the image tags in the Kubernetes manifests and commit the changes to the Git repository.
spec:
git:
checkout:
ref:
branch: main
commit:
author:
email: fluxbot@users.noreply.github.com
name: fluxbot
messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
interval: 1m0s
sourceRef:
kind: GitRepository
name: flux-helloenv-app
update:
path: ./demo/kustomize/staging
strategy: Setters
This YAML file is very similar to the previous one but has an additional push configuration. This means that after updating the image tags in the Git repository, Flux will commit and push the changes to the flux-image-update branch. This is done specially for prod setup since we are not going to push directly to the main branch for prod manifests.
This is useful in our case, as this will help us create a PR to the main branch from the flux-image-update branch, which will then go through manual approval. The process of creating PR is handled by the CI section in GitHub Actions.
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: helloenv-prod
namespace: flux-system
spec:
git:
checkout:
ref:
branch: main
commit:
author:
email: fluxbot@users.noreply.github.com
name: fluxbot
messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
push:
branch: flux-image-update
interval: 1m0s
sourceRef:
kind: GitRepository
name: flux-helloenv-app
update:
path: ./demo/kustomize/prod
strategy: Setters
Now that we’ve seen all the configurations, we can proceed to the bootstrap command.
Note the --owner
and --repository
switches here: we are explicitly looking for the ${GITHUB_USER}/flux-gitops-helloenv
repo. Make sure you fork both repos under your user and follow on.
flux bootstrap github \
--components-extra=image-reflector-controller,image-automation-controller \
--owner=$GITHUB_USER \
--repository=flux-gitops-helloenv \
--path=./clusters/my-cluster/ \
--branch=main \
--read-write-key \
--personal --private=false
flux bootstrap
commandLet’s understand what the above command does.
To enable the Flux image automation feature, the extra components can be specified with the --components-extra
flag. We are enabling the image reflector and automation controllers.
--path=./clusters/my-cluster/
specifies the path within the repository where the configuration files will be stored.
my-cluster
folder in the repository contains two files infrastructure.yaml
and apps.yaml
.
infrastructure.yaml
defines a Kustomization resource called ingress-nginx. This Kustomization lives in the flux-system namespace, doesn’t depend on anything, and has kustomize files at infrastructure/ingress-nginx.
apps.yaml
, which defines the flux-helloenv-app
application itself. Within this file, you’ll find references to the apps directory. This directory is a central location containing Kustomizations and configuration settings for setting up the flux-helloenv-app
application in both the prod and staging environments. Additionally, it includes configurations for image automation for helloenv
app in staging and prod env.
The process of bootstrapping everything may take some time. To monitor the progress and ensure everything is proceeding as expected, we can utilize the --watch
switch.
flux get kustomizations --watch
Verify that all Flux pods are in running state by running get pods.
$ kubectl -n flux-system get pods
NAME READY STATUS RESTARTS AGE
image-automation-controller-6c4fb698d4-zrp78 1/1 Running 0 29s
image-reflector-controller-5dfa39212d-hnnvj 1/1 Running 0 29s
kustomize-controller-424f5ab2a2-u2hwb 1/1 Running 0 29s
source-controller-2wc41z892-axkr1 1/1 Running 0 29s
Since our flux-helloenv-app repository is public, application will get deployed as part of the bootstrap step. You can check both staging and prod environments with following commands.
kubectl rollout status -n helloenv-staging deployments
watch kubectl get pods -n helloenv-staging
kubectl rollout status -n helloenv-prod deployments
watch kubectl get pods -n helloenv-prod
After bootstrapping Flux, we will grant it write access to our GitHub repositories. This will allow Flux to update image tags in manifests, create pull requests etc.
The flux create secret git
command creates an SSH key pair for the specified host and puts it into a named Kubernetes secret in Flux’s management namespace (by default flux-system). The command also outputs the public key, which should be added to the forked repo’s “Deploy keys” in GitHub.
GITHUB_USER=<your_github_username>
flux create secret git ssh-credentials \
--url=ssh://git@github.com/${GITHUB_USER}/flux-helloenv-app
If you need to retrieve the public key later, you can extract it from the secret as follows:
kubectl get secret ssh-credentials -n flux-system -ojson \
| jq -r '.data."identity.pub"' | base64 -d
Use the public key as a Deploy key in your fork of the flux-helloenv-app repo. Browse to the following URL, replacing <your_github_username>
with your GitHub username: https://github.com/<your_github_username>/flux-helloenv-app/settings/keys
.
The page will appear as follows.
Click “Add deploy key” and paste the key data (starts with ssh-<alg>...
) into the contents. The name is arbitrary, we use flux-helloenv-app-secret here.
The default image version can be checked by accessing our application using curl command. If you are using minikube, make sure you enable the ingress addon so the ingress functionality works.
minikube addons enable ingress
If you are running on the local cluster, you will have to add the minikube IP to your /etc/hosts
file. The command minikube ip
will give the IP of your cluster.
For example, the entry in /etc/hosts
file will look like this:
192.168.49.2 helloenv.prod.com helloenv.stage.com
You can check the ingress that is available for stage and prod.
$ kubectl get ingress --all-namespaces
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
helloenv-prod helloenv <none> helloenv.prod.com 80 9m22s
helloenv-staging helloenv <none> helloenv.stage.com 80 9m21s
Now, you can access the application with following commands.
curl helloenv.prod.com
curl helloenv.stage.com
The result of these curl
commands will show the current default image versions that the deployment is using.
Now, let’s make some changes to the app and release it to the stage. We can make a minor change to the app.py file in the application code repository flux-helloenv-app
, by changing the message and pushing it to the main branch, of course, in real case it will be pushed to main through a PR.
This change triggers the CI job configured on the application code repository. This job contains the logic to check the tag and then create the image tag based on it.
run: |
if [[ $ =~ ^refs/tags/[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "IMAGE_ID=$" >> "$GITHUB_OUTPUT"
else
ts=$(date +%s)
branch=${GITHUB_REF##*/}
echo "IMAGE_ID=${branch}-${GITHUB_SHA::8}-${ts}" >> "$GITHUB_OUTPUT"
fi
From this snippet, it can be seen that it checks for the condition for prod and stage, as well as how we set the tagging of images.
So, since we are doing this for stage, it will tag the image as ${branch}-${GITHUB_SHA::8}-${ts}
.
A useful tag format is <branch>-<sha1>-<timestamp>
.
Including the branch information with an image makes it easier to trace the source code’s branch and commit associated with that image. Additionally, having the branch information and unix time allows you to filter for images originating from a specific branch when needed.
Since this CI stage for Staging will push the image to the image repository, the image update and image policy will kick in and replace the image with the latest built image based on timestamp.
We can check the status of image automation and image policy for staging.
$ flux get image all
NAME LAST SCAN SUSPENDED READY MESSAGE
imagerepository/helloenv 2023-10-17T12:19:26Z False True successful scan: found 9 tags
NAME LATEST IMAGE READY MESSAGE
imagepolicy/helloenv-prod docker.io/shapai/helloenv:1.0.2 True Latest image tag for 'docker.io/shapai/helloenv' resolved to 1.0.2
imagepolicy/helloenv-staging docker.io/shapai/helloenv:main-b470560d-1696668188 True Latest image tag for 'docker.io/shapai/helloenv' resolved to Build-b470560d-1696668188
NAME LAST RUN SUSPENDED READY MESSAGE
imageupdateautomation/helloenv-prod 2023-10-17T12:18:41Z False True no updates made
imageupdateautomation/helloenv-staging 2023-10-17T12:18:39Z False True no updates made; last commit f798e64 at 2023-10-07T09:25:32Z
Let’s curl the stage domain and see the output:
$ curl helloenv.stage.com
This is staging environment with (Version: main-b470560d-1696668188)
The output will give you the latest version, which you can verify with commit id and timestamp with which the image was created.
Now, assuming we have tested the release in stage and got go-ahead, we are ready to release in production by tagging the release.
We will be releasing it in prod by tagging the release.
git tag -a 1.0.2 -m "prod modified"
git push --tags
This will, in turn, trigger the CI and create an image tag with that Git tag that we pushed. Once this is triggered and pushed, the image automation in Flux will commit and push the changes to the flux-image-update branch.
This will then create a PR using workflow in flux-helloenv-app repo. This PR needs a manual review and approval since it is a prod environment. Once this gets approved, it changes the tag with the latest tag in prod kustomization.
curl to the prod DNS should give you the latest tag as the version.
$ curl helloenv.prod.com
This is production environment (Version: 1.0.2)
During an incident, you may want to halt Flux from updating images in your Git repository. You can accomplish this by suspending image automation either in-cluster or by editing the ImageUpdateAutomation manifest in Git.
You can suspend image automation directly in a cluster using the following command:
flux suspend image update helloenv-prod
Alternatively, you can suspend image automation by editing the ImageUpdateAutomation manifest in Git. Here’s an example of the manifest:
kind: ImageUpdateAutomation
metadata:
name: helloenv-prod
namespace: flux-system
spec:
suspend: true
Once the incident is resolved, you can resume the automation using the following command:
flux resume image update helloenv-prod
If you want to pause automation for a particular image only, you can suspend and resume image scanning for that specific image. For example:
flux suspend image repository helloenv
Assuming you’ve configured Flux to update an application to its latest stable version and an incident occurs, you can instruct Flux to revert to a previous image version.
For instance, to revert from version 1.0.1 to 1.0.0, you can use the following command:
flux create image policy helloenv-prod --image-ref=helloenv --select-semver=1.0.0
You can also make this change by editing the ImagePolicy manifest in Git. Here’s an example of the manifest:
kind: ImagePolicy
metadata:
name: helloenv-prod
namespace: flux-system
spec:
policy:
semver:
range: 1.0.0
When a new version, e.g., 1.0.2, becomes available, you can update the policy again to consider only versions greater than 1.0.1. This can be achieved using the following command:
flux create image policy helloenv-prod --image-ref=helloenv --select-semver=">1.0.1"
This change will prompt Flux to update the podinfo deployment manifest in Git and roll out the specified image version in-cluster.
Flux’s image automation and GitOps are powerful solutions for managing container image updates. By combining image automation and image reflector controllers, organizations can automate image version updates in their Git repositories. This not only simplifies the process but also ensures consistency and reliability in application deployment.
The practical example illustrated how GitOps with Flux can streamline the workflow from staging to production, providing a structured approach to managing deployments as code changes occur. This approach enhances efficiency and reliability in the development pipeline, making it a valuable asset in modern DevOps practices.
Thank you for taking the time to read our post. We hope you found it both informative and engaging. We highly value your feedback and would love to hear your thoughts on this topic. Let’s kickstart a meaningful conversation on LinkedIn to exchange ideas and insights.
If you’re seeking assistance in crafting a robust DevOps strategy or considering outsourcing your DevOps operations to seasoned experts, we invite you to discover why numerous startups and enterprises regard us as one of the top-tier DevOps consulting and services companies.