I have been bit late to Kubernetes world, but ever since I have started using it, I have been part of a great teams building awesome applications and using Helm package manager. With Helm, you package your Kubernetes application as charts, which are then stored in Helm chart repo. Helm also has a templating engine allowing you to set values in your charts dynamically allowing you to manage your applications more easily. Azure Container Registry (ACR) currently supports publishing Helm 3 charts to ACR and it is currently in preview.
In this post we will see how we can publish a sample Helm chart to ACR and also deploy the application to Azure Kubernetes Service (AKS) by consuming the published chart from ACR. We will also use ACR’s repository scoped tokens - a preview feature which offer great benefits.<!--more-->
We will cover following things in this post.
1) Prerequisites
- ACR
- Setting up the ACR to use repository scoped tokens
- Create scope-maps
- Create a token
- AKS
- Create AKS instance on Azure
For CI and CD, I am going to use Azure DevOps YAML multi-stage pipelines in this post and our pipeline will be divided as below.
2) CI Stage
- Get the latest chart from GitHub
- Publish to ACR
3) CD Stage
- Pull the latest version of chart from ACR
- Deploy the pulled to AKS
- View the deployed service in Azure DevOps
1. Prerequisites
You can authenticate with ACR in multiple ways.
Using a service principal - You can create separate service principals. This gives access to whole ACR and all the repositories inside that ACR instance. More info here
Using managed identity - Using user-assigned or system-assigned managed identity for easily consuming charts on an Azure VM. More info here
Using image pull secret - For your Kubernetes (K8s) cluster (including unmanaged AKS instance), you can also define a image pull secret, which then lets K8s cluster do pull images and run the application. More info on how you can do this with AKS is here
Using a repository scoped token - This feature of ACR is currently in preview and only available on Premium container registry service tier. This allows you to define scope maps and repository specific tokens for your ACR.
Setting up the ACR to use repository scoped tokens
I am going to use Tokens to authenticate ourselves with ACR. The biggest advantage is that, as a registry owner, you can have a single instance of ACR across your organisation and provide access to certain teams only selected repositories. There are other interesting scenarios highlighted in the documentation, mainly…
- Allow IoT devices with individual tokens to pull an image from a repository
- Provide an external organisation with permissions to a specific repository
- Limit repository access to different user groups in your organisation. For example, provide write and read access to developers who build images that target specific repositories, and read access to teams that deploy from those repositories.
If you have an existing ACR instance (which in my case I do), the first step is to upgrade your ACR instance to use Premium
tier.
Then select Premium
and click Save
.
Creating scope maps
Once, you are on Premium tier, you will be able to to create a tokens. But before we can create tokens, let us define two scope maps. Scope maps when associated with tokens sets permissions on what can be done on ACR.
-
chartpull - a scope limiting read permission to
helmdemo/vote-app
repo. I am usingcontent/read
andmetadata/read
permissions. -
chartpush - a scope limiting write permission inside
helmdemo/vote-app
repo. I am usingcontent/write
andcontent/read
permissions.
The idea here is that, development team pushes the charts to ACR and some other team is consuming the charts/application. But you can extend this idea to let users external to your organisation to pull images only from specific repositories.
So to create scope map, go to Scope maps
under Repository permissions
sections and then click + Add
. In my case, I am limiting these scopes only to helmdemo
repo.
I follow the same process to create another scope named chartpull
scope with content/read
, metadata/read
scopes.
- Notice that in
chartpush
scope we also allowcontent/read
permission. This is because, you needcontent/read
scope along withcontent/write
scope if you are going to push charts to ACR (for the repo defined in the scope).- You can also use Azure CLI to do the same. Use
az acr token
command. Refer the documentation for more information.
Creating token
Once you defined scopes, you are ready to create tokens. Go to Tokens
in Repository permissions
section and click + Add
. Then name the token (e.g. helmdemopull
) and select the scope map you created above. In my case I am creating two tokens, one for pull and another one for push.
You can repeat the process to create another token. Refreshing the screen you will see something like this below.
You will notice that these tokens (helmdemopull
and helmdemopush
) do not yet have passwords generated. Lets create passwords. Click on the token, click the icon under Actions
column for either password1
or password2
. You will also have an option to set an expiration date for the password if you planning to allow token to be valid only for specific days.
2. CI Stage
Get the latest chart from GitHub
To get the latest source from the GitHub repo, in our YAML pipeline we add a resources
section and point to the repo on GitHub. We will also need to pass the endpoint
parameter with the name of the GitHub service connection. More info about how to do this is here and here.
resources:
repositories:
- repository: helmrepo
type: github
name: utkarshpoc/azure-vote-helm-chart
endpoint: github
Publish chart to ACR
Before we write steps to publish to ACR, we need to ensure we store the tokens we generated in the pipeline as variables. Here I am adding tokens as variables using the variables UI of the pipeline and marking passwords as secret.
You can also define these in Azure Pipelines Library if you intend to use these tokens in multiple pipelines.
I also have few static and non-secret variables in YAML itself so that they are source controlled using variables
as below.
variables:
acr.name: acrdemoutkarsh
acr.repo.name: helmdemo/vote-app
The steps involved for publishing our Helm chart are simple.
- Installing Helm 3 on the agent
- Login to the ACR using Helm
- Save and push the chart
1. Installing Helm 3 on the agent
You can do this using OOB HelmInstaller
task. The YAML is as below. Here I am using latest
for helmVersion
, but you can stick specific to a version (3.x and above).
- task: HelmInstaller@0
displayName: install helm
inputs:
helmVersion: 'latest'
installKubectl: false
2. Login to the ACR using Helm
We can use simple script
step to login to the registry.
- script: |
helm registry login $(acr.name).azurecr.io --username $(acr.push.username) --password $(acr.push.password)
displayName: login to acr using helm
But if you run this, at the time of writing you will get an error as below
Error: this feature has been marked as experimental and is not enabled by default.
Please set HELM_EXPERIMENTAL_OCI=1 in your environment to use this feature
As the error states, this is because, publishing Helm charts which follow Open Container Initiative (OCI) standard is experimental and you are required to set HELM_EXPERIMENTAL_OCI
to 1
. We can do this by updating our variables
section in the YAML. The updated yaml looks as below.
variables:
acr.name: acrdemoutkarsh
acr.repo.name: helmdemo/vote-app
HELM_EXPERIMENTAL_OCI: 1
3. Save and push the chart to ACR
Next step is to save this chart locally and create an alias for the chart with our ACR registry URL. So our YAML looks as below.
- script: |
helm chart save $(build.sourcesdirectory)/src/azure-vote-helm-chart/ $(acr.name).azurecr.io/$(acr.repo.name):latest
displayName: save the chart and set the alias
And finally, we push the Helm chart to ACR using helm chart push
command.
- script: |
helm chart push $(acr.name).azurecr.io/$(acr.repo.name):latest
displayName: push the chart to acr
So our CI stage is now complete.
2. CD Stage
Pull the latest version of chart from ACR
The first step is to pull the latest version of the chart (one with tag latest
) to our agent machine and extract it in a folder. We can do that in pipeline as below. Notice I have added a new stage name cd
and its dependent of stage ci
.
- stage: cd
displayName: CD
dependsOn: ci
jobs:
- deployment: helm_publish_aks
displayName: deploy to aks
environment:
name: PROD
resourceName: helmdemo
resourceType: Kubernetes
strategy:
runOnce:
deploy:
steps:
- task: HelmInstaller@0
displayName: install helm
inputs:
helmVersion: 'latest'
installKubectl: false
- script: |
echo "$(acr.pull.password)" | helm registry login $(acr.name).azurecr.io --username $(acr.pull.username) --password-stdin
displayName: login to acr using helm
- bash: |
helm chart pull $(acr.name).azurecr.io/$(acr.repo.name):latest
displayName: get helm chart on agent
- bash: |
helm chart export $(acr.name).azurecr.io/$(acr.repo.name):latest --destination $(build.stagingdirectory)
displayName: export the chart to folder
Because each job in Azure DevOps run in a separate agent, I have to ensure agent has Helm tool, so install Helm tool again in the first step. Also, note that I am using token with scope-map permission set only to pull charts.
Deploy the chart to AKS
Now that the latest chart is pulled from our ACR and is available on agent, only step remaining is to deploy to AKS. We can do that using HelmDeploy
task.
- task: HelmDeploy@0
displayName: deploy chart to aks
inputs:
connectionType: 'Azure Resource Manager'
azureSubscription: '$(azure.service.connection)'
azureResourceGroup: 'demos'
kubernetesCluster: 'aksdemoutkarsh'
namespace: 'helmdemo'
command: 'upgrade'
chartType: 'FilePath'
chartPath: '$(build.stagingdirectory)/azure-vote/'
releaseName: 'helmdemo'
arguments: '--create-namespace --install'
View the deployed service in Azure DevOps
If you have added your Kubernetes cluster as a resource in Azure DevOps environment, you can now view deployed services directly from Azure DevOps.
You can also view deployed services in YAML and also see any ports exposed.
Browsing the exposed port, you will see application.
Conclusion
That is it! If you have read this far, thank you 🙏🏼. As we saw, with the support of tokens and OCI artifacts, ACR is one of the best in class container registry. Also, with deep integration with AKS (features like browsing services, logs, yaml), Azure DevOps brings in lots of productivity for your teams. Hope you found this post informative. If so, please share and tweet!
Top comments (2)
Hi, how are you? I have a question, when I make a second deployment AKS does not detect a new image and deletes the pod to be created with the new image automatically, I have to delete the pod manually. How did you solve this?
Great article, would check it out. I just started playing around with helm.