Internship at Nirmata on Kyverno(CNCF sanboxed project)...

About myself ....

Hello Everyone!! Myself Vivek Kumar Sahu, sophomore(at the time of writting this blog) in Electronics and Telecommunication from Jabalpur Engineering College, India. My field of interests are cloud based technology as well as Linux world. I started exploring all these technologies just after Lockdown started and continued learning Linux and cloud-based technologies. After learning wanted to apply my knowledge and skills through some projects. Then i came to know that best way to make use of your skills and knowledge is by contributing to some open source project or by working in a company. But, being a student open source would be good choice. From there open source took place in my mind. Seriously, if you are complete beginner then 1-2 months you have to roam around the projects, meetings, slack channels, etc. And once you have understanding about the projects then you have to roam around the code written in those projects. It's not easy in the beginning but later on it is.
After exploring different projects from CNCF, I came across a Sandboxed project known as Kyverno.

And it was my right decision to pick the Kyverno project for contribution. For newbie contributor this is one of the good project to start their contribution with it and the reason behind it:-
Firstly, the Kyverno policies written for Kubernetes resources are written in the same language(i.e. YAML) as the resource manifest written in the Kubernetes.
And secondly, easy to relate different concepts of Kubernetes while applying different types of policies for different resources. So, it looks quite a relatable project with my skills.

After one month of exploring this project I got an opportunity from Nirmata company (drives Kyverno Project) to work as an intern for 3 months in Kyverno Project.
The mentor assigned to me was shuting zhao and she is currently maintainer of the Kyverno project. She is really a calm, helpful and supportive in nature. She understand beginner contributor and gives time to settle down.

About Project

Kyverno is the policy engine for Kubernetes. In technical terms Kyverno is care-taker of Kubernetes in terms of security. If you want to understand Kyverno in more detail, will recommend to you go through these blogs:-

https://www.linkedin.com/pulse/kyverno-native-policy-manager-kubernetes-code-arun-singh/?trackingId=xlfXp3sLzav3R71xJ%2FtCEA%3D%3D

https://medium.com/techloop/understanding-kyverno-policies-7e2d8651d7b1

During internship, I had to work on feature enhancement and it was To extend the test command to support or handle mutate Policy and also
To cover all sample policies. The same reaction I also had earlier after reading this project headline. Didn't understand anything. So, the recommendation to newbie contributors that the best way to understand any project is to read the documentation first and then install that project in your local machine and try to use it, play with it until and unless you get the feel about that project.

Description of feature-enhacement

Let me explain about this enhancement feature which I had to work upon. The feature needed to be added in the Kyverno-CLI to extend the support of the test command to support or handle mutate policy. At that time test command used to support for validate policy but not for mutate and generate policy. So, it was like a extension feature for test command to also support for mutate policy.

Let me break the topic into 2 parts i.e.
What is test command, it's uses and how it works and
What is mutate policy and describe them separately.

What is a test and Why to use it and how does it work?

test is a command (in the Kyverno-cli) like other commands, which provides facility to test policies on resources before deploying policy directly to the cluster. To be more specific test command provides facility to the user to check or test the policies whether it's properly working on resources or not. Since, it's related to security of Kubernetes resource, so it is recommended to check policies before directly applying to live clusters.

But how does the test command make sure that whatever policy applied on resources is correctly applied or not ??
Good question. Here comes the actual use of test command. Basically test command internally work is compares the actual result which is generated from Kyverno engine with expected result provided by the user.
Note: that expected result provided by the user may be wrong but actual result obtained from engine can not be. And if you want to get the actual result then use apply command. To know more about apply command visit here.

Let's understand more detail how test command works ?
As the user will run below command,

$kyverno  test  <path of folders containing test.yaml file>

First of all, test command looks for it's configuration file i.e. test.yaml in the provided folder by the user.

Structure of test.yaml file

name: mytests
policies: (contains list of path of policies file)
  - <path/to/policy.yaml>
  - <path/to/policy.yaml>
Resources: (contains list of path of resource file)
  - <path/to/resource.yaml>
  - <path/to/resource.yaml>
variables: variables.yaml (optional)
results:
- policy: <name>
  rule: <name>
  resource: <name>
  kind: <name>
  patchedResource: <path/to/patchedResource.yaml> (path of patchedResource file) 
  status/result: <pass/fail/skip>

Path of Policy provided by the user under the policies section in the test.yaml file is fetched from it and similarly all resources are fetched from the path of resources provided by the user under the resources section in the test.yaml file. After the policies and resources are fetched from the respective paths, then the policy with the help of match/exclude block selects the resources one by one. If the policy doesn’t selects the resource then policy skip that resource, which means the rule of the policy won’t be applied to that resource.
But if policy selects the resource which means that the further rule of the policy will be applied to resources and
Lastly, irrespective of policy applied on resource or not applied, the Kyverno engine will generates a result which is known as an actual result or engine response.

And also on the other hand, user also need to provide the expected result or user-defined result under the results section of the test.yaml file. Expected result here means that what user thinks of the result could be after policy applied on resources. The common fields on whose basis comparison of actual result or engine response with expected result or user-defined result are done:-
policy name(name of the policy),
resource name(name of the resource),
rule name,
kind(type of resource),
patchedResource(path of patched resource or updated resource file,
namespace(optional)
result/status: (fail/pass/skip),

Pass ----> when policy selects the resource and rule applied on the resource + patchedResource obtained from the engine response must be equal to the patched Resource provided by the user
Fail ----> when policy selects the resource and rule applied on the resource + patchedResource obtained from the engine response is not equal to the patched Resource provided by the user
Skip ----> When policy doesn’t select the resource because the resource description doesn’t match with the match/exclude block of rule of the policy, therefore rule is not applied on the resource. So, policy skips the resource.

What is Mutate Policy ?

As the mutate names suggest mutating something or updating something. Here updation could be anything like removing of any field or addition of any field or replacing of any field on Kubernetes resources. And resources updated through mutation policy is known as patched Resource.

This was the overall design part of how the test command will work with mutate policy.

To know more about test command switch here
To know more about writting mutate policies switch here

Hands-on

Let's see the hand's on using test command for mutate policy.

For ClusterPolicy:-

Let's try to understand the field in the Policy first.
1) kind: ClusterPolicy --> which means selecting resources in all namespaces or cluster wide.
2) metadata:
name: add-labels ---> it is the name of Policy

3) spec:
rules:
- name: add-labels ---> name of the rule,
here coincidently, the name of policy is same as rule name. But don't think that they are related something like that. They are totally independent.
NOTE:- Under the rule section there can be one or more than one rules. But the type of policy i.e. validate/mutate/generate must be any among them throughout the policy.

Let's try to understand the policy through the below diagram.

The policy wants to say that selects all the resource of kind: Pod in whole cluster(i.e. in all namespaces). And after selecting resources add the label foo: bar to those resources.

policy.yaml

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-labels
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
    policies.kyverno.io/description: >-
      Labels are used as an important source of metadata describing objects in various ways
      or triggering other functionality. Labels are also a very basic concept and should be
      used throughout Kubernetes. This policy performs a simple mutation which adds a label
      `foo=bar` to Pods, Services, ConfigMaps, and Secrets.
spec:
  rules:
  - name: add-labels
    match:
      resources:
        kinds:
        - Pod
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            foo: bar

Below is the resource of kind: Pod. Which means it will be selected by the policy.

resource.yaml

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
spec:
  containers:
  - name: nginx
    image: nginx:latest

Below, is the patchedResource with added label foo: bar with it. After mutate rule is applied to the above resource, label foo: bar will be added to that resource, the resource gets updated because of that. And updated resource is known as a patchedResource.

patchedResource,yaml

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    foo: bar
spec:
  containers:
  - name: nginx
    image: nginx:latest

Below, is the configuration file of test command. It contains path of above policies, path of above resources, path of above patched resource and results defined by the user.

test.yaml

name: add-labels
policies:
  - add_labels.yaml
resources:
  - resource.yaml
results:
  - policy: add-labels
    rule: add-labels
    resource: myapp-pod
    patchedResource: patchedResource.yaml
    kind: Pod
    result: pass

Now run the below command:-
$ kyverno test <path_of_folder_containing_test.yaml_file>

Executing add-labels...
applying 1 policy to 1 resource...
│───│────────────│────────────│───────────│────────│
│ # │ POLICY     │ RULE       │ RESOURCE  │ RESULT │
│───│────────────│────────────│───────────│────────│
│ 1 │ add-labels │ add-labels │ myapp-pod │ Pass   │
│───│────────────│────────────│───────────│────────│
(base)

For Namespaced-policy ( Policy)

Namespaced-policy are applied only on particular namespace.
First of all, let's read out the policy. It says that select resource of kind: Pod from testing namespace and after selecting mutate or add the below field in it.
"dnsConfig:
options:
- name: ndots
value: "1"

Policy.yaml

apiVersion: kyverno.io/v1
kind: Policy
metadata:
  name: add-ndots
  namespace: testing
  annotations:
    policies.kyverno.io/title: Add ndots
    policies.kyverno.io/category: Sample
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      The ndots value controls where DNS lookups are first performed in a cluster
      and needs to be set to a lower value than the default of 5 in some cases.
      This policy mutates all Pods to add the ndots option with a value of 1.
spec:
  background: false
  rules:
  - name: add-ndots
    match:
      resources:
        kinds:
        - Pod
    mutate:
      patchStrategicMerge:
        spec:
          dnsConfig:
            options:
              - name: ndots
                value: "1"

resource.yaml

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    foo: bar
  namespace: testing
spec:
  containers:
  - name: nginx
    image: nginx:latest

Patched Resource with added
dnsConfig:
options:
- name: ndots
value: "1"
in the resource.

patchedResource.yaml

apiVersion: v1
kind: Pod
metadata:
  labels:
    foo: bar
  name: myapp-pod
  namespace: testing
spec:
  containers:
  - image: nginx:latest
    name: nginx
  dnsConfig:
    options:
    - name: ndots
      value: "1"

test.yaml

name: add-nodeselector
policies:
  - policy.yaml
resources:
  - resource.yaml
results:
  - policy: testing/add-ndots   
    rule: add-ndots
    resource: myapp-pod
    patchedResource: patchedResource.yaml
    namespace: testing
    kind: Pod
    result: pass 

$ kyverno test <path>

Executing add-nodeselector...
applying 2 policies to 1 resource...
│───│───────────────────│───────────│───────────────────────│────────│
│ # │ POLICY            │ RULE      │ RESOURCE              │ RESULT │
│───│───────────────────│───────────│───────────────────────│────────│
│ 1 │ testing/add-ndots │ add-ndots │ Pod/testing/myapp-pod │ Pass   │
│───│───────────────────│───────────│───────────────────────│────────│

NOTE:-
The difference in the test.yaml between the Namespaced policy and ClusterPolicy is in their name of policy under the results section. If it is a Namespaced policy then user need to provide the policy name as: <namespace>/<policy_name>

whereas if it is a ClusterPolicy then policy name as: .

The Namespaced policy is applied to resources present in that particular namespace. Whereas ClusterPolicy is applied in all namespaces present in whole cluster.

Challenges I faced during project :-

1) There are 2 types of policies:-
ClusterPolicy --> Applied on whole cluster
Policy --> Applied on a particular namespace, which means it will only select those resources having the same namespace as namespaced policy.
Initially the solution works fine for ClusterPolicy but it wasn't supporting namespaced Policy. So, for the same I have to understand the concept of namespaced policy and then apply separate checks to filter resources according to the namespace of the policy.

I think you all have got the basic idea about this enhancement feature for Kyverno-cli. And to know more about the Kyverno project go to kubectl get kyverno project
I would highly thanks to Jim Bugwadia for giving this opportunity. And once again highly thankful to my mentor Shuting zhao, for her continuous support and guidance throughout the project. And also special thanks to Vyankatesh, Pooja and Chip Zoller for their continous help. It was really amazing to work with this community. I don't know how these 3 months passed, it seems like it started only a few days ago. In these 3 months learned many new things from the community members.

That was all from mine side. Thanks for reading it. Hope you liked it. If you have any doubt regarding the Kyverno project join the slack channel.

Resources for beginners wanted to contribute to Kyverno project:-
Kyverno --> https://github.com/kyverno/kyverno
Kyverno policies ---> https://kyverno.io/policies/
Youtube channel ---> https://www.youtube.com/c/Nirmata/videos
golang resources --> https://gobyexample.com/ and https://zetcode.com/all/#go