Writing yet another Kubernetes templating tool

If you are working with Kubernetes environment then you probably make use of several existing templating tools, some of them being a part of package managers such as Helm or Ksonnet, or just templating languages (Jinja2, Go template etc.). All of them have their own drawbacks as well as advantages and we are going to go through them and write our own tool that will try to combine the best features.

So, why not Helm?

There are a number of articles criticizing Helm (e.g. just one of them: Think twice before using Helm). The main issue with Helm is that it works with string representations and Kubernetes manifests are (json) objects. The real hell for a Helm chart developer begins when (s)he needs to calculate indents for a yaml manifest, sometimes it looks like this (it's a real example from my chart):

spec:
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: my-awesome-container
            resources:
{{ toYaml .Values.resources | indent 14 }}

But Helm today is de-facto the standard for Kubernetes applications packaging. The main advantage of Helm is large community and a big number of public repositories with charts. And recently Helm developers have announced a Helm hub. So Helm today is like Docker — it's not the only one but it has community and support.

There are promising changes coming with Helm 3 release but nobody knows when it could be.

To conclude, Helm advantages:

• Large community and a number of public charts
• (Relatively) human-friendly syntax. At least it's yaml + go template ;)

Drawbacks:

• Working with strings and not objects
• Limited number of operators and functions you can use

OK, then maybe Ksonnet?

If you are comparing Helm to Ksonnet the latter has a huge advantage, namely it works with objects. Ksonnet is a tool based on JSON templating language Jsonnet. Another cool feature about Ksonnet is that it has Kubernetes-API-compatible Jsonnet libraries that you can import into your template and work with Kubernetes objects like in any OOP language:

local k = import "k.libsonnet";
local deployment = k.apps.v1beta1.deployment;

local appDeployment = deployment
  .new(
    params.name,
    params.replicas,
    container
      .new(params.name, params.image)
      .withPorts(containerPort.new(targetPort)),
    labels);

Looks impressive, doesn't it?
It is a little less neat when you are working not with API objects but with just json objects imported from yaml/json file:

{
  global: {},
  components: {
    "deployment-nginx-deployment-dkecx"+: {
      spec+: {
        replicas: 10,
        template+: {
          spec+: {
            containers+: [
              {
                name: "nginx",
                image: "nginx:latest",
                ports: [
                  {
                    containerPort: 80,
                  },
                ],
              },
            ],
          },
        },
      },
    },
  },
}

But still it is something and it's better than working with strings in Helm. The disadvantage of Ksonnet is that it has smaller community and less packages than Helm (though you can import Helm charts into your Ksonnet project, but you will be working with them as json objects, not as jsonnet-library objects). And as a result of a smaller community and contribution there is lack of some features when you trying to write your own chart. One of them I experienced myself: you know that in Helm you can build up a ConfigMap from a directory containing a number of config files this way:

apiVersion: v1
kind: ConfigMap
metadata:
  name: conf
data:
  {{- (.Files.Glob "foo/*").AsConfig | nindent 2 }}

You can imagine my frustration when I found out that there is no such a feature in Ksonnet. There are workarounds though. But the point is that it's just example of the situation when you are happily writing your chart and then suddenly a lack of some feature stops you on the halfway.
In total, advantages of Ksonnet:

• Working with objects
• Kubernetes-API-compatible Jsonnet libraries
• Helm chart import support

Drawbacks:

• Smaller community and smaller number of Ksonnet-native packages
• Lack of some functionality you can use in Helm
• New syntax => increased learning time => increased bus-factor
• Syntax can sometimes get ugly and less human-readable (especially when making workarounds for lacking features)

Let's think of an ideal templating tool

Here some criteria for the "ideal" templating tool:

• It should work with objects, not strings
• It should have an ability to work with Kubernetes-API-compatible objects
• It should have a decent set of functions for working with strings
• It should work nicely with json and yaml formats
• It should be human-friendly
• It should be simple
• It should have an ability to import existing Helm charts (because this is reality and we want to make use of Helm community)

That's enough for now. I went through this list in my head and thought to myself: okay, why not try Python? Let's see if it fits into our criteria:

• Work with objects, not strings. Yes, we can use dict and list types for that.
• Have an ability to work with Kubernetes-API-compatible objects. Yes, from kubernetes import client
• Have a decent set of functions for working with strings. Plenty!
• Work nicely with json and yaml formats. Very nicely.
• Human-friendly. No shit.
• Simple. Yes.
• Ability to import existing Helm charts. That, we are going to add ourselves.

Ok, looks promising. I decided to write simple templating tool atop of Official Python client library for kubernetes and now let me show you what came out of it.

Meet Karavel

There is nothing special or complicated about this tool. I just took Kubernetes library (which gave me an ability to work with Kubernetes objects) and wrote some basic functionality for existing Helm charts (so that one could fetch them and add into their own chart). So, lets have a tour.
First of all, this tool is accessible at Github repo and you can find a directory with examples there.

Quick start with Docker image

If you want to try it out, the simplest way is to use this docker image:

$ docker run greegorey/karavel -h
usage: karavelcli.py [-h] subcommand ...

optional arguments:
  -h, --help  show this help message and exit

list of subcommands:
  subcommand
    template  generates manifests from template
    ensure    ensure helm dependencies

Of course, if you want to template charts you need to mount your chart's directory:

$ cd example
$ docker run -v $PWD:/chart greegorey/karavel template .

So, let's have a look at the chart structure. It is very similar to one of Helm:

$ cd example
$ tree .
.
├── dependencies
├── prod.yaml
├── requirements.yaml
├── templates
│   ├── custom-resource.py
│   ├── deployment.py
│   └── service-helm.py
└── values.yaml

2 directories, 6 files

Like Helm, it has requirements.yaml file with the same layout:

dependencies:
  - name: mysql
    version: 0.13.1
    repository: https://kubernetes-charts.storage.googleapis.com/

Here you just list your Helm dependencies you want to import into your chart. The dependencies go to the dependencies directory. To fetch or update them use the ensure command:

$ karavel ensure .

After that your dependencies directory will look like this:

$ tree dependencies
dependencies
└── mysql-0.13.1
    └── mysql
        ├── Chart.yaml
        ├── README.md
        ├── templates
        │   ├── NOTES.txt
        │   ├── _helpers.tpl
        │   ├── configurationFiles-configmap.yaml
        │   ├── deployment.yaml
        │   ├── initializationFiles-configmap.yaml
        │   ├── pvc.yaml
        │   ├── secrets.yaml
        │   ├── svc.yaml
        │   └── tests
        │       ├── test-configmap.yaml
        │       └── test.yaml
        └── values.yaml

4 directories, 13 files 

Now after we ensured our dependencies let's have a look at templates. First, we create a simple nginx deployment:

from kubernetes import client

from karavel.helpers import Values

def template():
    values = Values().values
    # Configure Pod template container
    container = client.V1Container(
        name='nginx',
        image='{}:{}'.format(values.nginx.image.repository, values.nginx.image.tag),
        ports=[client.V1ContainerPort(container_port=80)])
    # Create and configurate a spec section
    template = client.V1PodTemplateSpec(
        metadata=client.V1ObjectMeta(labels={'app': 'nginx'}),
        spec=client.V1PodSpec(containers=[container]))
    # Create the specification of deployment
    spec = client.ExtensionsV1beta1DeploymentSpec(
        replicas=3,
        template=template)
    # Instantiate the deployment object
    deployment = client.ExtensionsV1beta1Deployment(
        api_version='extensions/v1beta1',
        kind='Deployment',
        metadata=client.V1ObjectMeta(name='nginx-deployment'),
        spec=spec)

    return deployment # [deployment], (deployment, deployment) are valid

So, for the template to be valid you need to have template() function which returns either single Kubernetes object or a list/tuple of them. You can find the list of API objects for Python client here.
As you can see, the code is clean, simple, readable. You can wonder where from values.nginx.image.repository comes? It gets values from the value files you pass when templating the chart, just like in Helm: karavel template -f one.yaml --values two.yaml. We will have a look at them later.

Okay, what about Helm charts?

Now, we created our own Deployment. But what if we want to import Helm chart or a part of a chart? Let's take a look at templates/service-helm.py:

from kubernetes import client

from karavel.helm import HelmChart
from karavel.helpers import Values

def template():
    values = Values().values
    # Initialize the chart (== helm template --values)
    chart = HelmChart(name='mysql', version='0.13.1', values=values.mysql.helm)
    # Get the desired object from chart
    service = chart.get(name='svc', obj_class=client.V1Service)
    # Create custom objects to add
    custom_ports = [
        client.V1ServicePort(
            name='my-custom-port',
            protocol=values.mysql.protocol,
            port=values.mysql.port,
            target_port=39000,
        )
    ]
    # Add custom objects to the service
    service.spec['ports'] = custom_ports
    # Change Helm-generated label
    service.metadata['labels']['release'] += '-suffix'
    # Delete Helm-generated label `heritage: Tiller`
    del service.metadata['labels']['heritage']

    return service # [service], (service, service) are valid

Simple, huh? Note this line: service = chart.get(name='svc', obj_class=client.V1Service) — we created object of class V1Service form Helm yaml file. If you don't want/need to do that — you can always work with just dict.

What if I want to create custom resource?

Well, there is a small issue with that. Kubernetes API doesn't add CRD objects into swagger json definition at /openapi/v2, and Python-client objects are build upon this definition. But you can still easily work with dict objects. Like this:

from kubernetes import client

def template():
    resource = {
        'apiVersion': 'stable.example.com/v1',
        'kind': 'Whale',
        'metadata': client.V1ObjectMeta(
            name='my-object',
        ),
        'spec': {
            'image': 'my-whale-image:0.0.1',
            'tail': 1,
            'fins': 4,
        }
    }

    return resource # [resource], (resource, resource) are valid

Still looks nice, doesn't it?

Can I have values for different environments, e.g. dev/prod?

Yes, you can!
Let's look at values.yaml first:

nginx:
  image:
    repository: nginx
    tag: 1.15-alpine

mysql:
  port: 3307
  protocol: TCP
  helm:
    releaseName: my-release
    namespace: prod
    imageTag: '5.7.14'
    service:
      type: NodePort

Note the helm key inside mysql dict: we used it when specifying values for helm chart chart = HelmChart(name='mysql', version='0.13.1', values=values.mysql.helm). Some Helm charts need releaseName for application naming and namespace for RBAC policies. These two values are passed to Helm as --namespace and NAME arguments in helm template.

Now, you can specify additional file for prod env, and template all our examples:

$ karavel template -f values.yaml -f prod.yaml .
---
# Source: templates/custom-resource.py
apiVersion: stable.example.com/v1
kind: Whale
metadata:
  name: my-object
spec:
  fins: 4
  image: my-whale-image:0.0.1
  tail: 1

---
# Source: templates/deployment.py
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.14-alpine
        name: nginx
        ports:
        - containerPort: 80

---
# Source: templates/service-helm.py
apiVersion: v1
kind: Service
metadata:
  annotations: null
  labels:
    app: prod-release-mysql
    chart: mysql-0.13.1
    release: prod-release-suffix
  name: prod-release-mysql
spec:
  ports:
  - name: my-custom-port
    port: 3308
    protocol: TCP
    targetPort: 39000
  selector:
    app: prod-release-mysql
  type: NodePort

After that you can do kubeclt apply and deploy these objects into the cluster.

Cool! What about encoding and base64?

import base64

What about using Vault for secrets?

import hvac

Fetching urls?

import importlib

Secure hash functions?

import Crypto

You got it. With Python you can do a lot of things with your Kubernetes manifests.

Is it NIH syndrome?

No :)
I am happily using Helm in my current projects. There are things that I miss though. I used Ksonnet in some of my projects as well.
I would like to think of this tool as a proof-of-concept that we can have templating tools better than Helm and it's not very difficult to develop them using Python. If there is a community interest/need in such a tool we can together continue to develop it. Or we can wait for Helm 3 release ;)

Conclusion

I have showed you Python-based templating tool for Kubernetes which has Kubernetes-API-compatible objects support and support for importing Helm charts. Any comments and discussion from community are welcome, and again welcome to the repo.

Thank you for reading this and have a nice day!

References

• Official Python client library for kubernetes
• Helm
• Ksonnet
• This article on Medium

Автор: gree-gorey

Источник