Making a local MicroK8s environment available externally (Part 4 - Running Docker Containers)

As I mentioned previously, I'm currently building a .Net based Open Source LegalTech platform called Panache Legal. For development and testing I build docker images for the various microservices which are then hosted on Docker Hub and while I can use these with Docker Desktop, I'd prefer to get things up and running with Kubernetes so that I can learn more about that environment and also test things like scaling, which is why we're building this MicroK8s system.

In this example I'm going to be showing how I get Panache Legal up and running, but you may want to choose different containers more suited to your needs, either way, this should still provide the groundwork you need to get going.

Deployment files

To spin up our containers in MicroK8s we could simply jump straight into the command line and deploy our app that way using the following

microk8s kubectl create ......

But for Panache Legal we have multiple containers we need to create, all of which have various command line arguments, and this can become a little cumbersome.

Instead of the command line we'll use deployment files so that we have all the setup in an easy to maintain place.

For Panache Legal I maintain some basic deployment files in the GitHub repository here.

There are individual deployment files for each of the microservices, along with two other scripts DeleteServices.sh and StartServices.sh which provide an easy single script to run which will create or remove all of the services in one go.

Lets take a look inside these files and see how they are setup. For reference you can also check out the kubernetes documentation for deployments.

Let's have a look at the Client service, which provides a .NET Web API for storing and maintaining client information. This Microservice has its own database associated with it and is protected by JWT token authentication, handled by the Identity service.

Looking at the first part of the file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: panachesoftware-service-client
  labels:
    app: panachesoftware-service-client

We specify that this is a Deployment and we provide a name for the app panachesoftware-service-client This is the internal name MicroK8s will use for the app and can be set to whatever you need.

The spec part of the file.

spec:
  replicas: 1
  selector:
    matchLabels:
      app: panachesoftware-service-client
  template:
    metadata:
      labels:
        app: panachesoftware-service-client

Sets out what pods this deployment will manage, in this case we're keeping it simple so we just reference the name we provided in the metadata section.

In addition here we specify the number of replicas. For the moment I simply use 1 replica, but the microservice based approach to Panache Legal means that certain services, like Identity and the UI may be set to use multiple replicas so that more than one instance of the service will run, allowing our application to scale as demand increases.

The next section provides details of the container itself.

spec:
      containers:
      - env:
          - name: ASPNETCORE_ENVIRONMENT
            value: Development
          - name: ASPNETCORE_URLS
            value: http://+:55005
          - name: ConnectionStrings__MySQL
            value: server={db-server-IP};port=3306;database=PanacheSoftware.Client.K8S;user={db-user};password={db-password};GuidFormat=Char36
          - name: PanacheSoftware__CallMethod__APICallsSecure
            value: "False"
          - name: PanacheSoftware__CallMethod__UICallsSecure
            value: "False"
          - name: PanacheSoftware__CallMethod__UseAPIGateway
            value: "True"
          - name: PanacheSoftware__DBProvider
            value: MySQL
          - name: PanacheSoftware__Secret__ClientServiceSecret
            value: AA04416A-A87B-4D88-956B-27CBFFCC2802
          - name: PanacheSoftware__StartDomain
            value: panachesoftware.com
          - name: PanacheSoftware__Url__IdentityServerURL
            value: http://{server-IP}:30002
          - name: PanacheSoftware__Url__IdentityServerURLSecure
            value: https://{server-IP}:30002
        image: panachesoftware/panachesoftwareserviceclient:latest
        name: panachesoftware-service-client
        ports:
        - containerPort: 55005
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet

The name, value pairs shown here all correspond to environment variables expected by the service, so if you were to look at the appsettings.json file of the client service .NET project, you will see corresponding settings.

"ConnectionStrings": {
    "MSSQL": "Data Source=localhost;Database=PanacheSoftware.Client;User ID=sa;Password=Passw0rd123!;Connect Timeout=30;Encrypt=False;TrustServerCertificate=False;ApplicationIntent=ReadWrite;MultiSubnetFailover=False",
    "MySQL": "server=raspberrypi;port=3306;database=PanacheSoftware.Client;user=pi;password=Passw0rd123!;GuidFormat=Char36"
  },
  "PanacheSoftware": {
    "DBProvider": "MySQL",
    "StartDomain": "panachesoftware.com",
    "CallMethod": {
      "APICallsSecure": "false",
      "UICallsSecure": "false",
      "UseAPIGateway": "true"
    },
    "Url": {
      "IdentityServerURL": "http://localhost:55002",
      "IdentityServerURLSecure": "https://localhost:44302",
      "APIGatewayURL": "http://localhost:55003",
      "APIGatewayURLSecure": "https://localhost:44303"
    },
    "Secret": {
      "ClientServiceSecret": "1314EF18-40FA-4B16-83DF-B276FF0D92A9"
    }
  }

All I've changed in the example deployment files is to put placeholders, like {db-server-IP} and {db-user}, in the file where you will need to change these to be applicable in your system.

We setup MySQL in a previous part so you should have the relevant username and password for the services that require a database, and we're using the same machine to host MicroK8s and our MySQL database so {db-server-IP} and {server-IP} will be the same.

if you need to determine the IP address of your VM you can use the following command

hostname -I | awk '{print $1}'

Be aware that in general, a new IP address will likely be assigned each time you VM starts. One way to change this is to update the DHCP IP reservations within your router to make sure that the IP address assigned to your VM (based on its hostname) always remains the same.

Outside of the environment variables there are a couple of other important items.

image specifies the docker image that will be used in this deployment. This is the image name along with the appropriate tag, in this case the client service with the latest tag.

panachesoftware/panachesoftwareserviceclient:latest

Alongside this the containerPort specifies the port number that the service will be running on, in this case 55005.

When an app is deployed in MicroK8s it will be made available on the cluster IP, however we want to make the service available by going to the IP of the VM itself so we add an additional section to the deployment file.

---
apiVersion: v1
kind: Service
metadata:
  name: panachesoftware-service-client-service
spec:
  type: NodePort
  selector:
    app: panachesoftware-service-client
  ports:
    - protocol: TCP
      port: 55005
      targetPort: 55005
      nodePort: 30005

You could have included this service detail in a separate file, but for our setup it is easier to include it within the deployment file.

By choosing NodePort for our type it specifies that the app will be exposed via a port on the host, rather than on the cluster IP. If you don't add anything further it will be assigned a random port within the '30000-32767' range by default.

We need our services to talk to each other, as shown by the references to other other parts of the platform in the environment variables i.e. PanacheSoftware__Url__IdentityServerURL providing the address of the Identity Server, so we need to specify a specific port that will always be used, this is the final section.

ports:
    - protocol: TCP
      port: 55005
      targetPort: 55005
      nodePort: 30005

Where we tell MicroK8s to map the port '55005' of the service to be exposed on port '30005' of the host.

Performing the deployment

Now we've gone through the details of the deployment files go ahead and run through the Panache Legal examples adjusting the placeholders as appropriate.

If you edit the files on another machine, maybe the Windows Host rather than on the Linux VM we created then you can copy the files to your Linux VM using a tool like WinSCP.

If you want to use the provided scripts to run the deployment files for you then make sure to set those files as executable with the following commands.

chmod +x DeleteServices.sh
chmod +x StartServices.sh

One thing to note here is that if you have the Linux Firewall running, as discussed when we were setting up the MicroK8s dashboard, you may need to allow the ports that the services will use to be exposed. MicroK8s will actually edit iptables behind the scenes and allow access to the nodeport ports, 30001-30010 for our services, but if we go to these addresses in our browser it will fail, we need to enable the 55001-55010 port range in ufw, which is confusing, but works!

Allow these ports via the following command.

sudo ufw allow 55001:55010/tcp

Now, either run the StartServices.sh script with the following command.

./StartServices.sh

Or start-up your services individually with something like.

microk8s kubectl apply -f panachesoftware-client-service.yaml

If you're running the StartServices.sh script, once it completes you should be able to see the 10 pods it creates starting up by using the following command.

microk8s kubectl get pods -o wide

You can also watch the progress of the pod creation via the dashboard we configured in Step 2.

On first creation it will take a bit of time to complete as the various containers need to be downloaded from Docker Hub, but after a while you should see the status of the pods move from ContainerCreating to Running.

Once everything is running let's test it out, on your windows host you should be able to got to http://{VM-IP-Address}:30001 and see the Panache Legal Login screen.

Assuming you kept the PanacheSoftware__StartDomain environment variable as panachesoftware.com the system will have started up and created a default user with the details, Username: admin@panachesoftware.com, Password: Passw0rd123!. So hit Login and use those credentials to access the system.

In the left menu, if you go to Settings > API Access you should see a list of the individual services that make up the Panache Legal Platform, all of which are hosted as separate pods in MicroK8s. On this screen you can see the links to those services swagger definition, and you can note that they are all assigned to the ports specified in the deployment files.

If you head on over to phpMyAdmin, that we setup in step 2, you should be able to see that all of the databases associated with our services have been automatically created via the entity framework.

If you find that any pods didn't create, maybe they are stuck with a status of crash loop or some other error, you can check the logs for the service within the 'Pods' area of the dashboard, just click on the 3 dots on the right hand side and choose 'Logs'.

In general, the majority of errors often relate to the service not being able to connect to a database, so check usernames and passwords and all the access settings we configured in Step 3.

Next Steps

The main bulk of the work is now complete. So far we've.

• Created a Linux VM
• Setup MicroK8s
• Setup MySQL
• Setup NGINX
• Setup phpMyAdmin
• Spun up some services in MicroK8s

If all we need to do is host some docker containers locally then your work is done. But if you want a little bit more, lets head over to our final step where we're going to make our local environment available to the outside world by using SSH.

Pete