/ helm

Waiting For Tiller

Level - Advanced. Read Time ~ <10 minutes.

tl;dr

Scroll down to Final Solution section.

Installing Helm

Recently I've been fraternizing with Kubernetes. I'm already comfortable with Docker and the next logical step seems to be container orchestration. This is a very popular way of running workloads in a microservice architecture. So time for me to jump on the band wagon and get hands on experience with the magic which is Kubernetes.

This is not an overview of Kubernetes or Helm, just a couple of useful commands if you are using Helm. You can do your own studying on what Helm is, if you have arrived here and not heard of it before. I'll dive straight in with the reason for writing this post.

helmsv

A lot of tutorials on Kubernetes tell you to reach for Minikube if you are looking to monkey around with a local cluster. I persevered with Minikube for a while. But I found it problematic. I work on a Mac and I found when I turned the Mac on and off, I couldn't get the cluster up and running again. Also, the initial creation of the cluster was very slow and at times completely unresponsive. The only way I was able to get it back was running minikube delete and also deleting the VM created in Virtual Box. I got to the point that I thought local development of Kubernetes is just too tiring and becoming a time sink. Then I discovered Docker for Mac has built in Kubernetes support. Read about it here.

saviour

Right, so with my growing pains of running k8s locally all cleared up. I proceeded to start looking into Helm. I heard about it on Twitter and podcast and a Kubernetes package manager sounds a very useful tool. With using aptitude on Linux and Homebrew on the Mac (to name a couple), the concept of a package manager feels right at home. I wanted to install Redis in my cluster for an application I was attempting to make k8s ready. So I reached for the official Redis Helm chart.

redisk8

In my wisdom I decided to stitch together a repeatable bash script for creating and instrumenting a cluster. If you look below, you'll see the script I put together. This gets helm installed on your machine and checks you are using Docker for Mac. It also sets the Kubernetes config context to your local machine.

#!/usr/bin/env bash

echo "Checking Helm is installed"
if [[ $((helm) 2>&1 | grep "command not found" ) ]]; then
    echo "You must install Helm. https://github.com/helm/helm/blob/master/docs/install.md"
    exit 1
fi

echo "Checking Kubernetes is installed with Docker for Mac"
if [[ $((kubectl config use-context docker-for-desktop) 2>&1 | grep "error" ) ]]; then
    echo "This deployment requires Kubernetes running with Docker for Mac"
    exit 1
else
    echo "Kubernetes is installed with Docker For Desktop. Yaay!!!"
fi

echo "Setting context as docker-for-desktop"
kubectl config use-context docker-for-desktop

So, that is fine for the basics. But for my use case I wanted to install the Redis Helm chart. This was the reason I wanted to install Helm in my cluster. With the above script the next stage was to install Redis into the cluster using Helm on cluster creation. Kubernetes cluster creation should be ephemeral. See below how I was installing Redis: -

helm install --name redis --namespace hackerlite stable/redis

The problem I was having that after installing the Tiller agent in the cluster and immediately after installing Redis, it was not working. Scratching my head with these errors. After further investigation I realised by the running the following command that Tiller was not running when attempting to deploy Redis.

kubectl get pods -n kube-system | grep "tiller" 

So my first step was to write a loop and wait for up to 2 minutes for it to become ready. This can be seen below: -

counter=0

until kubectl get pods -n kube-system | grep "tiller" | grep -q "Running"; do
    echo "Waiting for tiller (Helm) to initialise  - try ${counter}"
    sleep 2

    if [[ "$counter" -gt 120 ]]; then
        echo "Helm taking longer than 2 minutes to start up"
        exit 1
    fi

    counter=$((counter+1))
done

echo "Tiller is running"

So great I won't deploy until it's ready. Surely this will work, but no. Another error. This time I was getting hit with this error message Error: could not find a ready tiller pod. So we've come to the conclusion that it's running but not ready. So time for a similar check to the running check, but this time to ensure it's ready. See my script below: -

# Tiller ready
counter=0

until helm status 2> >(grep Error) | grep "release name"; do
    echo "Waiting for tiller (Helm) to become ready  - try ${counter}"
    sleep 2

    if [[ "$counter" -gt 120 ]]; then
        echo "Helm is taking to long to become ready"
        exit 1
    fi

    counter=$((counter+1))
done

echo "Tiller is ready"

That's all I had to do to make my cluster creation script ephemeral. Disclaimer this has only been tested locally on the Mac. So I'd appreciate any feedback or ways this can be improved. Or a complete different solution where this is unnecessary. I'm all ears and willing to learn. Hope this helps someone out there.

Final Solution

success1

#!/usr/bin/env bash

echo "Checking Helm is installed"
if [[ $((helm) 2>&1 | grep "command not found" ) ]]; then
    echo "You must install Helm. https://github.com/helm/helm/blob/master/docs/install.md"
    exit 1
fi

echo "Checking Kubernetes is installed with Docker for Mac"
if [[ $((kubectl config use-context docker-for-desktop) 2>&1 | grep "error" ) ]]; then
    echo "This deployment requires Kubernetes running with Docker for Mac"
    exit 1
else
    echo "Kubernetes is installed with Docker For Desktop. Yaay!!!"
fi

echo "Setting context as docker-for-desktop"
kubectl config use-context docker-for-desktop

# set up helm
helm init --service-account tiller

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

# Tiller running
counter=0

until kubectl get pods -n kube-system | grep "tiller" | grep -q "Running"; do
    echo "Waiting for tiller (Helm) to initialise  - try ${counter}"
    sleep 2

    if [[ "$counter" -gt 120 ]]; then
        echo "Helm taking longer than 2 minutes to start up"
        exit 1
    fi

    counter=$((counter+1))
done

echo "Tiller is running"

# Tiller ready
counter=0

until helm status 2> >(grep Error) | grep "release name"; do
    echo "Waiting for tiller (Helm) to become ready  - try ${counter}"
    sleep 2

    if [[ "$counter" -gt 120 ]]; then
        echo "Helm is taking to long to become ready"
        exit 1
    fi

    counter=$((counter+1))
done

echo "Tiller is ready"