Skip to content

How to set up scaling and autoscaling in Kubernetes.

Today I am going to show how to scale docker containers on Kubernetes and you will see how easy it is.
Then we will look at how pods could be autoscaled based on the performance degradation and CPU Utilisation.


1. Deploy simple stack to k8s
2. Scaling the deployment manually.
3. Autoscaling in k8s based on CPU Utilisation.

1. Deploy simple stack to k8s

If you don’t have Kubernetes installed on your machine in this article I demonstrate how easily this can be achieved on MacOS, it literally takes few minutes to set up.

So let’s create a deployment of a simple test http server container:

  
➜  ~ kubectl  run busybox --image=busybox --port 8080  \
         -- sh -c "while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; \
         env | grep HOSTNAME | sed 's/.*=//g'; } | nc -l -p  8080; done"
deployment "busybox" created

I have also set it up in a way so it returns it’s hostname in the response to http get request, we will need it to distinguish
responses from different instances later on. Once deployed, we can check our deployment and pod status:

➜  ~ kubectl get deployments
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox   1         1         1            1           3m
➜  ~ kubectl get pod
NAME                       READY     STATUS        RESTARTS   AGE
busybox-7bcdf6684b-jnp6w   1/1       Running       0          18s
➜  ~

As you can see it’s current ‘DESIRED’ state equals to 1.

Next step is to expose our deployment through a service so it can be queried from outside of the cluster:

➜  ~ kubectl expose deployment busybox --type=NodePort
service "busybox" exposed

This will expose our endpoint:

➜  ~ kubectl get endpoints
NAME         ENDPOINTS         AGE
busybox      172.17.0.9:8080   23s

Once it is done, we can ask our cluster manager tool to get us it’s api url:

➜  ~ minikube service busybox --url
http://192.168.99.100:31623

If we query it we will get it’s hostname in the response:

➜  ~ curl http://192.168.99.100:31623
busybox-7bcdf6684b-jnp6w

2. Scaling the deployment manually.
Now our deployment is ready to be scaled:

➜  ~ kubectl scale --replicas=3 deployment busybox
deployment "busybox" scaled

If we check the status immediately then we might get different numbers in the ‘AVAILABLE’ column:

➜  ~ kubectl get deployments
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox   3         3         3            2           4m

As you can see it is still starting up so we only got 2 pods available, but a bit later on,
if you try again:

➜  ~ kubectl get deployments
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox   3         3         3            3           4m

We got 3 now!

➜  ~ kubectl get pods
NAME                       READY     STATUS        RESTARTS   AGE
busybox-7bcdf6684b-jnp6w   1/1       Running       0          4m
busybox-7bcdf6684b-ltvkd   1/1       Running       0          15s
busybox-7bcdf6684b-pczxz   1/1       Running       0          15s
➜  ~
➜  ~ kubectl get endpoints
NAME         ENDPOINTS                                         AGE
busybox      172.17.0.4:8080,172.17.0.7:8080,172.17.0.9:8080   2m
kubernetes   10.0.2.15:8443                                    1d

Now if we query our service we should get responses from different instances:

➜  ~ curl http://192.168.99.100:31623
busybox-7bcdf6684b-ltvkd
➜  ~ curl http://192.168.99.100:31623
busybox-7bcdf6684b-jnp6w
➜  ~ curl http://192.168.99.100:31623
busybox-7bcdf6684b-jnp6w
➜  ~ curl http://192.168.99.100:31623
busybox-7bcdf6684b-pczxz
➜  ~

Let’s scale it back to one and then scale again with autoscaling:

➜  ~ kubectl scale --replicas=1 deployment busybox
deployment "busybox" scaled

➜  ~ kubectl get deployments
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox   1         1         1            1           6m

➜  ~ kubectl get pods
NAME                       READY     STATUS        RESTARTS   AGE
busybox-7bcdf6684b-jnp6w   1/1       Running       0          6m
busybox-7bcdf6684b-ltvkd   1/1       Terminating   0          2m
busybox-7bcdf6684b-pczxz   1/1       Terminating   0          2m

As you can see, k8s is removing the other two pods, now if we send a new request again,
we should only get back from single instance:

➜  ~  curl http://192.168.99.100:31623
busybox-7bcdf6684b-jnp6w
➜  ~  curl http://192.168.99.100:31623
busybox-7bcdf6684b-jnp6w

3. Autoscaling in k8s based on CPU Utilisation.

First thing first, you can’t autoscale with default settings minikibe comes with, lets check the config:

➜  ~ minikube addons list
- default-storageclass: enabled
- kube-dns: enabled
- registry: disabled
- addon-manager: enabled
- dashboard: enabled
- ingress: disabled
- registry-creds: disabled
- coredns: disabled
- heapster: disabled

As you can see heapster is disabled. What is heapster?
Heapster is resource usage analysis and monitoring tool which collects compute resource usage
and then other tools can request this data for various reasons.

So let’s enable it:

➜  ~ minikube addons enable heapster
heapster was successfully enabled
➜  ~

Now, we can redeploy the stack:

➜  ~ kubectl delete deployment busybox
deployment "busybox" deleted
➜  ~ kubectl run busybox --image=busybox --port 8080 --requests="cpu=200m" \
         -- sh -c "while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; \
         env | grep HOSTNAME | sed 's/.*=//g'; } | nc -l -p  8080; done"

Please pay attention that this time I am using –requests=”cpu=200m” argument,
if we don’t use it then Horizontal Pod Autoscaler, which is responsible for autoscaling,
will not be able to compare current CPU usage with requested CPU usage, in order to perform scaling.

Let’s check our deployment and see how it changes after autoscaling is enabled.

➜  ~ kubectl get deployments
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox   1         1         1            1           5s

Now let’s autoscale our deployment:

➜  ~ kubectl autoscale deployment busybox --min=1 --max=4 --cpu-percent=20
deployment "busybox" autoscaled

And then check autoscaling status:

  ~ kubectl get hpa
NAME      REFERENCE            TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
busybox   Deployment/busybox   <unknown> / 20%   1         4         0          14s

Here hpa stands for for Horizontal Pod Autoscaler. So what we have just done is declared
that whenever CPU usage goes above 20%, increase number of the pods to reach desired level, but not above 4 pods.

Now interesting parts starts, let’s give some hard time to our pod, let’s query it in a loop:

while true; do curl  http://192.168.99.100:31623 ; done

You may increase the load by running two instances of the script, if you don’t want to wait for ages.

Let’s check the state of hpa, bear in mind that the next figure is a result of running the requests for about 15 minutes on my Mac:

 
➜  ~ kubectl describe hpa
Name:                                                  busybox
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Tue, 28 Nov 2017 20:16:21 +0000
Reference:                                             Deployment/busybox
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (0) / 20%
Min replicas:                                          1
Max replicas:                                          4
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     False   BackoffBoth       the time since the previous scale is still within both the downscale and upscale forbidden windows
  ScalingActive   True    ValidMetricFound  the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count was less than the minimum replica count
Events:
  Type     Reason                        Age   From                       Message
  ----     ------                        ----  ----                       -------
  Warning  FailedGetResourceMetric       14m   horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from heapster
  Warning  FailedComputeMetricsReplicas  14m   horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster
  Normal   SuccessfulRescale             11m   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             7m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             2m    horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
➜  ~

As you can see, over 12 minutes time, the size has increased from 1 to 4, and then, as I stopped the loops, it got back to one.
You can also run:

watch kubectl  describe hpa

so it will be updated as events come in, at the same time in a different bash screen you can run the following:

➜  ~ kubectl get hpa --watch
NAME      REFERENCE            TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
busybox   Deployment/busybox   0% / 20%   1         4         1          2m
busybox   Deployment/busybox   39% / 20%   1         4         1         3m
busybox   Deployment/busybox   39% / 20%   1         4         2         4m
busybox   Deployment/busybox   59% / 20%   1         4         2         4m
busybox   Deployment/busybox   59% / 20%   1         4         2         5m
busybox   Deployment/busybox   53% / 20%   1         4         2         5m
busybox   Deployment/busybox   53% / 20%   1         4         2         6m
busybox   Deployment/busybox   53% / 20%   1         4         2         6m
busybox   Deployment/busybox   53% / 20%   1         4         2         7m
busybox   Deployment/busybox   54% / 20%   1         4         2         7m
busybox   Deployment/busybox   54% / 20%   1         4         4         8m
busybox   Deployment/busybox   42% / 20%   1         4         4         8m
busybox   Deployment/busybox   42% / 20%   1         4         4         9m
busybox   Deployment/busybox   30% / 20%   1         4         4         9m
busybox   Deployment/busybox   30% / 20%   1         4         4         10m
busybox   Deployment/busybox   28% / 20%   1         4         4         10m
busybox   Deployment/busybox   28% / 20%   1         4         4         11m
busybox   Deployment/busybox   5% / 20%   1         4         4         11m
busybox   Deployment/busybox   5% / 20%   1         4         4         12m
busybox   Deployment/busybox   0% / 20%   1         4         4         12m
busybox   Deployment/busybox   0% / 20%   1         4         1         13m
busybox   Deployment/busybox   0% / 20%   1         4         1         13m

and thankfully it actually supports embedded ‘watch’ feature.

Meanwhile you can check deployments and replica sets during this 15 minutes time interval to see how they get automatically updated.

➜  ~ kubectl get deployment
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox   2         2         2            2           7m

➜  ~ kubectl get rs
NAME                 DESIRED   CURRENT   READY     AGE
busybox-6d9d77755f   2         2         2         7m

In the end the should go back to their initial stage though.

By the way, as you enabled heapster addon earlier, now we have some more pods running, but
in order to see them you need to run get pod with kube-system namespace:

kubectl  get pod  --namespace=kube-system
NAME                          READY     STATUS    RESTARTS   AGE
heapster-q5rnm                1/1       Running   2          2d
influxdb-grafana-fpn9b        2/2       Running   4          2d
kube-addon-manager-minikube   1/1       Running   3          3d
kube-dns-6fc954457d-pkd2s     3/3       Running   9          3d
kubernetes-dashboard-cwpdp    1/1       Running   3          3d

As you can see, heapster brought grafana, meaning we can now check it’s dashboard, to view it run:

minikube addons open heapster

You can see per node and per pod load:

You can also view k8s dashboard with some stats:

minikube dashboard