Today I am going to show how to scale docker containers on Kubernetes and you will see how easy it is.
Then we will look at how pods could be autoscaled based on the performance degradation and CPU Utilisation.
1. Deploy simple stack to k8s
2. Scaling the deployment manually.
3. Autoscaling in k8s based on CPU Utilisation.
1. Deploy simple stack to k8s
If you don’t have Kubernetes installed on your machine in this article I demonstrate how easily this can be achieved on MacOS, it literally takes few minutes to set up.
So let’s create a deployment of a simple test http server container:
➜ ~ kubectl run busybox --image=busybox --port 8080 \
-- sh -c "while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; \
env | grep HOSTNAME | sed 's/.*=//g'; } | nc -l -p 8080; done"
deployment "busybox" created
I have also set it up in a way so it returns it’s hostname in the response to http get request, we will need it to distinguish
responses from different instances later on. Once deployed, we can check our deployment and pod status:
➜ ~ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox 1 1 1 1 3m
➜ ~ kubectl get pod NAME READY STATUS RESTARTS AGE busybox-7bcdf6684b-jnp6w 1/1 Running 0 18s ➜ ~
As you can see it’s current ‘DESIRED’ state equals to 1.
Next step is to expose our deployment through a service so it can be queried from outside of the cluster:
➜ ~ kubectl expose deployment busybox --type=NodePort service "busybox" exposed
This will expose our endpoint:
➜ ~ kubectl get endpoints NAME ENDPOINTS AGE busybox 172.17.0.9:8080 23s
Once it is done, we can ask our cluster manager tool to get us it’s api url:
➜ ~ minikube service busybox --url http://192.168.99.100:31623
If we query it we will get it’s hostname in the response:
➜ ~ curl http://192.168.99.100:31623 busybox-7bcdf6684b-jnp6w
2. Scaling the deployment manually.
Now our deployment is ready to be scaled:
➜ ~ kubectl scale --replicas=3 deployment busybox deployment "busybox" scaled
If we check the status immediately then we might get different numbers in the ‘AVAILABLE’ column:
➜ ~ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox 3 3 3 2 4m
As you can see it is still starting up so we only got 2 pods available, but a bit later on,
if you try again:
➜ ~ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox 3 3 3 3 4m
We got 3 now!
➜ ~ kubectl get pods NAME READY STATUS RESTARTS AGE busybox-7bcdf6684b-jnp6w 1/1 Running 0 4m busybox-7bcdf6684b-ltvkd 1/1 Running 0 15s busybox-7bcdf6684b-pczxz 1/1 Running 0 15s ➜ ~
➜ ~ kubectl get endpoints NAME ENDPOINTS AGE busybox 172.17.0.4:8080,172.17.0.7:8080,172.17.0.9:8080 2m kubernetes 10.0.2.15:8443 1d
Now if we query our service we should get responses from different instances:
➜ ~ curl http://192.168.99.100:31623 busybox-7bcdf6684b-ltvkd ➜ ~ curl http://192.168.99.100:31623 busybox-7bcdf6684b-jnp6w ➜ ~ curl http://192.168.99.100:31623 busybox-7bcdf6684b-jnp6w ➜ ~ curl http://192.168.99.100:31623 busybox-7bcdf6684b-pczxz ➜ ~
Let’s scale it back to one and then scale again with autoscaling:
➜ ~ kubectl scale --replicas=1 deployment busybox deployment "busybox" scaled ➜ ~ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox 1 1 1 1 6m ➜ ~ kubectl get pods NAME READY STATUS RESTARTS AGE busybox-7bcdf6684b-jnp6w 1/1 Running 0 6m busybox-7bcdf6684b-ltvkd 1/1 Terminating 0 2m busybox-7bcdf6684b-pczxz 1/1 Terminating 0 2m
As you can see, k8s is removing the other two pods, now if we send a new request again,
we should only get back from single instance:
➜ ~ curl http://192.168.99.100:31623 busybox-7bcdf6684b-jnp6w ➜ ~ curl http://192.168.99.100:31623 busybox-7bcdf6684b-jnp6w
3. Autoscaling in k8s based on CPU Utilisation.
First thing first, you can’t autoscale with default settings minikibe comes with, lets check the config:
➜ ~ minikube addons list - default-storageclass: enabled - kube-dns: enabled - registry: disabled - addon-manager: enabled - dashboard: enabled - ingress: disabled - registry-creds: disabled - coredns: disabled - heapster: disabled
As you can see heapster is disabled. What is heapster?
Heapster is resource usage analysis and monitoring tool which collects compute resource usage
and then other tools can request this data for various reasons.
So let’s enable it:
➜ ~ minikube addons enable heapster heapster was successfully enabled ➜ ~
Now, we can redeploy the stack:
➜ ~ kubectl delete deployment busybox deployment "busybox" deleted
➜ ~ kubectl run busybox --image=busybox --port 8080 --requests="cpu=200m" \
-- sh -c "while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; \
env | grep HOSTNAME | sed 's/.*=//g'; } | nc -l -p 8080; done"
Please pay attention that this time I am using –requests=”cpu=200m” argument,
if we don’t use it then Horizontal Pod Autoscaler, which is responsible for autoscaling,
will not be able to compare current CPU usage with requested CPU usage, in order to perform scaling.
Let’s check our deployment and see how it changes after autoscaling is enabled.
➜ ~ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox 1 1 1 1 5s
Now let’s autoscale our deployment:
➜ ~ kubectl autoscale deployment busybox --min=1 --max=4 --cpu-percent=20 deployment "busybox" autoscaled
And then check autoscaling status:
~ kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE busybox Deployment/busybox <unknown> / 20% 1 4 0 14s
Here hpa stands for for Horizontal Pod Autoscaler. So what we have just done is declared
that whenever CPU usage goes above 20%, increase number of the pods to reach desired level, but not above 4 pods.
Now interesting parts starts, let’s give some hard time to our pod, let’s query it in a loop:
while true; do curl http://192.168.99.100:31623 ; done
You may increase the load by running two instances of the script, if you don’t want to wait for ages.
Let’s check the state of hpa, bear in mind that the next figure is a result of running the requests for about 15 minutes on my Mac:
➜ ~ kubectl describe hpa Name: busybox Namespace: default Labels: <none> Annotations: <none> CreationTimestamp: Tue, 28 Nov 2017 20:16:21 +0000 Reference: Deployment/busybox Metrics: ( current / target ) resource cpu on pods (as a percentage of request): 0% (0) / 20% Min replicas: 1 Max replicas: 4 Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale False BackoffBoth the time since the previous scale is still within both the downscale and upscale forbidden windows ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited True TooFewReplicas the desired replica count was less than the minimum replica count Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedGetResourceMetric 14m horizontal-pod-autoscaler unable to get metrics for resource cpu: no metrics returned from heapster Warning FailedComputeMetricsReplicas 14m horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster Normal SuccessfulRescale 11m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target Normal SuccessfulRescale 7m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target Normal SuccessfulRescale 2m horizontal-pod-autoscaler New size: 1; reason: All metrics below target ➜ ~
As you can see, over 12 minutes time, the size has increased from 1 to 4, and then, as I stopped the loops, it got back to one.
You can also run:
watch kubectl describe hpa
so it will be updated as events come in, at the same time in a different bash screen you can run the following:
➜ ~ kubectl get hpa --watch NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE busybox Deployment/busybox 0% / 20% 1 4 1 2m busybox Deployment/busybox 39% / 20% 1 4 1 3m busybox Deployment/busybox 39% / 20% 1 4 2 4m busybox Deployment/busybox 59% / 20% 1 4 2 4m busybox Deployment/busybox 59% / 20% 1 4 2 5m busybox Deployment/busybox 53% / 20% 1 4 2 5m busybox Deployment/busybox 53% / 20% 1 4 2 6m busybox Deployment/busybox 53% / 20% 1 4 2 6m busybox Deployment/busybox 53% / 20% 1 4 2 7m busybox Deployment/busybox 54% / 20% 1 4 2 7m busybox Deployment/busybox 54% / 20% 1 4 4 8m busybox Deployment/busybox 42% / 20% 1 4 4 8m busybox Deployment/busybox 42% / 20% 1 4 4 9m busybox Deployment/busybox 30% / 20% 1 4 4 9m busybox Deployment/busybox 30% / 20% 1 4 4 10m busybox Deployment/busybox 28% / 20% 1 4 4 10m busybox Deployment/busybox 28% / 20% 1 4 4 11m busybox Deployment/busybox 5% / 20% 1 4 4 11m busybox Deployment/busybox 5% / 20% 1 4 4 12m busybox Deployment/busybox 0% / 20% 1 4 4 12m busybox Deployment/busybox 0% / 20% 1 4 1 13m busybox Deployment/busybox 0% / 20% 1 4 1 13m
and thankfully it actually supports embedded ‘watch’ feature.
Meanwhile you can check deployments and replica sets during this 15 minutes time interval to see how they get automatically updated.
➜ ~ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox 2 2 2 2 7m ➜ ~ kubectl get rs NAME DESIRED CURRENT READY AGE busybox-6d9d77755f 2 2 2 7m
In the end the should go back to their initial stage though.
By the way, as you enabled heapster addon earlier, now we have some more pods running, but
in order to see them you need to run get pod with kube-system namespace:
kubectl get pod --namespace=kube-system NAME READY STATUS RESTARTS AGE heapster-q5rnm 1/1 Running 2 2d influxdb-grafana-fpn9b 2/2 Running 4 2d kube-addon-manager-minikube 1/1 Running 3 3d kube-dns-6fc954457d-pkd2s 3/3 Running 9 3d kubernetes-dashboard-cwpdp 1/1 Running 3 3d
As you can see, heapster brought grafana, meaning we can now check it’s dashboard, to view it run:
minikube addons open heapster
You can see per node and per pod load:
You can also view k8s dashboard with some stats:
minikube dashboard
