Today I am going to show how to setup Kubernetes cluster on AWS using kops(k8s operations).
In order to provision k8s cluster and deploy a Docker container we will need to install and setup couple of things, so here is the list:
0. Setup a VM with CentOS Linux as a control center.
1. Install and configure AWS cli to manage AWS resources.
2. Install and configure kops to manage provisioning of k8s cluster and AWS resources required by k8s.
3. Create a hosted zone in AWS Route53 and setup ELB to access deployed container services.
4. Install and configure kubectl to manage containers on k8s.
5. (Update September 2018) Set up authentication in Kubernetes with AWS IAM Authenticator(heptio).
6. (Update June 2019) The advanced way: Automating Highly Available Kubernetes and external ETCD cluster setup with terraform and kubeadm on AWS..
0. Setup a VM with CentOS Linux
Even though I am using MacOS, sometimes it is annoying that you can’t run certain commands or some arguments are different, so let’s spin up a Linux VM first, I choose centos this time, you can go with ubuntu if you wish. Here is how Vagrantfile looks like:
➜ kops cat Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(2) do |config|
config.vm.define "kops" do |m|
m.vm.box = "centos/7"
m.vm.hostname = "kops"
end
end
Let’s start it up and logon:
vagrant up && vagrant ssh Bringing machine 'kops' up with 'virtualbox' provider... ==> kops: Checking if box 'centos/7' is up to date... ==> kops: A newer version of the box 'centos/7' is available! You currently ==> kops: have version '1708.01'. The latest is version '1710.01'. Run ==> kops: `vagrant box update` to update. ==> kops: Machine already provisioned. Run `vagrant provision` or use the `--provision` ==> kops: flag to force provisioning. Provisioners marked to run always will still run. Last login: Sun Dec 17 10:48:48 2017 from 10.0.2.2 [vagrant@kops ~]$
1. Install and configure awscli
In order to install aws cli we will need a pip – tool for installing Python packages. As centos doesn’t offer it in it’s
core repositories we will need to enable EPEL first:
[vagrant@kops ~]$ sudo yum -y update [vagrant@kops ~]$ sudo yum install epel-release -y
Now we can install pip and then aws cli:
[vagrant@kops ~]$ sudo yum -y install python-pip [vagrant@kops ~]$ sudo pip install awscli
Next we need to configure awscli:
[vagrant@kops ~]$ aws configure AWS Access Key ID [None]:
yeah, but we don’t have any access keys, so let’s go and configure it first, go to your AWS IAM console and create
a user with admin rights, as shown below:
it doesn’t matter how you call user, important thing is – it should have programmatic access and admin rights, as kops will use it to create a hell lots of things on AWS.
Once you got your access key and secret let’s configure aws cli:
[vagrant@kops ~]$ aws configure AWS Access Key ID [None]: AKIAJSQKBOVAVI**** AWS Secret Access Key [None]: PsAJX************* Default region name [None]: eu-west-2 Default output format [None]:
I used eu-west-2 as I am in London, you can use any region in fact if you want.
2. Install and configure kops
Now we are ready to download and configure kops.
[vagrant@kops ~]$ curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64 [vagrant@kops ~]$ chmod +x kops-linux-amd64 [vagrant@kops ~]$ sudo mv kops-linux-amd64 /usr/local/bin/kops [vagrant@kops ~]$ kops version Version 1.8.0 (git-5099bc5)
It’s almost ready, but before we start using it, we need to configure S3 bucket, as kops is using S3 to store clusters state, so let’s create a bucket in the same region where your earlier configured your aws cli:
[vagrant@kops ~]$ aws s3api create-bucket --bucket kayan-kops-state --region eu-west-2 --create-bucket-configuration
LocationConstraint=eu-west-2
{
"Location": "http://kayan-kops-state.s3.amazonaws.com/"
}
[vagrant@kops ~]$
I called ‘kayan-kops-state’ you can call it as you wish just make sure it is unique otherwise it complains:
[vagrant@kops ~]$ aws s3api create-bucket --bucket kops-state --region eu-west-2 --create-bucket-configuration LocationConstraint=eu-west-2 An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again. [vagrant@kops ~]$
3. Create a hosted zone in AWS Route53.
Now we need to create a subdomain in route53 so kops can use it to setup DNS records for our cluster.
If you don’t have a domain, you can register one with AWS, it is quite cheap, around 10 quid, alternatively you can
just create a subdomain in route53 and update your root domain’s NS records. I have a domain registered with AWS ifritltd.net, so I will be just creating a subdomain k8s.ifritltd.net as shown below:
You can also do it through command:
aws route53 create-hosted-zone --name k8s.ifritltd.net --caller-reference 1
{
"HostedZone": {
"ResourceRecordSetCount": 2,
"CallerReference": "1",
"Config": {
"PrivateZone": false
},
"Id": "/hostedzone/Z3I6D40LJGM0M0",
"Name": "k8s2.ifritltd.net."
},
"DelegationSet": {
"NameServers": [
"ns-632.awsdns-15.net",
"ns-1454.awsdns-53.org",
"ns-131.awsdns-16.com",
"ns-1630.awsdns-11.co.uk"
]
},
"Location": "https://route53.amazonaws.com/2013-04-01/hostedzone/Z3I6D40LJGM0M0",
"ChangeInfo": {
"Status": "PENDING",
"SubmittedAt": "2017-12-17T14:37:30.685Z",
"Id": "/change/C38C5CQRHYEMOS"
}
}
Now copy NameServers, create a NS record in your root domain, and paste there. Once everything is ok you can check it with dig:
[vagrant@kops ~]$ dig NS k8s.ifritltd.net -bash: dig: command not found
Obviously first need to install it:
[vagrant@kops ~]$ sudo yum -y install dig Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.ukfast.co.uk * epel: es-mirrors.evowise.com * extras: mirror.netw.io * updates: www.mirrorservice.org No package dig available. Error: Nothing to do
Of course it would be naive to expect that, lets see which package has it:
[vagrant@kops ~]$ yum whatprovides '*bin/dig' Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.ukfast.co.uk * epel: mirror.vorboss.net * extras: mirrors.ukfast.co.uk * updates: mirrors.coreix.net 32:bind-utils-9.9.4-50.el7.x86_64 : Utilities for querying DNS name servers Repo : base Matched from: Filename : /usr/bin/dig
Now let’s install it:
[vagrant@kops ~]$ sudo yum -y install bind-utils
And finally lets check it:
[vagrant@kops ~]$ dig NS k8s.ifritltd.net ; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.1 <<>> NS k8s.ifritltd.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17688 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 9 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;k8s.ifritltd.net. IN NS ;; ANSWER SECTION: k8s.ifritltd.net. 300 IN NS ns-855.awsdns-42.net. k8s.ifritltd.net. 300 IN NS ns-1094.awsdns-08.org. k8s.ifritltd.net. 300 IN NS ns-1901.awsdns-45.co.uk. k8s.ifritltd.net. 300 IN NS ns-29.awsdns-03.com. ;; ADDITIONAL SECTION: ns-1094.awsdns-08.org. 300 IN A 205.251.196.70 ns-1094.awsdns-08.org. 300 IN AAAA 2600:9000:5304:4600::1 ns-1901.awsdns-45.co.uk. 300 IN A 205.251.199.109 ns-1901.awsdns-45.co.uk. 300 IN AAAA 2600:9000:5307:6d00::1 ns-29.awsdns-03.com. 300 IN A 205.251.192.29 ns-29.awsdns-03.com. 300 IN AAAA 2600:9000:5300:1d00::1 ns-855.awsdns-42.net. 300 IN A 205.251.195.87 ns-855.awsdns-42.net. 300 IN AAAA 2600:9000:5303:5700::1 ;; Query time: 29 msec ;; SERVER: 10.0.2.3#53(10.0.2.3) ;; WHEN: Sun Dec 17 14:03:37 UTC 2017 ;; MSG SIZE rcvd: 357
If you messed it up, or before NS record is added to root domain, you should not get answers:
[vagrant@kops ~]$ dig NS k8s.ifritltd.net ; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.1 <<>> NS k8s.ifritltd.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 24172 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1280 ;; QUESTION SECTION: ;k8s.ifritltd.net. IN NS ;; AUTHORITY SECTION: ifritltd.net. 600 IN SOA ns-1462.awsdns-54.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 384 msec ;; SERVER: 10.0.2.3#53(10.0.2.3) ;; WHEN: Sun Dec 17 14:24:40 UTC 2017 ;; MSG SIZE rcvd: 130 [vagrant@kops ~]$
if you still get this, it means something went wrong, go double check, google etc, fix it.
4. Install and configure kubectl.
One of the requirements of kops is to have kubectl installed, even if it wasn’t we still would need it in order to deploy our containers to k8s. So let’s install it:
[vagrant@kops ~]$ curl -LO \ https://storage.googleapis.com/kubernetes-release/release/$(curl -s \ https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/kubectl
Lets test it is working:
vagrant@kops ~]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
All good and we are ready to provision our cluster. At least if you don’t stumble across one issue, lets look what is that.
So, in order to run a command to create a k8s cluster, kops needs to know availability zone of your cluster, in order to get it
I used aws cli:
[vagrant@kops ~]$ aws ec2 describe-availability-zones --region eu-west-2 An error occurred (AuthFailure) when calling the DescribeAvailabilityZones operation: AWS was not able to validate the provided access credentials [vagrant@kops ~]$
Very weird, thankfully running another command saved me from googling further:
[vagrant@kops ~]$ aws s3 ls An error occurred (RequestTimeTooSkewed) when calling the ListBuckets operation: The difference between the request time and the current time is too large.
as now it is obvious why aws cli doesn’t work, that is because we are using VM and it’s time is not in synch, let’s fix it:
[vagrant@kops ~]$ sudo yum install -y ntp ntpdate ntp-doc [vagrant@kops ~]$ sudo ntpdate pool.ntp.org 17 Dec 15:04:41 ntpdate[2888]: step time server 195.219.205.9 offset 1527.918257 sec
Let’s give it another try:
[vagrant@kops ~]$ aws ec2 describe-availability-zones --region eu-west-2
{
"AvailabilityZones": [
{
"State": "available",
"ZoneName": "eu-west-2a",
"Messages": [],
"RegionName": "eu-west-2"
},
{
"State": "available",
"ZoneName": "eu-west-2b",
"Messages": [],
"RegionName": "eu-west-2"
}
]
}
finally, so we have eu-west-2a and eu-west-2b, I will only use one of them, let’s try to create a cluster:
vagrant@kops ~]$ kops create cluster \ --name=k8s.ifritltd.net --state=s3://kayan-kops-state \ --zones="eu-west-2a" --node-count=2 --node-size=t2.micro --master-size=t2.micro --dns-zone=k8s.ifritltd.net --cloud aws error reading SSH key file "/home/vagrant/.ssh/id_rsa.pub": open /home/vagrant/.ssh/id_rsa.pub: no such file or directory [vagrant@kops ~]$
again, this time I forgotten that kops also needs ssh key pair, indeed, how otherwise we will be able to connect to EC2 instances once they are created:
[vagrant@kops ~]$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/vagrant/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/vagrant/.ssh/id_rsa. Your public key has been saved in /home/vagrant/.ssh/id_rsa.pub. The key fingerprint is: SHA256:x2+CADMjZZPipCjgB0NuuQaVSSKr1jQUkLdz0c0HTAA vagrant@kops The key's randomart image is: +---[RSA 2048]----+ |++===Eo.*o. | |+BB+.o . + . | |*B=+* . . | |B.==.* . | |o+..o . S o | |o . o . | | . . o | | o | | | +----[SHA256]-----+ [vagrant@kops ~]$
another try:
[vagrant@kops ~]$ vagrant@kops ~]$ kops create cluster \
--name=k8s.ifritltd.net --state=s3://kayan-kops-state \
--zones="eu-west-2a" --node-count=2 --node-size=t2.micro
--master-size=t2.micro --dns-zone=k8s.ifritltd.net --cloud aws
I1217 15:19:38.887426 2919 create_cluster.go:971] Using SSH public key: /home/vagrant/.ssh/id_rsa.pub
I1217 15:19:39.699575 2919 subnets.go:184] Assigned CIDR 172.20.32.0/19 to subnet eu-west-2a
Previewing changes that will be made:
I1217 15:19:42.256098 2919 executor.go:91] Tasks: 0 done / 73 total; 31 can run
I1217 15:19:44.006091 2919 executor.go:91] Tasks: 31 done / 73 total; 24 can run
I1217 15:19:44.577779 2919 executor.go:91] Tasks: 55 done / 73 total; 16 can run
I1217 15:19:44.824853 2919 executor.go:91] Tasks: 71 done / 73 total; 2 can run
I1217 15:19:44.910682 2919 executor.go:91] Tasks: 73 done / 73 total; 0 can run
Will create resources:
AutoscalingGroup/master-eu-west-2a.masters.k8s.ifritltd.net
MinSize 1
MaxSize 1
Subnets [name:eu-west-2a.k8s.ifritltd.net]
Tags {Name: master-eu-west-2a.masters.k8s.ifritltd.net, KubernetesCluster: k8s.ifritltd.net, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: master-eu-west-2a, k8s.io/role/master: 1}
LaunchConfiguration name:master-eu-west-2a.masters.k8s.ifritltd.net
AutoscalingGroup/nodes.k8s.ifritltd.net
MinSize 2
MaxSize 2
Subnets [name:eu-west-2a.k8s.ifritltd.net]
Tags {k8s.io/role/node: 1, Name: nodes.k8s.ifritltd.net, KubernetesCluster: k8s.ifritltd.net, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: nodes}
LaunchConfiguration name:nodes.k8s.ifritltd.net
DHCPOptions/k8s.ifritltd.net
DomainName eu-west-2.compute.internal
DomainNameServers AmazonProvidedDNS
...
.....
.......
..........
Cluster configuration has been created.
Suggestions:
* list clusters with: kops get cluster
* edit this cluster with: kops edit cluster k8s.ifritltd.net
* edit your node instance group: kops edit ig --name=k8s.ifritltd.net nodes
* edit your master instance group: kops edit ig --name=k8s.ifritltd.net master-eu-west-2a
Finally configure your cluster with: kops update cluster k8s.ifritltd.net --yes
[vagrant@kops ~]$
After a long list of resources it is going to create, kops suggest what we can do next, it is similar to terraform’s plan command, which is used to provision AWS resources, if you don’t know about it, I have a nice short post about how to provision AWS ec2 with terraform here
So, I suppose we can go ahead and finally make everything happen, let’s copy paste a suggestion from kops:
[vagrant@kops ~]$ kops update cluster k8s.ifritltd.net --yes State Store: Required value: Please set the --state flag or export KOPS_STATE_STORE. A valid value follows the format s3://<bucket>. A s3 bucket is required to store cluster state information.
ah, we need to provide s3 bucket as well:
[vagrant@kops ~]$ kops update cluster k8s.ifritltd.net --yes --state=s3://kayan-kops-state I1217 15:24:19.458665 2930 executor.go:91] Tasks: 0 done / 73 total; 31 can run I1217 15:24:20.777084 2930 vfs_castore.go:430] Issuing new certificate: "ca" I1217 15:24:20.781929 2930 vfs_castore.go:430] Issuing new certificate: "apiserver-aggregator-ca" I1217 15:24:21.852334 2930 executor.go:91] Tasks: 31 done / 73 total; 24 can run I1217 15:24:23.306964 2930 vfs_castore.go:430] Issuing new certificate: "kubecfg" I1217 15:24:23.518178 2930 vfs_castore.go:430] Issuing new certificate: "kube-controller-manager" I1217 15:24:23.955319 2930 vfs_castore.go:430] Issuing new certificate: "apiserver-proxy-client" I1217 15:24:23.986033 2930 vfs_castore.go:430] Issuing new certificate: "kube-proxy" I1217 15:24:24.202773 2930 vfs_castore.go:430] Issuing new certificate: "kubelet" I1217 15:24:24.215050 2930 vfs_castore.go:430] Issuing new certificate: "apiserver-aggregator" I1217 15:24:24.263349 2930 vfs_castore.go:430] Issuing new certificate: "kops" I1217 15:24:24.270783 2930 vfs_castore.go:430] Issuing new certificate: "master" I1217 15:24:24.354607 2930 vfs_castore.go:430] Issuing new certificate: "kubelet-api" I1217 15:24:24.569538 2930 vfs_castore.go:430] Issuing new certificate: "kube-scheduler" I1217 15:24:26.903693 2930 executor.go:91] Tasks: 55 done / 73 total; 16 can run I1217 15:24:28.205377 2930 launchconfiguration.go:333] waiting for IAM instance profile "masters.k8s.ifritltd.net" to be ready I1217 15:24:28.293855 2930 launchconfiguration.go:333] waiting for IAM instance profile "nodes.k8s.ifritltd.net" to be ready I1217 15:24:38.827416 2930 executor.go:91] Tasks: 71 done / 73 total; 2 can run I1217 15:24:39.278717 2930 executor.go:91] Tasks: 73 done / 73 total; 0 can run I1217 15:24:39.278760 2930 dns.go:153] Pre-creating DNS records I1217 15:24:40.338846 2930 update_cluster.go:248] Exporting kubecfg for cluster kops has set your kubectl context to k8s.ifritltd.net Cluster is starting. It should be ready in a few minutes. Suggestions: * validate cluster: kops validate cluster * list nodes: kubectl get nodes --show-labels * ssh to the master: ssh -i ~/.ssh/id_rsa admin@api.k8s.ifritltd.net The admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS. * read about installing addons: https://github.com/kubernetes/kops/blob/master/docs/addons.md [vagrant@kops ~]$
73 tasks! wow, well done kops, btw every time I use it, with a newer release, there are more and more tasks are added, just imagine what would you do, should there be no kops!
Let’s check our cluster, if you don’t want to provide –state every time you can export it with KOPS_STATE_STORE:
[vagrant@kops ~]$ export KOPS_STATE_STORE=s3://kayan-kops-state [vagrant@kops ~]$ kops validate cluster Using cluster from kubectl context: k8s.ifritltd.net Validating cluster k8s.ifritltd.net INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-eu-west-2a Master t2.micro 1 1 eu-west-2a nodes Node t2.micro 2 2 eu-west-2a NODE STATUS NAME ROLE READY ip-172-20-46-134.eu-west-2.compute.internal node True ip-172-20-55-66.eu-west-2.compute.internal node True ip-172-20-62-164.eu-west-2.compute.internal master True Your cluster k8s.ifritltd.net is ready
Please note it might take some times before it is fully ready, especially if you create and delete and then recreate
as DNS records could be cached etc. In case it fails you may get these sort of errors:
vagrant@kops ~]$ kops validate cluster Using cluster from kubectl context: k8s.ifritltd.net Validating cluster k8s.ifritltd.net Validation Failed The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: 203.0.113.123. Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start. Cannot reach cluster's API server: unable to Validate Cluster: k8s.ifritltd.net
or when one of the nodes is not ready:
[vagrant@kops ~]$ kops validate cluster Using cluster from kubectl context: k8s.ifritltd.net Validating cluster k8s.ifritltd.net INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-eu-west-2a Master t2.micro 1 1 eu-west-2a nodes Node t2.micro 2 2 eu-west-2a NODE STATUS NAME ROLE READY ip-172-20-55-96.eu-west-2.compute.internal master True Validation Failed Ready Master(s) 1 out of 1. Ready Node(s) 0 out of 2. your nodes are NOT ready k8s.ifritltd.net
But once it is ready you can start playing with kubectl, let’s check our nodes and pods and then try to deploy something:
[vagrant@kops ~]$ kubectl cluster-info Kubernetes master is running at https://api.k8s.ifritltd.net KubeDNS is running at https://api.k8s.ifritltd.net/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. [vagrant@kops ~]$
[vagrant@kops ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-172-20-46-134.eu-west-2.compute.internal Ready node 8m v1.8.4 ip-172-20-55-66.eu-west-2.compute.internal Ready node 7m v1.8.4 ip-172-20-62-164.eu-west-2.compute.internal Ready master 9m v1.8.4 [vagrant@kops ~]$
[vagrant@kops ~]$ kubectl get pod --namespace=kube-system NAME READY STATUS RESTARTS AGE dns-controller-fc68bd97b-2ldv7 1/1 Running 0 9m etcd-server-events-ip-172-20-62-164.eu-west-2.compute.internal 1/1 Running 0 10m etcd-server-ip-172-20-62-164.eu-west-2.compute.internal 1/1 Running 0 10m kube-apiserver-ip-172-20-62-164.eu-west-2.compute.internal 1/1 Running 0 9m kube-controller-manager-ip-172-20-62-164.eu-west-2.compute.internal 1/1 Running 0 9m kube-dns-7f56f9f8c7-52qkx 3/3 Running 0 9m kube-dns-7f56f9f8c7-pvq8h 3/3 Running 0 7m kube-dns-autoscaler-f4c47db64-hg2x6 1/1 Running 0 9m kube-proxy-ip-172-20-46-134.eu-west-2.compute.internal 1/1 Running 0 8m kube-proxy-ip-172-20-55-66.eu-west-2.compute.internal 1/1 Running 0 7m kube-proxy-ip-172-20-62-164.eu-west-2.compute.internal 1/1 Running 0 9m kube-scheduler-ip-172-20-62-164.eu-west-2.compute.internal 1/1 Running 0 10m [vagrant@kops ~]$
As you can see all internal pods are running. Lets deploy something now:
vagrant@kops ~]$ kubectl run busybox --image=busybox --port 8080 \
-- sh -c "while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; \
echo 'smallest http server'; } | nc -l -p 8080; done"
deployment "busybox" created
Now we need to expose it:
vagrant@kops ~]$ kubectl expose deployment busybox --type=NodePort service "busybox" exposed
We need to find out the port it used when created a service:
[vagrant@kops ~]$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE busybox NodePort 100.70.222.39 <none> 8080:31155/TCP 7s
As you can see, we have a container running and exposing port 8080, k8s exposed it on all nodes on port 31155,
given we have ssh keys we created earlier, we can now use it to connect to our nodes and check that very port:
You can see for all nodes there is a public IP address, so we can now check the port on that hosts:
[vagrant@kops ~]$ ssh admin@35.177.142.69 -i .ssh/id_rsa curl localhost:30822
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 21 0 21 0 0 3039 0 --:--:-- --:--:-- -smallest http server
-:--:-- 3500
[vagrant@kops ~]$
[vagrant@kops ~]$
[vagrant@kops ~]$ ssh admin@35.177.179.222 -i .ssh/id_rsa curl localhost:30822
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- smallest http server
100 21 0 21 0 0 1899 0 --:--:-- --:--:-- --:--:-- 2100
[vagrant@kops ~]$
[vagrant@kops ~]$
[vagrant@kops ~]$ ssh admin@52.56.234.121 -i .ssh/id_rsa curl localhost:30822
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 21 0 21 0 0 3259 0 --:--:-- --:--:--smallest http server
--:--:-- 3500
[vagrant@kops ~]$
As you can see it works.
Next we will expose the service through LoadBalancer, in the end of the day we deployed the whole thing
not just for NodePort testing as this could easily be tested with minikube locally in 2-3 steps.
So let’s expose the deployment through LoadBalancer:
vagrant@kops ~]$ kubectl expose deployment busybox --port 80 --target-port=8080 --type=LoadBalancer Error from server (AlreadyExists): services "busybox" already exists [vagrant@kops ~]$
because we already exposed busybox deployment, it creates a service with the same name as deployment, so when we try second
time it is reserved, so we need to explicitly set a different name:
[vagrant@kops ~]$ kubectl expose deployment busybox --name busyboxlb --port 80 --target-port=8080 --type=LoadBalancer service "busyboxlb" exposed
Now, lets check the load balancer address:
vagrant@kops ~]$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE busybox NodePort 100.68.181.5 <none> 8080:30822/TCP 47m busyboxlb LoadBalancer 100.68.59.82 <pending> 80:32473/TCP 3s kubernetes ClusterIP 100.64.0.1 <none> 443/TCP 1h .. ... [vagrant@kops ~]$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE busybox NodePort 100.68.181.5 <none> 8080:30822/TCP 54m busyboxlb LoadBalancer 100.68.59.82 a53bfecf2e354... 80:32473/TCP 6m kubernetes ClusterIP 100.64.0.1 <none> 443/TCP 1h [vagrant@kops ~]$
First it was still pending, later it was ready. Let’s describe the load balancer:
[vagrant@kops ~]$ kubectl describe svc busyboxlb Name: busyboxlb Namespace: default Labels: run=busybox Annotations: <none> Selector: run=busybox Type: LoadBalancer IP: 100.68.59.82 LoadBalancer Ingress: a53bfecf2e35411e7965c06578e71d83-433398422.eu-west-2.elb.amazonaws.com Port: <unset> 80/TCP TargetPort: 8080/TCP NodePort: <unset> 32473/TCP Endpoints: 100.96.2.2:8080 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal EnsuringLoadBalancer 7m service-controller Ensuring load balancer Normal EnsuredLoadBalancer 7m service-controller Ensured load balancer [vagrant@kops ~]$
let’s take the load balancer’s FQ name and send request:
vagrant@kops ~]$ curl a53bfecf2e35411e7965c06578e71d83-433398422.eu-west-2.elb.amazonaws.com smallest http server [vagrant@kops ~]$
Now our service is publicly accessible! Let’s create an alias busybox.k8s.ifritltd.net now for our load balancer, go to route53,
and in our subdomain, k8s.ifritltd.net, create a new recordset of type A, Alias=Yes,
click alias target and pick up you load balancer from list of ELB classic load balancers:
We can now query it by name:
vagrant@kops ~]$ curl busybox.k8s.ifritltd.net. smallest http server [vagrant@kops ~]$
That is it, we looked all basic steps required to provision production ready k8s cluster on AWS.
Finally, If you don’t want to pay for all those resources kops has provisioned, please dont’ forget to delete your cluster, as it is so easy to create a new one anyway 🙂
[vagrant@kops ~]$ kops delete cluster k8s.ifritltd.net --yes --state=s3://kayan-kops-state
5. Set up authentication in Kubernetes with AWS IAM Authenticator(heptio).
Now if you are setting up k8s cluster for production use, obviously default setup which is using x509 key/certificate for cluster authentication won’t be something secure that you want to use. If you are already using AWS, you can then setup up authentication for you cluster using AWS IAM Authenticator. The details are provided in the link.
Although using kops could help you to spin up your cluster quickly, if you want to go further and embark on a mission of setting up HA cluster for production use, you may well consider other tools, like kubeadm, in the link above you can find more advanced way of setting k8s cluster.