Skip to content

Automating Highly Available Kubernetes and external ETCD cluster setup with terraform and kubeadm on AWS.

Today I am going to show how you can fully automate the advanced process of setting up the highly available k8s cluster in the cloud. We will go through a set of terraform and bash scripts which should be sufficient enough for you to literally just run terraform plan/apply to get your HA etcd and k8s cluster up and running without any hassle around.

    Part 0 – Intro.
    Part 1 – Setting up HA ETCD cluster.
    Part 2 – The PKI infra
    Part 3 – Setting up k8s cluster.

Part 0 – Intro.

If you do a short research on how to setup k8s cluster you may find quite a lot of ways this could be achieved.
But in general, all this ways could be grouped into 3 types:

1) No setup
2) Easy Set up
3) Advanced Set up
4) Hard way

By No setup I simply mean something like EKS, it is a managed service, you don’t need to maintain or care about details while AWS will do all for you. Never used it can’t say much on that one.

Easy setup, tools like kops and alike make it quite easy – couple commands run kinda setup:

kops ~]$ kops create cluster \
  --name=k8s.ifritltd.net --state=s3://kayan-kops-state \
  --zones="eu-west-2a" --node-count=2 --node-size=t2.micro 
  --master-size=t2.micro --dns-zone=k8s.ifritltd.net  --cloud aws

All you need is setup s3 bucket and dns records and run the command above which I described two years ago in this article

The downside is first of all it is mainly only for AWS, and generates all AWS resources as it wants, so lets say it would generate security groups, asg, etc in it’s own way which means
if you already have terraform managed infra with your own rules, strategies and framework, it won’t feet into that model but just added as some kind of alien infra. Long story short if you want fine grained control over how your infra should be managed from single centralised terraform, it isn’t best solution, yet still easy and balanced tool.

Before I start explaining how to use Advanced Set up, I am just going to mention that 4th, The Hard way is probably only good if you want to learn how k8s works, how all components interact with each other, and as it doesn’t use any external tool to set up components, you do everything manually, you literally know all the guts of the system. Obviously it could become a nightmare to support such system in production unless all members of the ops team are k8s experts or there are some requirements not supported by other bootstrapping tools.

Finally the Advanced Set up.

While we are going to use kubeadm to setup HA k8s cluster, there are also two ways of doing that:
1) The Stacked etcd topology, where the etcd cluster is just part of k8s cluster and
2) External etcd topology:

So what is etcd – think of it like database, the data layer of k8s cluster. Obviously external ectd is more secure due to separation of data and service layers, can be even managed by other team, if needed.

If one day your cluster is broken, you literally can bring up a new one from the scratch and all the data of the api server will just be restored from etcd, hence this is the most preferred option. Whereas in the stacked version etcd is just a docker container running along with other components as part of control pane.

Before we start setting up our cluster, we need to set up etcd cluster in HA way, so our k8s cluster can use it as external database.

Part 1 – Setting up HA ETCD cluster.

As I have already mentioned I am going to use terraform, our cluster will need the next resources:

DNS
Ubuntu server running etcd server and PKI infra installed.
Disk volume to store the data permanently

So straight to the etc module code:

tail -n 1000 *.tf
==> data.tf <==
data "aws_ami" "ubuntu_1604" {
  most_recent = true
  name_regex  = "ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-[0-9]*"
}

data "aws_route53_zone" "zone" {
  name = "k8s.ifritltd.co.uk."
}

==> ebs.tf <==
resource "aws_ebs_volume" "ebs-volume" {
  availability_zone = "${var.availability_zone}"
  size              = "2"

  tags {
    Name        = "ebs_etcd_${var.zone_suffix}"
  }

  lifecycle {
    prevent_destroy = false
    //todo enable back when finished
  }
}

==> main.tf <==
resource "aws_instance" "etcd" {
  ami           = "${data.aws_ami.ubuntu_1604.id}"
  instance_type = "t2.micro"
  key_name      = "terra"
  user_data     = "${file("etcd.sh")}"
  availability_zone = "${var.availability_zone}"

  vpc_security_group_ids = [
    "${var.sg_id}",
  ]

  iam_instance_profile = "${var.iam_instance_profile}"

  tags {
    Name = "example etcd ${var.zone_suffix}"
  }
}


resource "aws_route53_record" "etcd" {
  zone_id = "${var.zone_id}"
  name    = "etcd-${var.zone_suffix}"
  type    = "A"
  ttl     = "300"
  records = ["${aws_instance.etcd.private_ip}"]
}

==> output.tf <==
output "public_ip" {
  value = "${aws_instance.etcd.public_ip}"
}


As you see from the code above we define an instance running Ubuntu 16.04 backed with ebs and register it’s ip with dns record.

Now we can use the module to create resources in all 3 AZs in London region:

➜  etcd git:(master) ✗ tail -n 1000 *.tf


==> main.tf <==
module "etcd-a" {
  source               = "module/etcd"
  availability_zone    = "eu-west-2a"
  zone_suffix          = "a"
  iam_instance_profile = "${aws_iam_instance_profile.aws_iam_instance_profile.name}"
  sg_id                = "${aws_security_group.etcd.id}"
  zone_id              = "${aws_route53_zone.k8s_private_zone.zone_id}"
}

module "etcd-b" {
  source               = "module/etcd"
  availability_zone    = "eu-west-2b"
  zone_suffix          = "b"
  iam_instance_profile = "${aws_iam_instance_profile.aws_iam_instance_profile.name}"
  sg_id                = "${aws_security_group.etcd.id}"
  zone_id              = "${aws_route53_zone.k8s_private_zone.zone_id}"
}

module "etcd-c" {
  source               = "module/etcd"
  availability_zone    = "eu-west-2c"
  zone_suffix          = "c"
  iam_instance_profile = "${aws_iam_instance_profile.aws_iam_instance_profile.name}"
  sg_id                = "${aws_security_group.etcd.id}"
  zone_id              = "${aws_route53_zone.k8s_private_zone.zone_id}"
}

resource "aws_route53_zone" "k8s_private_zone" {
  name = "k8s.ifritltd.co.uk"

  vpc {
    vpc_id = "${data.aws_vpc.default.id}"
  }
}


==> security-groups.tf <==

resource "aws_security_group_rule" "all-internal" {
  type              = "ingress"
  security_group_id = "${aws_security_group.etcd.id}"
  self              = true
  from_port         = 2379
  to_port           = 2380
  protocol          = "tcp"
}


As you can see we created 3 instances in private dns zone k8s.ifritltd.co.uk and the nodes exposed as https://etcd-a.k8s.ifritltd.co.uk:2379 in each zone.

You also need to grant some permission to your instances in the IAM:

 "ec2:DescribeVolumes",
 "ec2:AttachVolume",
 "ssm:PutParameter",
 "ssm:GetParametersByPath",
 "ssm:GetParameters",
 "ssm:GetParameter"

With this, ec2 can get ebs attached and use ssm for storing pki. The cloud init is probably most important part once infra is set up:

#!/bin/bash

set -euxo pipefail

sleep 30

AVAILABILITY_ZONE=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | cut -b 10)

apt-get update
apt-get -y install wget python-pip

locale-gen en_GB.UTF-8
pip install --no-cache-dir awscli

VOLUME_ID=$(aws ec2 describe-volumes --filters "Name=status,Values=available"  Name=tag:Name,Values=ebs_etcd_$AVAILABILITY_ZONE --query "Volumes[].VolumeId" --output text --region eu-west-2)

INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

aws ec2 attach-volume --region eu-west-2 \
              --volume-id "${VOLUME_ID}" \
              --instance-id "${INSTANCE_ID}" \
              --device "/dev/xvdf"

while [ -z $(aws ec2 describe-volumes --filters "Name=status,Values=in-use"  Name=tag:Name,Values=ebs_etcd_$AVAILABILITY_ZONE --query "Volumes[].VolumeId" --output text --region eu-west-2) ] ; do sleep 10; echo "ebs not ready"; done

sleep 5

if [[ -z $(blkid /dev/xvdf) ]]; then
  mkfs -t ext4 /dev/xvdf  
fi

mkdir -p /opt/etcd
mount /dev/xvdf /opt/etcd


ETCD_VERSION="v3.3.8"
ETCD_URL="https://github.com/coreos/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz"
ETCD_CONFIG=/etc/etcd


apt-get update
apt-get -y install wget python-pip
pip install --no-cache-dir awscli

useradd etcd

wget ${ETCD_URL} -O /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz
tar -xzf /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz -C /tmp
install --owner root --group root --mode 0755     /tmp/etcd-${ETCD_VERSION}-linux-amd64/etcd /usr/bin/etcd
install --owner root --group root --mode 0755     /tmp/etcd-${ETCD_VERSION}-linux-amd64/etcdctl /usr/bin/etcdctl
install -d --owner root --group root --mode 0755 ${ETCD_CONFIG}

cat > /etc/systemd/system/etcd.service <<EOF
[Unit]
Description=etcd key-value store

[Service]
User=etcd
Type=notify
ExecStart=/usr/bin/etcd --config-file=/etc/etcd/etcd.conf
Restart=always
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=ready.target

EOF

chmod 0644 /etc/systemd/system/etcd.service


mkdir -p /opt/etcd/data
chown -R etcd:etcd /opt/etcd


cat > /etc/etcd/etcd.conf <<EOF

name: 'etcd-AZONE.k8s.ifritltd.co.uk'
data-dir: /opt/etcd/data
listen-peer-urls: https://0.0.0.0:2380
listen-client-urls: https://0.0.0.0:2379
..
....
.....
initial-advertise-peer-urls: https://etcd-AZONE.k8s.ifritltd.co.uk:2380
advertise-client-urls: https://etcd-AZONE.k8s.ifritltd.co.uk:2379
discovery-fallback: 'proxy'
initial-cluster: 'etcd-a.k8s.ifritltd.co.uk=https://etcd-a.k8s.ifritltd.co.uk:2380,etcd-b.k8s.ifritltd.co.uk=https://etcd-b.k8s.ifritltd.co.uk:2380,etcd-c.k8s.ifritltd.co.uk=https://etcd-c.k8s.ifritltd.co.uk:2380'
...
....
...
client-transport-security:
  cert-file: /etc/ssl/server.pem
  key-file: /etc/ssl/server-key.pem
  client-cert-auth: false
  trusted-ca-file: /etc/ssl/certs/ca.pem
  auto-tls: false
peer-transport-security:
  cert-file: /etc/ssl/server.pem
  key-file: /etc/ssl/server-key.pem
  peer-client-cert-auth: false
  trusted-ca-file: /etc/ssl/certs/ca.pem
  auto-tls: false
....
......
EOF

sed -i s~AZONE~$AVAILABILITY_ZONE~g /etc/etcd/etcd.conf


aws ssm get-parameters --names "etcd-ca" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/ssl/certs/ca.pem 
aws ssm get-parameters --names "etcd-server" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/ssl/server.pem
aws ssm get-parameters --names "etcd-server-key" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/ssl/server-key.pem

chmod 0600  /etc/ssl/server-key.pem
chmod 0644 /etc/ssl/server.pem
chown etcd:etcd /etc/ssl/server-key.pem
chown etcd:etcd /etc/ssl/server.pem

systemctl enable etcd
systemctl start etcd


So what is happening here. First of all, we install some tools like python, pip, awscli. Then we attach EBS volume, create filesystem on it then mount it to /opt/etcd, same location we later specify in etcd config as data folder. We also advertise peers in the config by replacing AZONE with AZ where instance is running, and finally, we pull PKI infra, which we need to preinstall prior to spinning up our instances. The less important steps in etcd config are skipped but available in the source code

Part 2 – The PKI infra

The whole process is described in here
https://coreos.com/os/docs/latest/generate-self-signed-certificates.html
https://coreos.com/etcd/docs/latest/getting-started-with-etcd.html
Here are the steps to reproduce it:


brew install cfssl

cfssl print-defaults config > ca-config.json
cfssl print-defaults csr > ca-csr.json

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

cfssl print-defaults csr > server.json
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

And the config files used:


 tail  -n 100 *.json
==> ca-config.json <==
{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "server": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

==> ca-csr.json <==
{
    "CN": "IfritLTD CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "O": "IfritLTD",
            "ST": "San Francisco",
            "OU": "Org Unit 1",
            "OU": "Org Unit 2"
        }
    ]
}

==> server.json <==
{
    "CN": "etcd.k8s.ifritltd.co.uk",
    "hosts": [
        "etcd1.k8s.ifritltd.co.uk",
        "etcd2.k8s.ifritltd.co.uk",
        "etcd3.k8s.ifritltd.co.uk"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "ST": "CA",
            "L": "San Francisco"
        }
    ]
}

If you fancy you can also use openssl for the above, the important thing you need to generate CA, then cert and server key signed by CA and having 3 SANs per node, once you done you will have bunch of files:

ll
total 80
-rw-r--r--  1 kayanazimov  staff   832B 25 Nov 17:24 ca-config.json
-rw-r--r--@ 1 kayanazimov  staff   307B 25 Nov 17:31 ca-csr.json
-rw-------  1 kayanazimov  staff   1.6K 25 Nov 17:32 ca-key.pem
-rw-r--r--  1 kayanazimov  staff   1.0K 25 Nov 17:32 ca.csr
-rw-r--r--  1 kayanazimov  staff   1.3K 25 Nov 17:32 ca.pem
-rw-r--r--  1 kayanazimov  staff     0B 25 Nov 18:48 cacert.pem
-rw-r--r--  1 kayanazimov  staff   2.1K 25 Nov 17:52 cert.crt
-rw-------  1 kayanazimov  staff   227B 25 Nov 18:16 server-key.pem
-rw-r--r--  1 kayanazimov  staff   586B 25 Nov 18:16 server.csr
-rw-r--r--@ 1 kayanazimov  staff   357B 25 Nov 18:13 server.json
-rw-r--r--  1 kayanazimov  staff   1.2K 25 Nov 18:16 server.pem


You only needs some of these files stored by AWS ssm so etcd and later k8s cluster can read them and use.

Right, time to run some terraform and ssh to instance, once on the instance you can test server logs with journalctl and test api as well:


curl  -k https://127.0.0.1:2379/v2/keys/message -XPUT -d value="Hello"
curl  -k https://127.0.0.1:2379/v2/keys/message

etcdctl --endpoints=https://etcd-a.k8s.ifritltd.co.uk:2379  get / --prefix --keys-only

Try to write something in one node and access it in the other to test all good.

Part 3 – Setting up k8s cluster

As with etcd, we first need to setup some resources with terraform:


resource "aws_instance" "master_node1" {
  ami           = "${data.aws_ami.ubuntu_1604.id}"
  instance_type = "t2.micro"
  key_name      = "terra"
  user_data     = "${file("userdata_master_init.sh")}"

  vpc_security_group_ids = [
    "${aws_security_group.kubernetes_sg.id}",
  ]

  availability_zone    = "eu-west-2a"
  iam_instance_profile = "${aws_iam_instance_profile.aws_iam_instance_profile.name}"

  tags {
    Name = "example k8s master1"
  }
}
...

resource "aws_instance" "master_node2" {
..
resource "aws_instance" "master_node3" {
..
...


resource "aws_instance" "slave_node1" {
  ami           = "${data.aws_ami.ubuntu_1604.id}"
  instance_type = "t2.micro"
  key_name      = "terra"
  user_data     = "${file("userdata_node.sh")}"

  vpc_security_group_ids = [
    "${aws_security_group.kubernetes_sg.id}",
  ]

  availability_zone    = "eu-west-2a"
  iam_instance_profile = "${aws_iam_instance_profile.aws_iam_instance_profile.name}"

  tags {
    Name = "example k8s slave_node1"
  }
}

resource "aws_instance" "slave_node2" {
...
.....
resource "aws_instance" "slave_node3" {
....

So we spin up 3 masters and 3 nodes. In theory we can spin up as many masters and nodes we want, obviously in real prod environment we will use asg to ensure availability of each master and some cloud watch rules to scale nodes when load increases.

Then we hide the master nodes api server behind load balancer:

resource "aws_elb" "api_load_balancer" {
  name = "k8s-api"

  internal = true

  subnets = ["subnet-4e89bb03", "subnet-52ce2228", "subnet-892a81e0" ]

  instances = ["${aws_instance.master_node1.id}", "${aws_instance.master_node2.id}", "${aws_instance.master_node3.id}"]

  listener {
    instance_port     = "${var.listener_port}"
    instance_protocol = "${var.lb_protocol}"
    lb_port           = "${var.lb_port}"
    lb_protocol       = "${var.lb_protocol}"
  }

  health_check {
    interval            = "${var.hc_interval}"
    healthy_threshold   = "${var.hc_healthy_threshold}"
    unhealthy_threshold = "${var.hc_unhealthy_threshold}"
    target              = "${var.hc_target}"
    timeout             = "${var.hc_timeout}"
  }

  security_groups = ["${aws_security_group.kubernetes_sg.id}"]

  tags = [
    {
      key   = "KubernetesCluster"
      value = "kubernetes"
    },
  ]
}

data "aws_route53_zone" "k8s_zone" {
  name         = "k8s.ifritltd.co.uk."
  private_zone = true
}

resource "aws_route53_record" "kubernetes" {
  name    = "kubernetes"
  type    = "A"
  zone_id = "${data.aws_route53_zone.k8s_zone.zone_id}"

  alias {
    name                   = "${aws_elb.api_load_balancer.dns_name}"
    zone_id                = "${aws_elb.api_load_balancer.zone_id}"
    evaluate_target_health = false
  }
}

As we already created our dns zone in etcd terraform we just refer to it here to create a record for api server’s load balancer, which we will use to refer to our cluster from external world.

Don’t pay attention to VPC and subnets, for simplicity sake I just stick to defaults. For security groups please refer to next table

The most important magic happens in cloud init files. Let’s start with masters ones, that is userdata_master_init.sh and userdata_master_join.sh, why two? Because starting from N version of kubeadm init-master config is slightly different from other joiners-masters.

Right, all in order, first of all we install docker! You didn’t know it? Oh yes, k8s uses Docker underneath, not always, so other container run-times are also available:


#!/bin/bash

set -euxo pipefail

# install docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable"
apt-get update && apt-get install -y  docker-ce=18.06.0~ce~3-0~ubuntu

# configure docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

# setup systemd docker service drop-in directory
mkdir -p /etc/systemd/system/docker.service.d

systemctl daemon-reload
systemctl restart docker
..
....

Once docker is ready, we go on and install kubectl – a tool to control our cluster, kubeadm – a tool to bootstrap the cluster, finally kubelet – part of the cluster, the node daemon, which talks to API server to find containers that should be running on its node:

...
......
systemctl restart docker

# K8S SETUP

# install required k8s tools
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat >/etc/apt/sources.list.d/kubernetes.list <<EOF
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

# all list https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages
K8S_VERSION=1.12.3-00
apt-get update
apt-get install -y kubelet=${K8S_VERSION} kubeadm:amd64=${K8S_VERSION} kubectl:amd64=${K8S_VERSION} python-pip
apt-mark hold kubelet kubeadm kubectl

locale-gen en_GB.UTF-8
pip install --no-cache-dir awscli

systemctl daemon-reload
systemctl restart kubelet
 

As soon as kubelet is up and running, if you check it’s logs:

journalctl -u kubelet -f

You will see that kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. Before we run kubeadm, we need to configure it:

...
....
systemctl restart kubelet

mkdir -p /etc/kubernetes/pki/etcd

aws ssm get-parameters --names "etcd-ca" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/kubernetes/pki/etcd/ca.crt
aws ssm get-parameters --names "etcd-server" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/kubernetes/pki/apiserver-etcd-client.crt
aws ssm get-parameters --names "etcd-server-key" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/kubernetes/pki/apiserver-etcd-client.key

# for initial master
cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- "kubernetes.k8s.ifritltd.co.uk"
controlPlaneEndpoint: "kubernetes.k8s.ifritltd.co.uk"
etcd:
    external:
        endpoints:
        - https://etcd-a.k8s.ifritltd.co.uk:2379
        - https://etcd-b.k8s.ifritltd.co.uk:2379
        - https://etcd-c.k8s.ifritltd.co.uk:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
networking:
    podSubnet: "192.168.0.0/16"
EOF


..
.....

First we pull the PKI infra we created in etcd, then create kubeadm config file with reference to etcd nodes and network for our pod subnet.
Once we are ready we can initiate our first master node!

...
....

kubeadm init --config kubeadm-config.yaml --ignore-preflight-errors=all

..

After this step kubeadm will start configuring our cluster, that is pulling all docker images and telling kubelet what to do.
At this moment we can see docker images pulled by kubeadm:

docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                v1.12.2             15e9da1ca195        13 days ago         96.5MB
k8s.gcr.io/kube-apiserver            v1.12.2             51a9c329b7c5        13 days ago         194MB
k8s.gcr.io/kube-controller-manager   v1.12.2             15548c720a70        13 days ago         164MB
k8s.gcr.io/kube-scheduler            v1.12.2             d6d57c76136c        13 days ago         58.3MB
k8s.gcr.io/coredns                   1.2.2               367cdc8433a4        2 months ago        39.2MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        10 months ago       742kB
root@ip-172-31-19-60:~#

If you were running stacked etcd then it also would appear in this list. Let’s check further:


kubectl get pod -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
coredns-576cbf47c7-mnd89                  1/1     Running   0          19m
coredns-576cbf47c7-sh4z8                  1/1     Running   0          19m
kube-apiserver-ip-172-31-19-60            1/1     Running   0          18m
kube-controller-manager-ip-172-31-19-60   1/1     Running   0          18m
kube-proxy-d7r49                          1/1     Running   0          19m
kube-scheduler-ip-172-31-19-60            1/1     Running   0          18m

As you can see our cluster is already running and ready to some extend.

Next is configuring calico for networking security:

# install calico
kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Once this steps done, our cluster should be up and running:


kubectl get node
NAME              STATUS     ROLES    AGE     VERSION
ip-172-31-19-60   NotReady   master   8m43s   v1.12.2
root@ip-172-31-19-60:~#

Just wait until master is ready…

And then next step is to prepare some stuff for next masters:


aws ssm put-parameter --name "k8s-ca" --value "$(cat /etc/kubernetes/pki/ca.crt)"  --type "SecureString" --region eu-west-2 --overwrite 
aws ssm put-parameter --name "k8s-ca-key" --value "$(cat /etc/kubernetes/pki/ca.key)"  --type "SecureString" --region eu-west-2 --overwrite 
aws ssm put-parameter --name "k8s-sa" --value "$(cat /etc/kubernetes/pki/sa.pub)"  --type "SecureString" --region eu-west-2 --overwrite 
aws ssm put-parameter --name "k8s-sa-key" --value "$(cat /etc/kubernetes/pki/sa.key)"  --type "SecureString" --region eu-west-2 --overwrite 
aws ssm put-parameter --name "k8s-front-proxy-ca" --value "$(cat /etc/kubernetes/pki/front-proxy-ca.crt)"  --type "SecureString" --region eu-west-2 --overwrite 
aws ssm put-parameter --name "k8s-front-proxy-ca-key" --value "$(cat /etc/kubernetes/pki/front-proxy-ca.key)"  --type "SecureString" --region eu-west-2 --overwrite 

aws ssm put-parameter --name "k8s-init-token" --value "$(kubeadm token create)"  --type "SecureString" --region eu-west-2 --overwrite 
aws ssm put-parameter --name "k8s-init-token-hash" --value "$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
 openssl dgst -sha256 -hex sed 's/^.* //')"  --type "SecureString" --region eu-west-2 --overwrite 

Here we store all PKI created by master to AWS ssm so next master can reuse it, we also store token and hash for joiner master.

Now we can spin up another N master:

..
.....
# wait for master node
while [ "None" = "$(aws ssm get-parameters --names 'k8s-init-token' --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2)" ];do echo "waiting for init master"; sleep 5;done
 
aws ssm get-parameters --names "k8s-ca" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/kubernetes/pki/ca.crt
aws ssm get-parameters --names "k8s-ca-key" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/kubernetes/pki/ca.key
aws ssm get-parameters --names "k8s-sa" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/kubernetes/pki/sa.pub
aws ssm get-parameters --names "k8s-sa-key" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/kubernetes/pki/sa.key
aws ssm get-parameters --names "k8s-front-proxy-ca" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/kubernetes/pki/front-proxy-ca.crt
aws ssm get-parameters --names "k8s-front-proxy-ca-key" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2 > /etc/kubernetes/pki/front-proxy-ca.key

TOKEN=$(aws ssm get-parameters --names "k8s-init-token" --query '[Parameters[0].Value]' --output text --with-decryption  --region eu-west-2)
TOKEN_HASH=$(aws ssm get-parameters --names "k8s-init-token-hash" --query '[Parameters[0].Value]' --output text  --with-decryption --region eu-west-2)

kubeadm join kubernetes.k8s.ifritltd.co.uk:6443 --token $TOKEN --discovery-token-ca-cert-hash sha256:$TOKEN_HASH --experimental-control-plane


I skipped the part which is similar to master-initiator, docker install etc, significant part is joiner master is waiting until token is created by first alpha master, so it can join the existing cluster with kubeadm join command.

Finally when you masters are running, you can join any number of slaves to the cluster:

....
kubeadm join kubernetes.k8s.ifritltd.co.uk:6443 --token $TOKEN --discovery-token-ca-cert-hash sha256:$TOKEN_HASH 

The cloud init is just like for master-joiner, but kubeadm init without –experimental-control-plane parameter.

Lets get nodes:

oot@ip-172-31-23-145:~# kubectl get nodes
NAME               STATUS   ROLES    AGE     VERSION
ip-172-31-23-145   Ready    master   10m     v1.12.2
ip-172-31-23-146   Ready    master   17m     v1.12.2
ip-172-31-29-77    Ready    <none>   7m20s   v1.12.2
root@ip-172-31-23-145:~#

You should see something similar, then consequently more nodes will just join the cluster as you spin them up. Once we finish our bootstrapping process, we should have 9 nodes in total, 3 per zone for etcd, 3 for masters and finally 3 for slaves. 9 nodes in total.

While the actual code to bootstrap the cluster may not be quite complete for production use, the approach I used is satisfactory in my opinion for automating the bootstrapping process of HA Kubernetes cluster with external etcd.

Now that you are familiar with bootstrapping process, you can clone and spin up the cluster using my git repo:

HA etcd: https://github.com/kenych/terraform_exs/tree/master/etcd
HA k8s: https://github.com/kenych/terraform_exs/tree/master/k8s_ha

Please note I am using the next versions:

K8S_VERSION=1.12.3-00
ETCD_VERSION=v3.3.8

Normally pinned version means stable code, so I really hope it is just going to work next time you run it. The only manual step is to create and store initial PKI infra which is very straightforward anyway.