DevOps Principles & Best Practices by Kayan Azimov
I just ran my tool to get the-most-demanded-devops-skills-stats and noticed a small (or may be not that small) change:
jobstats -c -2 | head -20 azure 212 aws 194 kubernetes 66 terraform 56 gcp 55 git 48 docker 43 python 40 ci/cd 30 jenkins 28 linux 18 bash 10 ansible 10 scripting 9 unix 3 puppet 3 groovy 2 elk 2 chef 2 zabbix 1Comments closed
This blog post demonstrates how anything in Jenkins could be configured as a code through Java API using groovy code, and how changes could be applied right inside Jenkins job. I particularly will demo how to configure Kubernetes plugin and credentials, but the same concept could be used later to configure any Jenkins plugin you are interested in. We will also look at how to create custom config which could be used either for all
or only specific Jenkins instances so you can setup different instances differently based on security policy or any other criteria.
The Why…
Recently I have been working on a task to improve deployment of our master Jenkins instances on Kubernetes.
On of the requirements was to improve the speed, as we have more than 40 Jenkins masters running on different
environments like test, dev, pre-prod, perf, prod etc and deployed in Kubernetes over AWS cluster. The deployment job took around an hour, involved downtime and required multiple steps.
I was curious the other day, what is the most demanded devops skills out there on the market?
Not that I didn’t have any clue, as for someone who has been in the industry for a while it is kinda obvious, but sometimes you simply curious or just want to get some sort of stats. So after couple googling attempts which didn’t give any reasonable results apart from boring marketing ads and stupid suggestions like soft skill (who cares!), I decided that best approach would be DIY!
So here is what I did, step by step
1) Went to the web site many have probably used to find a job and put some search criteria
Then switched to Classic View and changed summary to 200 jobs per page, which is the max. Now all I needed is to find all occurrences of some keyword on the resulting page. (this manual part would benefit from selenium/phantomjs if run regularly, I probably will add it later)
2) Obviously I didn’t want to count manually, so I decided hey let’s do it with curl and then scan the output with some predefined keywords. Initially the keywords file was too big, then I skipped some stuff as it appeared to be not that popular(1 or 2 occurrences). But in general the file needs to be maintained as overtime some new kids in the block will pop out. So here the list in the words.txt file:
➜ trendystuff cat words.txt aws azure terraform ansible puppet chef docker kubernetes mesos jenkins ci/cd elasticsearch kibana logstash elk prometheus openstack zabbix vault linux scripting unix bash python groovy ruby git maven ➜ trendystuff
3) Now let’s write some super simple dummy bash script to get though the output and make some stats:
➜ scrips cat jobstats #!/bin/bash if [[ $@ != **-c** ]]; then curl -s $1 > output.txt fi if [[ $@ == **-2** ]]; then sort_arg=" -k 2 -r" fi rm result.txt for word in `cat words.txt`; do echo "$word `grep -io $word output.txt \ | wc -l`" \ | xargs >> result.txt done; cat result.txt | sort -n $sort_arg% ➜ scrips
4) Finally let’s run it, we have to copy the url from the website, which will be generated once you put you search criteria and press search, and pass it as argument to the script:
Comments closed
In the 1st part of this tutorial we looked at how to dockerize installation of the Jenkins plugins, java and maven tool setup in Jenkins 2 and created declarative build pipeline for maven project with test and SonarQube stages. In this part we will focus on deployment part.
Couldn’t we just simply add another stage for deployment in part 1, you may ask? Well, in fact deployment requires quite a few steps to be taken, including maven pom and settings file configuration, artifact repository availability, repository credentials encryption, etc. Let’s add them to the list and then implement step by step like we did in previous session.
If you are already familiar with 1st part of this tutorial, created your project from the scratch and using your own repository, then you can just follow the steps as we go further, otherwise, if you are starting now, you can just clone/fork the work we did in the last example and then add changes as they follow in the tutorial:
git clone https://github.com/kenych/jenkins_docker_pipeline_tutorial1 && cd jenkins_docker_pipeline_tutorial1 && ./runall.sh
Please note all steps have been tested on MacOS Sierra and Docker version 17.05.0-ce and you should change them accordingly if you are using MS-DOS, FreeBSD etc 😉
The script above is going to take a while as it is downloading java 7, java 8, maven, sonarqube and jenkins docker images, so please be patient 🙂 Once done you should have Jenkins and Sonar up and running as we created in part 1:
If you got errors about some port being busy just use the free ports from your host, I explain this here. Otherwise you can use dynamic ports which is shown a bit later.
So let’s look at the first step. Obviously if we want to test the deployment in our example, we need some place to deploy our artifacts to. We are going to use a limited open source version of JFrog Artifactory called “artifactory oss”. Let’s run it on Docker to see how easy it is to have your own artifact repo. The port 8081 on machine was busy, so I had to run it on 8082, you should do according to free ports available on your machine:
docker run --rm -p 8082:8081 --name artifactory docker.bintray.io/jfrog/artifactory-oss:5.4.4
Alternatively you can use dynamic ports.
Comments closed
This is a practical example, so be ready to get your hands dirty. You can either follow this step by step guide, which would be really good for learning purposes and we will create everything from the scratch, or if you are lazy, just run the command bellow after reading for the demo:
git clone https://github.com/kenych/jenkins_docker_pipeline_tutorial1 && cd jenkins_docker_pipeline_tutorial1 && ./runall.sh
but by the end you should be able to run the pipeline on fully automated Jenkins Docker container.
As you may already know, with Jenkins 2 you can actually have your build pipeline right within your java project. So you can actually use your own maven java project in order to follow the steps in this article as long as it is hosted on a git repository.
Everything obviously will be running on Docker as it is the easiest way of deploying and running them.
So, let’s see how to run Jenkins on Docker
docker pull jenkins:2.60.1
While it is downloading in the background let’s see what we are going to do with it once it is done.
Default Jenkins comes quite naked and shows suggested plugins installation wizard. We will choose it, then we will capture all installed plugins and then automate this manual step in Docker image and will follow this simple rule throughout the all steps:
The image we are going to download is 600M so you can prepare yourself coffee and have couple sips before it is finished and I will take you through the steps we need to setup up build pipeline for java project. Let’s add them to the list and then look closer later:
Once you have your image downloaded let’s run the container:
docker run -p 8080:8080 --rm --name myjenkins jenkins:2.60.1
Please note I used a specific tag, I am not using latest tag, which is the default if you don’t specify one, as I don’t want anything to break in the future.
Also note we name the container so it is easier to refer to it later as otherwise docker will name it randomly and we added –-rm flag to delete container once we stop it, this will ensure we are running Jenkins in an immutable fashion and everything configured on the fly, and if we want to preserve any data we will do it explicitly.
Comments closed
