In the 1st part of this tutorial we looked at how to dockerize installation of the Jenkins plugins, java and maven tool setup in Jenkins 2 and created declarative build pipeline for maven project with test and SonarQube stages. In this part we will focus on deployment part.
Couldn’t we just simply add another stage for deployment in part 1, you may ask? Well, in fact deployment requires quite a few steps to be taken, including maven pom and settings file configuration, artifact repository availability, repository credentials encryption, etc. Let’s add them to the list and then implement step by step like we did in previous session.
- Running JFrog Artifactory on Docker
- Configuring maven pom file
- Configuring maven settings file
- Using Config File Provider Plugin for persistence of maven settings
- Dockerizing the installation and configuration process
If you are already familiar with 1st part of this tutorial, created your project from the scratch and using your own repository, then you can just follow the steps as we go further, otherwise, if you are starting now, you can just clone/fork the work we did in the last example and then add changes as they follow in the tutorial:
git clone https://github.com/kenych/jenkins_docker_pipeline_tutorial1 && cd jenkins_docker_pipeline_tutorial1 && ./runall.sh
Please note all steps have been tested on MacOS Sierra and Docker version 17.05.0-ce and you should change them accordingly if you are using MS-DOS, FreeBSD etc 😉
The script above is going to take a while as it is downloading java 7, java 8, maven, sonarqube and jenkins docker images, so please be patient 🙂 Once done you should have Jenkins and Sonar up and running as we created in part 1:
If you got errors about some port being busy just use the free ports from your host, I explain this here. Otherwise you can use dynamic ports which is shown a bit later.
Chapter 1. Running JFrog Artifactory on Docker
So let’s look at the first step. Obviously if we want to test the deployment in our example, we need some place to deploy our artifacts to. We are going to use a limited open source version of JFrog Artifactory called “artifactory oss”. Let’s run it on Docker to see how easy it is to have your own artifact repo. The port 8081 on machine was busy, so I had to run it on 8082, you should do according to free ports available on your machine:
docker run --rm -p 8082:8081 --name artifactory docker.bintray.io/jfrog/artifactory-oss:5.4.4
Alternatively you can use dynamic ports.
When using dynamic ports, you just publish the port required by image, and docker decides to which port on the host to map it based on ephemeral port range defined by /proc/sys/net/ipv4/ip_local_port_range on Linux. But obviously, as docker running inside VM on Mac(I keep forgetting you may have Windows NT or similar, so bear with me please), in order to see this range you would need to connect to the VM first:
➜ ~ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty / # cat /proc/sys/net/ipv4/ip_local_port_range 32768 60999 / #
If you want to see on what system VM is running:
uname -a Linux moby 4.9.27-moby #1 SMP Wed May 10 11:51:05 UTC 2017 x86_64 Linux
To terminate press: Ctrl+a Ctrl+-\ (followed by ‘y’). So if we ran artifactory on dynamic port the command would be like:
docker run --rm -p 8081 --name artifactory docker.bintray.io/jfrog/artifactory-oss:5.4.4
As you can see, the first part of 8082:8081 is missing now, and to find out what docker actually used, use docker port command:
➜ ~ docker port artifactory 8081/tcp -> 0.0.0.0:32828
We can see it allocated port 32828 for it, so we would find artifactory running at http://localhost:32828 in that case.
Normally you would run artifactory with volumes to preserve its state, that is the jar files in our case, but for the sake of our tutorial which is about Dockerizing Jenkins, we will simply run it on “in memory” fashion. Once it is up and running we should be ready to deploy our artifacts to it .
Please pay attention to the the name of the path for the default repository it has created, it was example-repo-local in my case, as we will refer to it very soon.
By the time I finished part two, I improved the runall script so it should run with no issues if some of the ports used by sonarqube, jenkins or artifactory were busy(8080, 8081, 9000 etc):
So let’s switch to updated script, it will just stop all containers, build Jenkins image and then rerun with dynamic ports:
#!/usr/bin/env bash
function getContainerPort() {
echo $(docker port $1 | sed 's/.*://g')
}
docker pull jenkins:2.60.1
docker pull sonarqube:6.3.1
if [ ! -d downloads ]; then
mkdir downloads
curl -o downloads/jdk-8u131-linux-x64.tar.gz http://ftp.osuosl.org/pub/funtoo/distfiles/oracle-java/jdk-8u131-linux-x64.tar.gz
curl -o downloads/jdk-7u76-linux-x64.tar.gz http://ftp.osuosl.org/pub/funtoo/distfiles/oracle-java/jdk-7u76-linux-x64.tar.gz
curl -o downloads/apache-maven-3.5.0-bin.tar.gz http://apache.mirror.anlx.net/maven/maven-3/3.5.0/binaries/apache-maven-3.5.0-bin.tar.gz
fi
docker stop mysonar myjenkins artifactory 2>/dev/null
docker build -t myjenkins .
docker run -d -p 9000 --rm --name mysonar sonarqube:6.3.1
docker run -d --rm -p 8081 --name artifactory docker.bintray.io/jfrog/artifactory-oss:5.4.4
sonar_port=$(getContainerPort mysonar)
artifactory_port=$(getContainerPort artifactory)
IP=$(ifconfig en0 | awk '/ *inet /{print $2}')
if [ ! -d m2deps ]; then
mkdir m2deps
fi
docker run -d -p 8080 -v `pwd`/downloads:/var/jenkins_home/downloads \
-v `pwd`/jobs:/var/jenkins_home/jobs/ \
-v `pwd`/m2deps:/var/jenkins_home/.m2/repository/ --rm --name myjenkins \
-e SONARQUBE_HOST=http://${IP}:${sonar_port} \
-e ARTIFACTORY_URL=http://${IP}:${artifactory_port}/artifactory/example-repo-local \
myjenkins:latest
echo "Sonarqube is running at ${IP}:${sonar_port}"
echo "Artifactory is running at ${IP}:${artifactory_port}"
echo "Jenkins is running at ${IP}:$(getContainerPort myjenkins)"
And run it:
bash +x runall.sh 2.60.1: Pulling from library/jenkins Digest: sha256:fa62fcebeab220e7545d1791e6eea6759b4c3bdba246dd839289f2b28b653e72 Status: Image is up to date for jenkins:2.60.1 6.3.1: Pulling from library/sonarqube Digest: sha256:d5f7bb8aecaa46da054bf28d111e5a27f1378188b427db64cc9fb392e1a8d80a Status: Image is up to date for sonarqube:6.3.1 mysonar myjenkins artifactory Sending build context to Docker daemon 223.7MB Step 1/6 : FROM jenkins:2.60.1 ---> f426a52bafa9 Step 2/6 : MAINTAINER Kayan Azimov ---> Using cache ---> 760e7bb0f335 Step 3/6 : ENV JAVA_OPTS "-Djenkins.install.runSetupWizard=false" ---> Using cache ---> e3dbac0834cd Step 4/6 : COPY plugins.txt /usr/share/jenkins/ref/plugins.txt ---> Using cache ---> 53607bbb60e4 Step 5/6 : RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt ---> Using cache ---> 28d643dfc7e5 Step 6/6 : COPY groovy/* /usr/share/jenkins/ref/init.groovy.d/ ---> Using cache ---> bd076517e84d Successfully built bd076517e84d Successfully tagged myjenkins:latest 28f61ad44e62c1437e323e1a533abb01f239a1a8f1d64977125b7c825cc0b462 aed4ec3384265232476ec7d88bdda20b776f7ebac8a6ac3daf1f93b23e2e3801 302e7b6991faa376254f0bc0a5421a761f727d882d4d2a01336ea82666688262 Sonarqube is running at http://192.168.1.3:32876 Artifactory is running at http://192.168.1.3:32877 Jenkins is running at http://192.168.1.3:32878
We now see the ports for all running containers in the logs, so we can access any of them if we like to. As you may have noticed our shell script has become a bit loo complicated, this is just to run 3 containers, so it means next time I should probably switch to using docker compose instead!
We obviously need to update our pipeline which we created in part 1, add a deployment step to it and push the updated Jenkins file and build the job. That is if you are using your own repo, alternatively, just amend the existing job as below by using Replay button on last successful build and then run it:
pipeline {
agent any
tools {
jdk 'jdk8'
maven 'maven3'
}
stages {
stage('install and sonar parallel') {
steps {
parallel(
install: {
sh "mvn -U clean test cobertura:cobertura -Dcobertura.report.format=xml"
},
sonar: {
sh "mvn sonar:sonar -Dsonar.host.url=${env.SONARQUBE_HOST}"
}
)
}
post {
always {
junit '**/target/*-reports/TEST-*.xml'
step([$class: 'CoberturaPublisher', coberturaReportFile: 'target/site/cobertura/coverage.xml'])
}
}
}
stage('deploy') {
steps {
sh "mvn deploy -DskipTests"
}
}
}
}
And this is what going to happen:
[INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 9.265 s [INFO] Finished at: 2017-07-16T18:39:51Z [INFO] Final Memory: 17M/95M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project berlin-clock: Deployment failed: repository element was not specified in the POM inside distributionManagement element or in -DaltDeploymentRepository=id::layout::url parameter -&amp;amp;gt; [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [Pipeline] }
Job fails, but maven is clever enough to figure out the reason, and it prompted us with message that distributionManagement section in the pom file is missing. So maven deploy plugin doesn’t basically know where to deploy the artifact, would you know? 🙂
Chapter 2. Configuring maven pom file
Let’s configure the missing part in the pom file inside the project. If you are not using your own git project for this tutorial you won’t be able to amend the POM file as it is in my repository and you don’t’ have an access unfortunately 😉 but you can switch to another project in that case which I prepared for you. So please create another pipeline job for the project maze-explorer which has necessary POM changes, if you are using your own git project just ignore this note. Obviously, I could have used just a branch in the same project, but let’s have couple jobs in Jenkins!
Creating new pipeline:
Configuring repository:
Here is how your changes in the POM will look like, we added distributionManagement to the project section:
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>kayan</groupId>
<artifactId>maze</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<distributionManagement>
<snapshotRepository>
<id>artifactory</id>
<name>artifactory</name>
<url>${artifactory_url}</url>
</snapshotRepository>
</distributionManagement>
(the rest of POM file)
The url should point to the url of the artifactory, so we need to pass it to maven through jenkins as environment variable, just like we did for sonar and obviously it can’t be static as the IP of the host can change.
If you run the job now, it is still going to fail, even if it has distributionManagement configured:
[INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.459 s [INFO] Finished at: 2017-07-16T20:13:34Z [INFO] Final Memory: 13M/137M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project maze: Failed to deploy artifacts: Could not transfer artifact kayan:maze:jar:1.0-20170716.201334-1 from/to artifactory (http://192.168.1.5:8082/artifactory/example-repo-local): Failed to transfer file: http://192.168.1.5:8082/artifactory/example-repo-local/kayan/maze/1.0-SNAPSHOT/maze-1.0-20170716.201334-1.jar. Return code is: 401, ReasonPhrase: Unauthorized. -&amp;amp;gt; [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging.
This is because we need to have credentials in order to deploy. The default credentials of JFrog are “admin:password”, try to login and check. In order to pass credentials to deployment plugin we need to set them in maven settings.xml file.
Chapter 3. Configuring maven settings file
If we have maven installed locally, then we can run it first just to check the deployment actually works with our configuration and then we can start looking at how to configure it with Jenkins. Lets check what we have in settings file:
mvn help:effective-settings
You will see something similar if setting file is absent or empty.
<settings xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
<localRepository xmlns="http://maven.apache.org/SETTINGS/1.1.0">/Users/kayanazimov/.m2/repository</localRepository>
<pluginGroups xmlns="http://maven.apache.org/SETTINGS/1.1.0">
<pluginGroup>org.apache.maven.plugins</pluginGroup>
<pluginGroup>org.codehaus.mojo</pluginGroup>
</pluginGroups>
</settings>
Now let’s go to /Users/YOUR_USER_NAME_HERE/.m2/ and change or create a settings.xml as below, in order to pass credentials of the repo we need to add a server on the servers section of the maven settings:
<?xml version="1.0" encoding="UTF-8"?>
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<servers>
<server>
<id>artifactory</id>
<username>admin</username>
<password>password</password>
</server>
</servers>
</settings>
Please note, id should be same as the id you used earlier in the POM file for snapshotRepository.
You can check if maven is picking it up:
mvn help:effective-settings
Now clone the project or if you using your own project just run from project folder:
mvn deploy -DskipTests -Dartifactory_url=http://localhost:8082/artifactory/example-repo-local [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building maze 1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ maze --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/kayanazimov/workspace/learn/maze-explorer/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ maze --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ maze --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 10 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ maze --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ maze --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ maze --- [INFO] Building jar: /Users/kayanazimov/workspace/learn/maze-explorer/target/maze-1.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ maze --- [INFO] Installing /Users/kayanazimov/workspace/learn/maze-explorer/target/maze-1.0-SNAPSHOT.jar to /Users/kayanazimov/.m2/repository/kayan/maze/1.0-SNAPSHOT/maze-1.0-SNAPSHOT.jar [INFO] Installing /Users/kayanazimov/workspace/learn/maze-explorer/pom.xml to /Users/kayanazimov/.m2/repository/kayan/maze/1.0-SNAPSHOT/maze-1.0-SNAPSHOT.pom [INFO] [INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ maze --- Downloading: http://localhost:8082/artifactory/example-repo-local/kayan/maze/1.0-SNAPSHOT/maven-metadata.xml Uploading: http://localhost:8082/artifactory/example-repo-local/kayan/maze/1.0-SNAPSHOT/maze-1.0-20170718.083952-1.jar Uploaded: http://localhost:8082/artifactory/example-repo-local/kayan/maze/1.0-SNAPSHOT/maze-1.0-20170718.083952-1.jar (28 kB at 126 kB/s) Uploading: http://localhost:8082/artifactory/example-repo-local/kayan/maze/1.0-SNAPSHOT/maze-1.0-20170718.083952-1.pom Uploaded: http://localhost:8082/artifactory/example-repo-local/kayan/maze/1.0-SNAPSHOT/maze-1.0-20170718.083952-1.pom (2.1 kB at 49 kB/s) Downloading: http://localhost:8082/artifactory/example-repo-local/kayan/maze/maven-metadata.xml Uploading: http://localhost:8082/artifactory/example-repo-local/kayan/maze/1.0-SNAPSHOT/maven-metadata.xml Uploaded: http://localhost:8082/artifactory/example-repo-local/kayan/maze/1.0-SNAPSHOT/maven-metadata.xml (753 B at 25 kB/s) Uploading: http://localhost:8082/artifactory/example-repo-local/kayan/maze/maven-metadata.xml Uploaded: http://localhost:8082/artifactory/example-repo-local/kayan/maze/maven-metadata.xml (267 B at 9.9 kB/s) [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.828 s [INFO] Finished at: 2017-07-18T09:39:52+01:00 [INFO] Final Memory: 13M/207M [INFO] ------------------------------------------------------------------------ ➜ maze-explorer git:(master)
Now check the artifactory:
As you can see we successfully deployed it, Yay! Now it is time to prepare this step in Jenkins.
Chapter 4. Using Config File Provider Plugin for persistence of maven settings
In order to apply the settings file to maven in Jenkins we need config file provider plugin which lets preserving multiple setting files(please be aware we might need more than one settings depending on a project running the job in real life). Now let’s install the plugin manually first:
We need to setup the maven settings file in the plugin. Go to manage Jenkins, managed files and add new config, set the id to “our_settings” and copy content from settings.xml we used before:
Click Managed files
select Global Maven Settings and set id:
And set content:
Now we can use Config File Provider plugin in our pipeline, please Replay the job and update the pipeline as below:
pipeline {
agent any
tools {
jdk 'jdk8'
maven 'maven3'
}
stages {
stage('install and sonar parallel') {
steps {
parallel(
install: {
sh "mvn -U clean test cobertura:cobertura -Dcobertura.report.format=xml"
},
sonar: {
sh "mvn sonar:sonar -Dsonar.host.url=${env.SONARQUBE_HOST}"
}
)
}
post {
always {
junit '**/target/*-reports/TEST-*.xml'
step([$class: 'CoberturaPublisher', coberturaReportFile: 'target/site/cobertura/coverage.xml'])
}
}
}
stage ('deploy'){
steps{
configFileProvider([configFile(fileId: 'our_settings', variable: 'SETTINGS')]) {
sh "mvn -s $SETTINGS deploy -DskipTests -Dartifactory_url=${env.ARTIFACTORY_URL}"
}
}
}
}
}
Lets runs the build:
If you are lucky you will get the scree above 🙂 Otherwise read what can go wrong with containers when running out of memory
Issues with docker containers when running out of memory
I will be honest with you, it didn’t run successfully from the first try for me. For some weird reason, deployment stage was killing artifactory! The only reason could be memory, I first thought, and I was damn right! When I checked JVM parameters passed to artifactory, I saw this:
This parameter is passed to artifactory and it kills container as soon as docker is running out of memory.
-XX:OnOutOfMemoryError=kill -9 %pSo I checked the memory consumed by docker(hyperkit):
It was using 2.65GB, and then how much docker could actually have:
Just 2GB! So I just increased it to 6GB and the problem was gone. If you have less memory just don’t run the jobs in parallel, it might help, I hope.
Chapter 5.Dockerizing the installation and configuration process
Now let’s dockerize the plugin installation and configuration. Create mvn_settings.groovy file, copy it to groovy folder and set content as below
import jenkins.model.*
import org.jenkinsci.plugins.configfiles.maven.*
import org.jenkinsci.plugins.configfiles.maven.security.*
def store = Jenkins.instance.getExtensionList('org.jenkinsci.plugins.configfiles.GlobalConfigFiles')[0]
println("Setting maven settings xml")
def configId = 'our_settings'
def configName = 'myMavenConfig for jenkins automation example'
def configComment = 'Global Maven Settings'
def configContent = '''<?xml version="1.0" encoding="UTF-8"?>
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<servers>
<server>
<id>artifactory</id>
<username>admin</username>
<password>password</password>
</server>
</servers>
</settings>
'''
def globalConfig = new GlobalMavenSettingsConfig(configId, configName, configComment, configContent, false, null)
store.save(globalConfig)
And don’t forget to install the “config-file-provider” plugin, just add it to the plugins.txt
Time to rebuild and rerun your jenkins, just run the runall script. Once Jenkins is ready run the build.
We now have the CI pipeline for our project and almost fully automated Jenkins. Why almost?
Because we need to hide the passwords firstly. But I am running late, and my girlfriend has already started complaining…
But I promise we will look at how to use encrypted passwords in Jenkins very soon in the next session.
Again, If you were lazy and don’t like “hands on” stuff, just clone the complete code for the 2nd part of tutorial, run it, enjoy it and share it if you like:
git clone https://github.com/kenych/dockerizing-jenkins-part-2 && cd dockerizing-jenkins-part-2 && ./runall.sh
Hope you managed to run everything without any issues and enjoyed this tutorial.