Vagrant Jenkins server git repo for testing purpose: https://github.com/chengdol/vagrant-jenkins
Certified Jenkins Enginee
Certified Jenkins Engineer (CJE): https://github.com/bmuschko/cje-crash-course
Jenkins
Jenkins is not a build system, it is a structured model.
Other interesting project: Tekton
, Jenkins X
(k8s involved)
Installing Jenkins, must have compatible openjdk
installed.
1 | ## can use war file: |
or using rpm install for centos/redhat, then using systemctl to start/enable Jenkins https://www.jenkins.io/doc/book/installing/#red-hat-centos
1 | ## you will see the default jenkins port is 8080 |
then you can open the web interface by <node ip>:8080
. 如果发现是中文版,可能是浏览器的语言设置出了问题,改一下chrome的语言设置为English即可.
If wizard install failed with some plugins, you can fix this later in Manage Plugins
.
Jenkins use file system to store everything, if using systemd, the configuration is in /var/lib/jenkins
. You can backup this folder, or if you want to wipe it out, then run systemctl restart jenkins
, then jenkins goes back to init state.
Even Jenkins UI to create project is not needed, you can mkdir and files in the Jenkins working directory, then go to Manage Jenkins
-> Reload Configuration from Disk
.
遇到一件很神奇的事情,有次Jenkins的Credentials配置消失了。。。重启也不见恢复,后来我直接stop daemon, 清空workspace下所有文件,再次重启初始化,就恢复了。后来想想我应该是在配置Credentials时把配置改错了,可以通过
Manage Jenkins
->Configure Credentials
改回去。
Creating app build
freestyle project -> pipeline (series of freestyle project), freestyle project is not recommended.
Jenkins Workspace, you can see it in console output or click Workspace
icon in your project dashboard. Everything running is in project workspace. Every build will override the previous one. You can use tar command to backup and restore this workspace, or clean workspace.
注意, 如果在Jenkins configuration中直接使用pipeline script 而不是 SCM, 是不会创建workspace的。
To store the build artifact, use Post-build Action
in configure. for example, you want to archive some jar or zip files. then after build is done, these archives will show in the build page.
Build trend
Right to Build History
list, there is a trend
button, click it will see the build time history statistics and distribution.
Testing and Continuous integration
Now start the pipeline
job type. After creating a pipeline job, you will see pipeline syntax
button in the page bottom, it contains necessary resources to start. You can also use Copy from
to copy a pipeline configure from another, for quick start.
Add slave nodes
Manage Jenkins
-> Manage Nodes and Clouds
To add slaves, usually use SSH to launch agent nodes. (如果node没有被发现,会显示错误,根据错误指示排查问题即可)
Before adding a slave node to the Jenkins master we need to prepare the node. We need to install Java
on the slave node. Jenkins will install a client program on the slave node.
To run the client program we need to install the same Java version we used to install on Jenkins master. You need to install and configure the necessary tool in the slave node.
1 | yum install -y java-1.8.0-openjdk |
When configure add node agent, Host Key Verification Strategy:
Pipeline steps
This is scripted pipeline syntax, not recommended! Please use declarative pipeline directives! what are the differences between them: https://www.jenkins.io/doc/book/pipeline/#pipeline-syntax-overview
如果直接在Jenkins configure UI中设置Jenkins file,则常用的Steps (in snippet generator):
-
node: Allocate node Jenkins use
master <-> agent model
, you can configure tasks to be executed in agent node. -
stage: stage
-
stash: stash some files to be used later in build
-
unstash: restore files previous stashed
-
parallel: execute in parallel (如果注册了多个slave nodes,则parallel会在上面执行并行的任务, 一般用在测试的时候,比如测试不同的环境和配置, see declarative pipeline demo code below)
-
git: Git
-
dir: Change Current Directory
-
sh: Shell Script
-
step: General Build Step
-
emailext: Extended Email
Triggering auto build
在pipeline configure中有Builder Triggers
可以选择:
- Build after other projects are built
- Build periodically
- Poll SCM
- Disable this project
- Quiet period
- Trigger builds remotely
Email Notification
Using emailext: Extended Email
,可以用groovy函数包装,传入email的subject以及内容再调用。
Managing plugins
Manage Jenkins
-> Manage Plugins
, then you can select and install plugin in Available
section. For example:
-
Pipeline (如果这个没安装,则不会在UI中显示pipeline的动态流程图)
-
Html publisher (常用于发布unit test后的html report,这些html文件其实是被相关test生成好的, publisher then renders it)
1
2
3
4
5
6
7publishHTML([allowMissing: true,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir: "$WORKSPACE/cognitive-designer-api/DSJsonApiServletJUnitTests/build/reports/tests/payloadtests",
reportFiles: 'index.html',
reportName: 'Payload Test',
reportTitles: '']) -
Green Balls (show green color for success)
-
Blue Ocean (embedded site with new Jenkins UI)
-
Job DSL: Allowing jobs to be defined in a programmatic form in a human readable file.
Pipeline compatible plugins: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md
在初始化设置Jenkins的时候,有可能有plugins安装失败,可以自己在Manage plugin
中安装,然后restart Jenkins (关于restart Jenkins请在没有job运行的情况下进行,不同的安装方式restart的方法不同,或者在安装plugin的时候选择restart jenkins after install), for example: systemctl restart jenkins
, 这可以消除控制面板上的plugin failure警告。。
Continuous delivery
In Blue Ocean
, you can run multiple builds in parallel. if more than one builds run in the same agent, the workplace path is distinguished by suffix (@count number). 但是能不能run multiple builds in one agent depends on how you design your pipeline and tasks.
Blue Ocean
中的UI对parallel的显示也很直观,方便查看。
Trigger builds remotely
Set pipeline can be triggered remotely by URL, also optionally set pipeline trigger token in pipeline configure UI (can be empty).
You also need to know the user token, set from current user profile menu, you must keep the your user token to somewhere, for example, store it in credential secret text. So you can refer the token for example:
1 | TRIGGER_USER = "chengdol.example.com" |
Then in the upstream pipeline script, trigger other pipeline by running curl command:
1 | ## can be http or https connection |
You don’t need to specify all parameters in URL, the parameters default values will be used if they are not specified in the URL.
Notice that para1
and para2
must exist in parameters section of the triggered pipeline, otherwise you cannot use them. So far, based on testing, I can pass string, bool and file parameter types.
Check Status of Another pipeline
reference: https://gist.github.com/paul-butcher/dc68adc1c05ca970158c18206756dab1
1 | curl --user ${LOGIN_USER}:${LOGIN_USER_TOEKN} --request GET http/https://<url>/job/${PIPELINE_NAME}/<build number>/api/json |
Then you can parse the json returned: artifacts -> result: SUCCESS, FAILURE, ABORTED
Flyweight executor
Flyweight executor reside in Jenkins master, used to execute code outside of node allocation. Others are heavyweight executors. Flyweight executor will not be counted into executor capacity.
For example, we use flyweight executor for pause, in Jenkins script: https://stackoverflow.com/questions/44737036/jenkins-pipeline-with-code-pause-for-input
1 | ## see below declarative pipeline demo code |
The job will be paused, you can go to Paused for Input
to decide what to do next: proceed or abort. (In Blue Ocean
, the pause interface is more clear)
Declarative pipeline
Please use Groovy
style to wirte declarative pipeline!
github demo: https://github.com/sixeyed/jenkins-pipeline-demos
Declarative Pipelines: https://www.jenkins.io/doc/book/pipeline/syntax/
这里有关于declarative pipeline的介绍视频, Jenkins file lives in source control! https://www.jenkins.io/solutions/pipeline/,
Using Blue Ocean
to setup pipeline from github need personal access token (must check out repo and user options):
https://www.youtube.com/watch?v=FhDomw6BaHU
In Jenkins UI, go to pipeline syntax then declarative directive generator
, it will help you generate pipeline code for declarative pipeline:
https://www.jenkins.io/doc/book/pipeline/getting-started/#directive-generator
Jenkins parameters variables can be accessed by both params
and env
prefix, see this issue.
This is just a basic structure demo:
1 | pipeline { |
if you don’t want to checkout SCM in stages that run in same agent, you can use this option:
1 | options { |
Configure slack
This is a slightly out-of-date video, 其实在各自pipeline中可以自己单独配置token以及选择channel。 https://www.youtube.com/watch?v=TWwvxn2-J7E
First install slack notifaction
plugin. Then go to Manage Jenkins
-> Configure System
, scroll down to bottom you will see slack section, see question mark for explanation.
Then go to your target slack channel, select Add an app
, search Jenkins CI
, then add it to slack, follow the instructions to get the secret token, add this token to Jenkins credentials and use it in above slack configuration.
After all set, try Test connection
, you will see message in your slack channel.
Reusable
Reusable functions and libraries are written in Groovy
.
https://www.eficode.com/blog/jenkins-groovy-tutorial
Let’s see some demos:
1 | pipeline { |
Shared library
Demo structure and code: https://github.com/sixeyed/jenkins-pipeline-demo-library Invoking shared library at head of jenkins file:
1 | // this is dynamic reference, explicitly specify the library in jenkins file |
You can add Global Pipeline Libraries
in Configure Jenkins
for any pipeline use.
这种情况下,可以设置默认的shared library,然后在jenkins file中直接调用相关函数。
Shared pipelines
You can put the shared pipeline into a shared library, for example:
1 | library identifier: 'jenkins-pipeline-demo-library@master', |
Shared library is under vars
folder. In Groovy, we can add a method named call to a class and then invoke the method without using the name call
, crossPlatformBuild is actually the file name, inside file there is a call method.
Multi-branch pipeline
https://www.jenkins.io/doc/book/pipeline/multibranch/ Jenkins automatically discovers, manages and executes Pipelines for branches which contain a Jenkinsfile in source control.
Orphaned item strategy, for deleted branch, you can discard or reserve it.
Pipeline development tools
Validating pipeline syntax
First enable anonymous read access
in Configure Global Security
:
https://www.jenkins.io/doc/book/pipeline/development/#linter
Issue curl command:
1 | ## if not success, it will show you the overall problems with your jenkins file |
Visual Studio code has Jenkins linter plugin, you need to configure it with linter url.
Restart or replay
In every build interface, restart from stage
, you can select which stage to restart (sometimes stage may fail due to external reason), replay
, you can edit your jenkins file and library then rerun, the changes only live in current build (after succeed, check in your updates to source control).
Unit test
- Supports running pipelines and library methods
- Can mock steps and validate calls
Jenkins with Docker
学习步骤: 首先是agent 使用docker,然后master + agent 都使用docker, 最后交由K8s去管理。
这块很有意思,加入容器后就太灵活了,只要有build agent支持docker且已安装,则jenkins就可以把container运行在之上。 https://www.jenkins.io/doc/book/pipeline/docker/
https://hub.docker.com/r/jenkins/jenkins you can parallelly run several jenkins version in one machine, for purposes like testing new features, testing upgrade, so on and so forth. But you may need to customize the jenkins docker image and expose to different port.
- containers as build agents
- customizing the build container
- using the docker pipeline plugin
Jenkins master and slave with docker: https://medium.com/@prashant.vats/jenkins-master-and-slave-with-docker-b993dd031cbd
From agent syntax: https://www.jenkins.io/doc/book/pipeline/syntax/#agent
1 | agent { |
If there is no label
option, Jenkins will dynamically provisioned on a node and it will fail if no docker installed, you can set docker label to filter:
https://www.jenkins.io/doc/book/pipeline/docker/#specifying-a-docker-label
I got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: https://stackoverflow.com/questions/47854463/docker-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socke
还解决了一个权限的问题,在实验中container默认用户是jenkins
,没有root权限无线运行container内部的程序,解决办法是 args '-u root'
在上面的配置中。
此外jenkins会把workspace注入到container中,通过环境变量查找:
1 | sh 'printenv' |
Additionally, you can use agent with Dockerfile
and install Docker Pipeline plugin.