Back to blog

Introducing the Pipeline Configuration History Plugin

Jochen A. Fürbacher
Stefan Brausch
Robin Schulz
July 15, 2019

Pipelines are the efficient and modern way how to create jobs in Jenkins. To recognize pipeline changes quickly and easily, we developed the Pipeline Configuration History plugin. This plugin detects changes of pipelines and provides the user an option to view changes between two builds (diffs) of pipeline configurations visibly and traceably.

How everything started

It all started 10 years ago — with classical job types (e.g. Freestyle, Maven, etc.). Every once in a while users contacted us because their jobs failed to build overnight. Why did the job fail? Was the failure related to a job configuration change? The users' typical answer was: "We didn’t change anything!", but is that really true? We thought about this and decided to develop a plugin that helped us solve this problem. This was the idea and the beginning of Job Configuration History.

Now it was possible to view changes of job configurations (like other branches, JDK versions, etc.) and more often the reason for breaking builds were changes of job configurations.

Screenshot of Job Configuration History

Over the years the plugin got developed and is still under development. New functions were added, that not only view job configurations, but also changes of global and agent configurations. It is also possible to recover old configuration versions. Today the plugin has more than 30,000 installations. For many years JobConfigHistory relieves our daily work — with more than 3,000 Jenkins jobs! Then there was a new type of job: Pipelines.

Pipelines - something new was needed

Pipeline jobs are fundamentally different than classical job types . While classic job types are configured via the Jenkins GUI, Pipeline jobs are configured as code. Every pipeline job indeed gets created via the Jenkins GUI, however that is not necessarily where the pipeline configuration is located. Pipelines can be configured:

  • Directly in the Jenkins job as script. The code gets inserted directly in the job configuration page.

  • As Jenkinsfile in the source code management system (SCM): The pipeline configuration is defined in a text file (Jenkinsfile) in the SCM. In the job itself only the path to the repository of the Jenkinsfile is configured. During the build the Jenkinsfile gets checked out from the SCM and processed.

  • As a shared library: A part of the pipeline configuration gets moved to separate files that can be used by several jobs. These files are also saved in the SCM. Even so a Jenkinsfile is still needed (or a pipeline script in the job).

With every save operation of the job configuration, JobConfigHistory creates a copy of the actual job configuration if something has changed. That only works for pipeline jobs if the pipeline configuration is inserted in the job configuration page as script. Changes in the Jenkinsfile or the shared libraries are not detected by JobConfigHistory. You have to use the SCM system to view changes of the Jenkinsfile or the shared libraries. It is complex and time intensive to find a correlation between the time of a build and a change to the Jenkinsfile or shared library.

This new problem is much more than JobConfigHistory. A new solution was needed to detect pipeline changes and show these changes in Jenkins. So we developed Pipeline Configuration History.

During every pipeline run the Jenkinsfile and related shared libraries are saved in the builds directory of the job. Pipeline Configuration History saves changes of the pipeline files between the last run and the previous run as history events. Therefore when a pipeline job ceases to build successfully, you can check if something has changed on any used pipeline file. You can also see the build where changes occurred.

Screenshot of Pipeline Configuration History

Because a pipeline configuration can consist of several files where changes could have occurred, only files with changes between two builds are shown in the diff. That makes the whole thing more compact and effective:

Screenshot of Pipeline Configuration History

But sometimes you may want to show more than the differences between pipeline files. You may want to see which pipeline files are in use or the content of those files when they were used. So it’s possible to view all files and their content. If required you can download them as well:

Screenshot of Pipeline Configuration History

Conclusion

We use Pipeline Configuration History successfully in production. It has helped us from the very first day as we solved problems that occurred due to pipeline configuration changes. Pipeline Configuration History won’t replace Job Configuration History. The plugins have different use cases. Many times small changes on job or pipeline configurations also have big impacts. Because of the correlation in time between changes of job or pipeline configurations and different build behavior, it is now possible to substantially reduce the time and effort to analyze build failures. The Job Configuration History and Pipeline Configuration History plugins let us help our users in consulting and in solving issues. We resolve problems much faster through easy access to the configuration history of jobs. These plugins are essential for our daily work.

About the authors

Jochen A. Fürbacher

Jochen is a software developer for development infrastructure at 1&1 Telecommunication. He has been working with Jenkins for many years and develops some Jenkins plugins.

Stefan Brausch

Stefan is a software developer for development infrastructure at 1&1 Telecommunication. He has been working with Jenkins for many years and develops some Jenkins plugins.

Robin Schulz

Robin is a software developer for development infrastructure at 1&1 Telecommunication. He has been working with Jenkins for many years and develops some Jenkins plugins.