Infrastructure

Overview

As an independent open source project, the Jenkins project maintains most of its own infrastructure including services which help to keep the project running. The kind of things that fall into "infrastructure" can span from operating virtual machines, containers, configuring network or developing and maintaining project-specific applications to make the development of Jenkins core and plugins more efficient.

Because we strongly believe in Open Source principles, we also apply them to our infrastructure. As such we consider ourself as an open infrastructure project where everybody is invited to learn, share, contribute.

Overview

Team

Jenkins infrastructure is maintained by the Jenkins Infrastructure team. This team consists of Jenkins contributors and volunteers who provide a best-effort support depending on their availability and other commitments. We are always looking for contributors who are interested to improve Jenkins services and to help with maintaining them.

Team structure

This section is under development

Meetings

Jenkins Infra team has a weekly meeting on Tuesday, 14:30 UTC. This event is a regularly scheduled meeting to sync on the various ongoing infrastructure initiatives within the Jenkins project. This meeting is open to everyone who is interested. See the Event Calendar for meeting links.

You can find Infra team meeting agenda and meeting notes on jenkins-infra/documentation.

Infrastructure

Provisioning

The Jenkins project runs the most of its infrastructure on Azure, sponsored by Continuous Delivery Foundation. This infrastructure is provisioned from the repository jenkins-infra/azure using Terraform and using AKS charts.

While we try to stick to Azure, we still have machines sponsored by different organizations such as OSUOSL, AWS, Rackspace or CloudBees.

Configuration Management

Infrastructure

The Jenkins project’s infrastructure is managed and orchestrated via a git repository named jenkins-infra/jenkins-infra using Puppet. We are also using a Kubernetes cluster orchestrated with jenkins-infra/charts using Helm & Helmfiles.

Monitoring

Infrastructure is monitored via Datadog. Datadog configuration is automated with Terraform on jenkins-infra/datadog

We also provide different dashboards to share the health of our systems.

Status Page

Confluence

Reporting

Pagerduty is used to report issues detected by Datadog, we have a follow the sun policy where we try to only be on call during our "available" hours.

Services

Another major aspect of the Jenkins Infrastructure project, is the maintenance of all services either provided by us or third parties. Here is a non-exhaustive list of services that we provide and maintain.

SaaS

Service Provider Issue tracker Repository

Artifactory

JFrog

GitHub issues

-

GitHub

GitHub

GitHub issues

-

Monitoring

Datadog

GitHub issues

Code

Pagerduty

Pagerduty

-

Gitter chat system

GitLab

-

Content Delivery Network

Fastly

-

DNS Registrar

Namecheap

-

Jira

Linux Foundation

Linux Foundation Support

-

Sub-project/SIG services

Jenkins infrastructure also hosts some services for sub-projects and special interest groups:

Service Owner Sub-project/SIG Issue tracker component(s) Repository

Code and Repository Signing

Release Team

release

DigiCert

Contributing

Our infrastructure is an open infrastructure project made by and for the Jenkins community. In other words, it’s a contributors driven project. While we can’t share publicly everything like secrets, certificates,…​ we still try to be as transparent as possible so that everybody can understand and improve our infrastructure without having privileged accesses. If you have any idea that could help the infra or interest the community, feel free to make suggestions.

Before going further, we assume that:

Contribution Workflow

In order to contribute to infrastructure project, we ask people to follow the next steps

Pick up a task => Communicate => Implement => Deploy => Review

Pick Up A Task

To keep track of the work that needs to be done on the Jenkins infrastructure project, we use the Github help desk repository. If you want to contribute, the first step is to find the issue in this project that you want to work on, assign it to you, then communicate about it (see Communicate).

If you can’t find an appropriate issue, please create a new one with a clear description:

  • Why (what is the problem to solve - high level value)?

  • What (what your proposal to solve the problem)?

  • How (what are the technical changes to do)?

You may also specify components and finally you can communicate about it, using the suggestions from the next section.

Good First Issues

If you want to start contributing on the Jenkins infrastructure, you can find a list of "first good issues" to be considered (they all have the label newbie-friendly) on the following page: good first issues.

Communicate

Before any implementation, it’s important to verify that first, there is (still) a need for some implementation and then that no work has been done in the past. The best way for that is to either look after similar issues, ask on IRC or on the mailing list. You can also join our weekly meetings to discuss and coordinate the changes.

When the subject is too broad or hard to explain in few lines, we write an IEP document which stand for "Infrastructure Enhancement Proposal", the purpose of this document is to explain why we need something, how we want to solve it, and why we took a final decision. Finally, once you have your ticket id, you can start looking for knowledgeable people.

Anyway keep in mind that it’s always better to have too much information than too little and in the end you’ll probably be the best person who can work on your case.

In short
+----------------------------------+
|                                  |
|  Pick up or Create INFRA Ticket  |
|                                  |
+----+----+------------------------+
     |    |   If no responses after few days
     |    |   promote it on
     |    |                    +------------------------------------------+
     |    |                    |                                          |
     |    +--------------------> IRC: Libera Chat #jenkins-infra          <----+
     |    |                    |                                          |    |
     |    |                    +------------------------------------------+    |
     |    |                    +------------------------------------------+    |
     |    |                    |                                          |    |
     |    +--------------------> Mail: jenkins-infra@googlegroups.com     <----+
     |                         |                                          |    |
     |                         +------------------------------------------+    |
     | If the topic is too big                                                 |
     |                                                                         |
     |                    +-------------------------------------------+        |
     |                    |                                           |        |
     +--------------------> IEP: https://github.com/jenkins-infra/iep |--------+
                          |                                           |
                          +-------------------------------------------+

Implement

Once there is an agreement about the approach and before any changes, we ask contributors to respect the following rules.

Those rules are just what we consider "best practices" for a contributors driven project and can be adapted depending on specific git repositories.

Rule #1: Everything is in a git repository under the jenkins-infra organization.

So it’s easier for everybody to find/review/audit changes and suggest improvements.

Rule #2: All changes are validated by at least one regular infra contributor via Pull Request.

So we always have different people who understand infrastructure changes.

Rem: Non regular contributors are more than welcome to share their expertise or just ask question which also help to spot incoherences.

Code Reviews Purpose:

  • Educate author and the team about the changes in code being made

  • Discover logic or security issues not covered by tests

  • Gather improvement suggestion for code readability or efficiency

    Rule #3: All Changes are tested on ci.jenkins.io

    So we feel more comfortable when merging PRs and we avoid regression issues.

    Rule #4: Everything is automated.

    So we only have one source of truth and we don’t break others people works. And if it’s not possible, then it needs to be well communicated and documented cf. rule #1.

    Rule #5: All changes follow the Github workflow.
Fork project -> Create Feature Branch -> Open Pull Request -> Ask Review -> Merge

Deployment

The deployment step is the only moment where we need approval from someone with elevated permission. As already mentioned, even if we try to be as open as possible, we don’t want to share privileged accesses with every contributors even if we trust them and that mainly for security reasons.

Various link which can helpful when looking at the Jenkins infra project