Infrastructure

Overview

As an independent open source project, the Jenkins project maintains most of its own infrastructure including services which help to keep the project running. The kind of things that fall into "infrastructure" can span from operating virtual machines, containers, configuring network or developing and maintaining project-specific applications to make the development of Jenkins core and plugins more efficient.

Because we strongly believe in Open Source principles, we also apply them to our infrastructure. As such we consider ourself as an open infrastructure project where everybody is invited to learn, share, contribute.

Overview

Team

Jenkins infrastructure is maintained by the Jenkins Infrastructure team. This team consists of Jenkins contributors and volunteers who provide a best-effort support depending on their availability and other commitments. We are always looking for contributors who are interested to improve Jenkins services and to help with maintaining them.

Team structure

This section is under development

Meetings

Jenkins Infra team has a weekly meeting on Tuesdays, 3PM UTC. This event is a regularly scheduled meeting to sync on the various ongoing infrastructure initiatives within the Jenkins project. This meeting is open to everyone who is interested. See the Event Calendar for meeting links.

You can find Infra team meeting agenda and meeting notes in this Google Doc.

Infrastructure

Provisioning

The Jenkins project has a partnership with Microsoft to run the infrastructure on Azure. This infrastructure is provisioned from the repository jenkins-infra/azure using Terraform.

While we try to stick to Azure, we still have machines sponsored by different organizations such as OSUOSL, Rackspace or CloudBees.

Configuration Management

Infrastructure

The Jenkins project’s infrastructure is managed and orchestrated via a git repository named jenkins-infra/jenkins-infra using Puppet and we are also experimenting Kubernetes cluster orchestration with jenkins-infra/charts using Helm & Helmfiles.

Monitoring

Infrastructure is monitored via Datadog. Datadog configuration is automated with Terraform on jenkins-infra/datadog

We also provide different dashboard to share the health of our systems.

Status Page

Jira&Confluence

Reporting

Pagerduty is used to report issues detected by Datadog, we have a follow the sun policy where we try to only be on call during our "available" hours.

Services

Another major aspect of the Jenkins Infrastructure project, is the maintenance of all services either provided by us or third parties. Here is a non-exhaustive list of services that we provide and maintain.

Self-hosted services

Service JIRA component(s) Repository

Accounts

account, accountapp

Code

IRC Bot

ircbot

Code

Javadoc

javadoc

Code

Jira

jira

Docker

Jenkins

ci.jenkins.io, ci

Deployment scripts

LDAP

ldap

Docker

Main Website

jenkins.io

Code

VPN

vpn

Code

Plugin Site

plugins.jenkins.io

Frontend, Backend

Stats

N/A

Scripts

UpLink

uplink

Code

Wiki

wiki

Docker

Jenkins Wiki Exporter

GitHub Issues

Code

SaaS

Service Provided by JIRA component(s) Repository

Artifactory

JFrog

artifactory

-

GitHub

GitHub

github

-

Monitoring

Datadog

datadog

Code

Pagerduty

Pagerduty

N/A

-

Sub-project/SIG services

Jenkins infrastructure also hosts some services for sub-projects and special interest groups:

Service Owner Sub-project/SIG JIRA component(s) Repository

Website in Chinese

Chinese Localization

cn.jenkins.io

Code

Evergreen

Evergreen

evergreen

Repository

Contributing

Our infrastructure is an open infrastructure project made by and for the Jenkins community in other words, it’s a contributors driven project. While we can’t share publicly everything like secrets, certificates,…​ we still try to be as transparent as possible so everybody can read, understand, improve our infrastructure without having privileged accesses. If you have any idea that could help the infra or interest the community, feel free to make suggestions.

Before going further, we assume that:

Contribution Workflow

In order to contribute to infrastructure project, we ask people to follow the next steps

Pick up a task => Communicate => Implement => Deploy => Review

Pick Up A Task

In order to keep track of the work that need to be done on the Jenkins infrastructure project, we use the project "INFRA" on Jira. Therefor the first thing to do before any contribution is to find the right ticket, assign it to you, then communicate about it.

If you can’t find an appropriate ticket, please create a new ticket with a clear description of what needs to be done and why. Some jenkins-infra git repository references can help to understand the context. You may also specify components and finally you can communicate about it, using the suggestions from the next section.

Remark

While a ticket can have different components assigned to it, we also use the label newbie-friendly to identify task which can be done by a new contributor.

Communicate

Before any implementation, it’s important to verify that first, there is (still) a need for some implementation and then that no work has been done in the past. The best way for that is to either look after similar Jira tickets, ask on IRC or on the mailing list. You can also join our weekly meetings to discuss and coordinate the changes.

When the subject is too broad or hard to explain in few lines, we write an IEP document which stand for "Infrastructure Enhancement Proposal", the purpose of this document is to explain why we need something, how we want to solve it, and why we took a final decision. Finally, once you have your ticket id, you can start looking after knowledgeable people.

Anyway keep in mind that it’s always better to have too more information than too few and in the end you’ll probably be the best person who can work on your case.

In short
+----------------------------------+
|                                  |
|  Pick up or Create INFRA Ticket  |
|                                  |
+----+----+------------------------+
     |    |   If no responses after few days
     |    |   promote it on
     |    |                    +------------------------------------------+
     |    |                    |                                          |
     |    +--------------------> IRC: Freenode #jenkins-infra             <----+
     |    |                    |                                          |    |
     |    |                    +------------------------------------------+    |
     |    |                    +------------------------------------------+    |
     |    |                    |                                          |    |
     |    +--------------------> Mail: jenkins-infra@lists.jenkins-ci.org <----+
     |                         |                                          |    |
     |                         +------------------------------------------+    |
     | If the topic is too big                                                 |
     |                                                                         |
     |                    +-------------------------------------------+        |
     |                    |                                           |        |
     +--------------------> IEP: https://github.com/jenkins-infra/iep |--------+
                          |                                           |
                          +-------------------------------------------+

Implement

Once there is an agreement about the approach and before any changes, we ask contributors to respect the following rules.

Those rules are just what we consider "best practices" for a contributors driven project and can be adapted depending on specific git repositories.

Rule #1: Everything is in a git repository under the jenkins-infra organization.

So it’s easier for everybody to find/review/audit changes and suggest improvements.

Rule #2: All changes are validated by at least one regular infra contributor via Pull Request.

So we always have different people who understand infrastructure changes.

Rem: Non regular contributors are more than welcome to share their expertise or just ask question which also help to spot incoherences.

Code Reviews Purpose:

  • Educate author and the team about the changes in code being made

  • Look after logic or security issues not covered by tests

  • Gather improvement suggestion for code readability or efficiency

    Rule #3: All Changes are tested on ci.jenkins.io

    So we feel more comfortable when merging PRs and we avoid regression issues.

    Rule #4: Everything is automated.

    So we only have one source of truth and we don’t break others people works. And if it can’t do that then it needs to be well communicated and documented cfr rule #1.

    Rule #5: All changes follow the Github workflow.
Fork project -> Create Feature Branch -> Open Pull Request -> Ask Review -> Merge Pull Request

Deployment

The deployment step is the only moment where we need approval from someone with elevated permission. As already mentioned, even if we try to be as open as possible, we don’t want to share privileged accesses with every contributors even if we trust them and that mainly for security reasons.

Various link which can helpful when looking at the Jenkins infra project