Cloud Native

Overview

The Cloud Native group of contributors and collaborators focuses on improving Jenkins to run on Cloud environments as a "Cloud Native" application. The improvements are targeted at both existing and new Jenkins users that use, or would prefer to use, Jenkins deployed in one of the cloud providers, or using cloud services for their operation.

(Back to List of Jenkins Special Interest Groups )

Description

This group of contributors and collaborators focuses on improving Jenkins to run on Cloud environments as a "Cloud Native" application.

Target Audience

The improvements are targeted at both existing and new Jenkins users that use, or would prefer to use, Jenkins deployed in one of the cloud providers, or using cloud services for their operation.

Meetings

There will be regular SIG meetings scheduled. Initially the meetings will be conducted once per month, then they may be adapted according to the activity.

Meetings will be conducted and recorded via Jenkins Hangouts-on-Air.

2018-08-16. Status meeting

Status sync-up: External Build Log Storage (JEP-207, JEP-210, JEP-212), External Configuration Storage (JEP-213), Jenkins Configuration-as-Code.

2018-07-31. External Build Log Storage

At this inaugural meeting we had introductions from SIG participants. Then Oleg Nenashev and Jesse Glick presented and discussed the current state of the External Build Log Storage work (JENKINS-38313).

Areas for Improvement

This section describes motivation and first topics for the SIG.

The current default approach of storing everything into the filesystem is the main reason why Jenkins does not fit the "Cloud Native" model, with features like HA, zero downtime, or pay per use. While there are plenty of plugins that implement parts of this vision, this becomes cumbersome to configure and a usability nightmare for users, as the Essentials JEP has pointed out. Going towards a model where cloud services are consumed where it makes sense, the overall complexity on operating Jenkins in a cloud or containerized environment is greatly reduced. Other related projects include Jenkins X which would greatly benefit from a Cloud Native Jenkins inside Kubernetes.

There are several clear areas open for improvement, which are summarized here and will be detailed in future documents. A mayor pain point is the usage of local disk as all-purpose storage, which causes issues running on containerized or distributed environments, requiring highly performant filesystems, and all the configuration pain like initial sizing and resizing with downtime. We believe that by using cloud provided services the overall usability, performance and scalability can be improved while enabling new demanded functionality.

By converting Jenkins to a "stateless" application long awaited features such as high availability and replication would help to operate Jenkins more efficiently, with zero downtime and reducing risk in upgrade operations by using canary or rolling deployments.

Artifact Storage

The current approach of storing artifacts in the filesystem prevents users from taking advantage of cloud provided blob stores (ie. AWS S3), which provide a lot more features for this use case.

Sizing becomes a problem as disk usage needs to be accounted for all the future artifacts stored or forces the need for resizing, which causes downtime. Typical use cases such as versioning, expiration or pay per use are impossible to implement today.

While currently it is possible to use specific plugins to store artifacts in blob stores such as the S3 plugin, this forces explicit configuration in jobs and makes configuration harder.

Log Storage

Logs disk usage causes the same issues previously mentioned for artifacts. Plus a external focused log storage such as AWS CloudWatch Logs allows demanded features as centralized log management, log retention policies, advanced querying, …​

There are already options to externally ship the logs to a backend, or plugins that would do that like the aws-cloudwatch-logs-publisher-plugin, but there is no integrated way to both send and display logs from external log storage.

The External log storage work is tracked as JENKINS-38313.

Configuration Storage

While xml files do not typically take a big amount of disk space, they make it really hard to query information about the system. This configuration is better suited in a cloud provided database in combination with https://github.com/jenkinsci/configuration-as-code-plugin Configuration as Code

Replication

If all the data currently stored in the filesystem is moved to external storage a replicated Jenkins service becomes possible, and the next steps are true High Availability, rolling or canary upgrades and zero downtime.

(Back to List of Jenkins Special Interest Groups)