JENKINS - SHARED LIBRARIES

If you work with Jenkins and have more than two or three projects, it has surely happened to you: you copy the same Jenkinsfile from one repo to another, change four things, and from then on each one lives its own life. The day you need to change something common (add a Slack notification, change how the build is done, add a security step) you have to go repo by repo, copying and pasting all over again. That's where Shared Libraries come in.

A Shared Library is, at the end of the day, a Groovy repository that Jenkins loads inside your pipelines, and that lets you centralize all that repeated logic in a single place. You change the library once, and every pipeline that uses it finds out.

Now, there are two ways to use them, and the difference is big. The most common one you see in tutorials is to use the library only for loose steps (a notifySlack(), a buildAndDeploy()) that you call from a Jenkinsfile that still lives in each repo. We went one step further: the whole pipeline lived in the library, and no application repo had a Jenkinsfile. I'll tell you about both, but mostly the second one, which is the one that really changed our day-to-day.

Repository structure

Jenkins expects a very specific folder structure for the library to work:

(root)
├── vars/
│   ├── fullPipeline.groovy
│   └── notifySlack.groovy
├── src/
│   └── org/mallotore/
│       └── PipelineUtils.groovy
└── resources/
    └── org/mallotore/
        └── config.json

• vars/: this is where the "steps" go that you'll be able to call directly from a pipeline, as if they were global functions. Every .groovy file you put here, if it's called fullPipeline.groovy, you invoke as fullPipeline().
• src/: plain, ordinary Groovy code, with packages like in Java. This is where you put classes, helpers, whatever you need that isn't directly an invocable "step".
• resources/: static files your library may need to load at runtime (templates, JSON, whatever), via libraryResource.

The whole pipeline as a single step

Here's the real difference from what's usually taught out there: instead of putting only loose pieces in vars/, we put in the complete pipeline. vars/fullPipeline.groovy had all the stages (checkout, build, test, deploy depending on the environment) and received as parameters which repo, which branch and which environment to deploy to. One detail before we look at it: the agents were Windows, but we had Cygwin installed, so instead of using bat we used sh as is, as if we were on Linux. That let us write the Rake tasks without thinking about whether the agent at hand was Windows or not.

def call(Map config) {
    pipeline {
        agent any
        stages {
            stage('Checkout CI scripts') {
                steps {
                    dir('ci-scripts') {
                        git url: config.ciScriptsRepoUrl, branch: 'main'
                    }
                }
            }
            stage('Checkout') {
                steps {
                    git url: config.repoUrl, branch: config.branch
                }
            }
            stage('Build') {
                steps {
                    sh 'rake -f ci-scripts/Rakefile build'
                }
            }
            stage('Test') {
                steps {
                    sh 'rake -f ci-scripts/Rakefile test'
                }
            }
            stage('Deploy') {
                steps {
                    script {
                        deployToEnvironment(config.targetEnv)
                    }
                }
            }
        }
    }
}

Notice also deployToEnvironment(config.targetEnv): it's nothing from Jenkins, it's just another step of the library, a vars/deployToEnvironment.groovy that I'm not showing here so as not to make the post longer. The way of deploying was the same for all the applications, and that's why it made sense to share it: a single place that knows how to deploy to dev, uat, staging or prod, and that all the pipelines call. If in your case each application is deployed in its own way, this step will be the first one you won't want to share.

Another important detail: the Rake tasks didn't live in the application repo, but in a separate repo dedicated only to CI scripts. That's why the pipeline starts with a checkout of that repo before touching the application's; that way, just like with the pipeline itself, the build/test tasks were also centralized and versioned in a single place, instead of repeated (or different) in each of the application repos.

And the job in Jenkins, instead of pointing to a Jenkinsfile inside the application repo, had a minimal pipeline script, practically these two lines:

@Library('mallotore-shared-lib') _

fullPipeline(
    repoUrl: params.REPO_URL,
    branch: params.BRANCH_NAME,
    targetEnv: params.TARGET_ENV
)

The parameters REPO_URL, BRANCH_NAME and TARGET_ENV (dev, uat, staging, prod) were defined in the job itself as Jenkins parameters. This means that no application repo needed to know anything about Jenkins or maintain its own Jenkinsfile: all the CI/CD logic lived in a single place, versioned separately, and the application repos just had their code.

Why this changes everything when creating jobs

The most practical consequence of this approach is that creating a new job stopped being an event. There was no need to touch the application repo, or add a Jenkinsfile, or fight with pipeline syntax the first time. A new job was: create the job, paste those same four lines as always, fill in the parameters (repo, branch, environment), and done. Anyone on the team could set up a new pipeline for a new service in five minutes, without needing to know Groovy or how Jenkins worked inside.

And why it also makes migrations easier

This same design has an added advantage that's noticeable above all when it's time to do a major Jenkins upgrade (a major version change, a big plugin change, whatever you'd be wary of doing directly in production). Since all the pipeline logic is in a separate Git repository and the jobs are almost trivial to recreate, setting up a new Jenkins in parallel was easy: you install a clean Jenkins, register the same Shared Library pointing to the same branch or to a specific tag, and recreate the jobs with the same four parameters as always. In a short while you have a new Jenkins running alongside the old one, without risking breaking anything in the one in production, and when you trust the new one, you switch off the old one.

With the classic model of one Jenkinsfile per repo this is much more tedious: you have the logic scattered across N different repos, and setting up a Jenkins in parallel means making sure all those Jenkinsfiles are compatible with the new version before you can trust anything.

When not all projects are the same

All of this works well because we started from a specific base: a single team, with about 25 .NET Framework applications that followed the same pattern, with Rake orchestrating the build and test tasks on top of MSBuild. But we had another group of about 5 .NET Core applications, which used dotnet's own tooling (also wrapped in Rake, but with different tasks underneath) and didn't fit in the fullPipeline designed for MSBuild. Instead of forcing them through if/else inside the same step, the solution was simply to have another step, fullPipelineNetCore or similar, with its own stages, living in the same library.

In the end each job picked, from the steps available in the library, the one that best fit its type of application. The library wasn't "a pipeline", it was "a catalog of pipelines", and creating a job was still trivial: you pick which of the steps applies to you, fill in the usual parameters and done. The reasonable limit of this approach isn't the number of applications, but the number of different patterns you have: as long as the variations fit in "one more step in vars/", the catalog scales without turning into a hell of conditionals.

Simple versioning with experimental pipelines

To test changes to a pipeline without risking breaking the one that's in production for 25 applications, the solution we used was as simple as it was effective: instead of complicating things with Git branches or tags for each test, we directly created a new step named after the next version, like fullPipelineV2, fullPipelineV3, and so on. Watch out: this is not the same thing as pinning the library version with @Library('...@v1.2.0'), which I'll cover further down: the Git tag freezes the whole library, whereas fullPipelineV2 lets you try out a new pipeline living alongside the old one within the same version of the library. The jobs you wanted to use as guinea pigs pointed to that specific version; the rest carried on calmly on the previous one. Once the new version was tested, you migrated the rest of the jobs one by one (or all at once if the change allowed it), without a half-tested change affecting anyone who hadn't decided to try it.

Other useful steps in vars/

For one-off things that aren't part of the main flow but that you want to call from anywhere, like a notification, a standalone step still makes sense:

def call(String message, String channel = '#builds') {
    slackSend(
        channel: channel,
        color: currentBuild.currentResult == 'SUCCESS' ? 'good' : 'danger',
        message: "${message} - ${env.JOB_NAME} #${env.BUILD_NUMBER}"
    )
}

And it's used the same way, both from the complete pipeline and from anywhere else: notifySlack('Deploy completado'). That call is what runs when you invoke the step directly by its name, with no need to instantiate anything; it's simply a Groovy convention: if a script defines call(), you can invoke it as if it were a function.

Registering the library in Jenkins

Before you can use it, you have to tell Jenkins where it is. This is done in Manage Jenkins > System > Global Pipeline Libraries (in older versions of Jenkins, the menu is called Configure System), giving it a name (for example mallotore-shared-lib), the URL of the Git repository, and optionally a default branch. You can also pin a specific version directly from the job's pipeline script, which I strongly recommend so you don't get surprises if someone changes the library on main without warning while you have deployments in progress:

@Library('mallotore-shared-lib@v1.2.0') _

And if you need logic with more state than a simple step (resolving credentials or configuration depending on the environment, for example), you put it in src/ as a normal Groovy class, but implementing Serializable: Jenkins needs to be able to serialize the pipeline state at any moment, and without that interface you'll run into rather cryptic exceptions.

Why the effort is worth it

The first time you set up a Shared Library like this it costs more than the classic "four loose steps" approach, because you have to think of the whole pipeline as something parameterizable for N repos and N different environments. But the payoff is huge: trivial jobs to create, zero Jenkinsfiles to maintain repo by repo, and the peace of mind of being able to spin up a new Jenkins in parallel the day a big upgrade comes, without anyone noticing the cold sweat.