Skip to main content

Domino CI build with Jenkins and Docker

 I wanted to make this work for a very long time, but there were always some parts missing, so I could not get the full process running. Finally, the wait is over. The following paragraphs describe a way to build Notes/Domino apps automatically on a Jenkins server, allowing parallel builds and all "normal" continuous-integration behavior, without having to think too much about Domino specifics.

The Problem

Until now, I was running my automated builds of Domino apps using Jenkins in two ways:

The official headless-designer way, where you need to pass special commands to Domino Designer and hope for the best as the Designer sometimes gets stuck. I have this wrapped inside a Jenkins pipeline, so I have some control and can e.g. avoid parallel builds by using locks on Jenkins, but still, sometimes it just dies. Some of my headless builds run for more than 30 minutes, so it's really hard to quickly spot an issue without actually connecting to the machine and looking at what's happening in the UI (if actually anything).

NSF ODP tooling way, using a Domino server that is running on the same machine. This works well even for more complex builds that involve custom extension libraries, but again, I would not dare to run multiple builds at the same time. 

All lives on one old Windows 8 VM that I've created years ago and honestly, I'm afraid to touch it. This was quite OK during the old 9.0.1 days where there were not so many changes in the product, but since HCL started to update the JVM and other parts of the platform, it's necessary to use the correct version during the builds. My customers are using everything from 9.0.1FP6 to 11.0.1FP2, so I need to be able to use different versions for different apps and I don't want to keep more VMs around. 

So I basically wanted:

  • Target specific Domino versions during the build
  • Run builds in parallel with no worries.
  • Use pre-created set of IDs to sign/build the databases
  • Keep the existing processes for existing apps which I in some cases will probably never need to build again
Update 24.3.2020 - V12 build images
V12 Beta now allows one-touch setup, so I could replace the domino-docker base images with official ones. Files are for now in a separate branch -

The Solution

Domino is more docker/container friendly now and so is Jenkins, so making all three work together sounds like the best approach to me. A high-level summary is:

  • Keep my current Jenkins VM as is, so I can continue to use it for existing builds
  • Add a remote Jenkins agent that will take care of the new builds (or more if I want to)
  • Create a Domino docker builder image that will contain Domino, Maven, NSF-ODP tooling, Domino update site

Let's dive into individual parts.

Docker images

We'll need 2 custom docker images. One is a docker enabled Jenkins remote agent and the second one is our Domino builder image.

Domino builder image

In order to build apps using NSF-ODP tooling, we need access to Notes/Domino binaries and a runtime. The tooling can run locally, even without having the server up and running, so the most simple way to get a version-specific environment is to build a derived image from the standard Domino docker image and just tweak it a bit. I originally wanted to use the official image from V12 beta and just trigger the one-touch setup, but it currently has 2 problems:
  1. After the one-touch setup, the server starts automatically, which I don't need/want (and afaik there is no way to stop it if you run it during image build)
  2. It can't use pre-created ids unless you set it up as an additional server
Until this is addressed, I've decided to use the community domino-docker project as my base image. I've started with 11.0.1FP2.

Creating the domino-docker base image

I'm not going to cover the details here as it's really simple to follow the documentation from Just clone the repo, download required packages from HCL Flexnet, and run ./ domino

It creates an image ready to be setup when used.

Own image with custom silent setup

Then we can begin with our own Dockerfile using it as the base image:

    FROM hclcom/domino:11.0.1FP2

As I've mentioned before, I was to use existing ids. Even this image can't do that as the pre-defined pds file contains options to create new ids that can't be overridden using cfgdomserver.jar. Luckily, I've tested this while ago and I already had ids and a custom pds file ready. I just need to copy them to the image.

  COPY ids/ /local/ids/

Few words of warning here - the ids and passwords will get stored in the image. Even if you delete them later, they will be accessible as a hidden layer in the image. I'm fine with that as the server actually won't be ever started and this image should never leave my environment. If you don't want this to happen, you can check how domino-docker project is downloading the software packages during the build.

If you want to know more about Domino silent setup, just check the official documentation

Then I add my customized scripts for Domino setup and Maven entrypoint.

COPY scripts/* /domino-docker/

Script is a trimmed down version of the original that only does what I need and ends with the silent setup - $LOTUS/bin/server -silent /local/ids/fullsetup.pds /local/ids/pwds.txt

Adding Maven

NSF-ODP tooling uses Maven to run the builds (and manage the dependencies etc.). We need to add it to our image. Since I've started with the default domino-docker image, it's Centos 8 based, so I can just add the packages. Before doing that I must switch to root as the base image is configured to run as user notes.

USER root
RUN yum update -y && \
yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel maven libnsl && \
yum clean all

During testing, I've discovered a problem with Daniel described it here. I had to add a legacy support package to fulfill the requirement. 

Domino server setup

Now it's getting more interesting. First, we do our silent server setup. We need/should to switch back to notes user before that.

USER notes
RUN /domino-docker/

Domino is now configured, so we have names.nsf, in the data directory etc.

Domino Update site

NSF-ODP tooling needs a p2 update site that is preferably extracted from the same Domino version. Luckily, Jesse has another tool for this in his toolbox -

We just need to configure Maven to use the OpenNTF artifactory. The easiest way is to add a predefined Maven settings.xml. Then we can simply use maven to run it.

COPY --chown=notes:notes .m2 /home/notes/.m2/ RUN mvn org.openntf.p2:generate-domino-update-site:4.0.0:generateUpdateSite \
  -Dsrc="/opt/hcl/domino/notes/latest/linux" \

COPY operation always copies the files as root, so we need to change the owner in order to make the directory accessible for the notes user. 
It's probably not necessary to specify the src as Domino is installed in one of well-known paths.

Adjusting the entry-point

We need to adjust the entry point, so it works well with Jenkins that expects the commands to be passed through. Without it, the Domino server itself would get started, not our Maven build. My is again a trimmed down version of a script from the standard image - here. I don't need to pre-populate the .m2 folder now as I'm doing this already, so for now, it's just a pass-through exec. 

ENTRYPOINT ["/domino-docker/"] CMD ["mvn"]


That's all. Now we can build an image that can be used for NSF-ODP builds. It takes some time as we are doing yum update during the build and we are not using cached Maven repository, so the creation of update-site also downloads a lot from the internet.

(no-cache on the screenshot is just to show the full build time).

The image could be further optimized, but as I will use it only on a local network, I don't worry too much about it.

Jenkins agent with Docker support

As I've mentioned earlier, I'm running my Jenkins on a system that is not capable of running containers. Luckily, Jenkins has very flexible support for distributed builds. The agents can be physical machines, vms, or even containers. Since we are doing a docker based build here for our Domino apps, we can try to use a container as our Jenkins agent too. The Jenkins agent in this setup can run anywhere, it just needs to have a way to call the master and register itself.

Docker in Docker

For our use case, the setup is a bit tricky. We'd run our Jenkins inside docker and this container would try to start our Domino builder container inside. There are several resources on the Internet that discuss this - primarily all point to this article. The cleanest solution seems to be to actually run the docker out of docker by giving the container access to hosts docker.sock and running all commands directly on the host. 

In theory, I could try to switch to podman, which is daemonless, so it should be possible to run it inside a container in an easier way, but this may cause problems on its own and the Jenkins/Docker setup I'm using is well documented online.

Adding Docker CLI to the image

We start with the official Jenkins inbound-agent image. Then we install docker using standard installation scripts. Since the official image is running as a non-root user, we need to make some security tweaks that I'll explain below. Here is the Dockerfile:

FROM jenkins/inbound-agent
USER root
RUN apt-get -y update && \
    apt-get -y install apt-transport-https ca-certificates curl gnupg-agent software-properties-common && \
    curl -fsSL | apt-key add - && \
    add-apt-repository \
    "deb [arch=amd64]$(. /etc/os-release; echo "$ID") \
     $(lsb_release -cs) \
     stable" && \
  apt-get update && \
    apt-get -y install docker-ce docker-ce-cli
RUN groupmod -g $HOST_GID docker
RUN usermod -aG docker jenkins
USER jenkins

Running the container

When we run the container, we need to bind the docker socket /var/run/docker.sock from the host to the container, which will allow the container to run the docker commands on the host.

  docker run --init -v /var/run/docker.sock:/var/run/docker.sock \
    jenkins-agent-docker:latest \
    -url <jenkins-master-url>  <secret>  <agent-name>

 The problem is that normally only docker group users can do run docker commands and the users/groups in the container live on their own, so they have different uids and gids. In order to get them in sync, I'm creating a docker group in the container that has the same groupId as the docker group on the host (in this case 1001). It's part of the image, so if the host uses a different id, I either need to build a new image or adjust the id.

Domino app project itself

So far we've created all the necessary images. Now we can proceed to the project itself. For testing, I'm using just a database with one XPage. It follows the NSF-ODP default convention with source in odp folder. Interesting bits are in pom.xml and Jenkinsfile for the pipeline. 

First, the pom.xml contains just group/artifactid and uses the defaults for the rest (see NSF-ODP tooling for all the options)

Jenkins file is more interesting. 

First, we tell it what image we want to use during the build. I also specify a Docker label, which tells it to run it only on agents that I've labeled as docker enabled. Optionally, we tell it to bind jenkins-m2-repo volume, so we don't download all the dependencies with every build. 

In the next step, we run a "plain" Maven build that uses all the NSF-ODP magic.

As the last step, we publish the created nsf, so we can download it from the Jenkins UI. 

Nothing else is needed in the Domino projects that use this infrastructure. If I want, I can run the build from multiple branches, using different builder images with different Domino versions, nice. 

Currently, there is a small issue in the NSF-ODP tooling which may cause the build to fail when it does not find any Java source file. I had to add one empty Java class to my sample project as a workaround.

Putting this all together

We should have all that we need by now. Let put this all together. To get the overall idea of what components are involved, check the graphics:

Yellow boxes are containers created from the images that we've prepared earlier. The involved parties are:

  • git repository - where we store the Domino app project
  • container registry - where we store our images, so docker hosts can get them
  • Jenkins master - my original Jenkins server, orchestrating the distributed agents
  • Docker host - machine hosting my containers with:
    • Jenkins agent - remote Jenkins agent, connected to and managed by the master
    • Domino builder - a temporary container that is used to build the app
    • docker.sock - "tunnel" to allow the agent to create containers on the host
    • jenkins-data - volume that is used to share Jenkins workspace with the builder container
    • jenkins-m2-repo - repository to cache the Maven repository, so we don't download everything every time

Pushing images to a private registry

Since I was building the images on my local machine, I need to make them accessible to the docker host. I can export/import them manually, but the best way is to share them using a private registry. All you need to do is to assign a tag based on the layout of your docker registry and push it. In my case e.g.:

  docker tag jenkins-agent-docker
  docker push

and do the same for the builder image

  docker tag domino-appbuild:11.0.1FP2
  docker push

Now they can be used by any machine that has access to the registry.

Configuring Jenkins master

We need to configure Jenkins master, so it expects a connection from our agent. Inside Manage Jenkins > Manage Nodes and Clouds create a New Node

Then configure the details:

The important pars are:

  • # of executors - for now, I allow just 1
  • Remote root directory - I used a folder that is then mapped to the volume jenkins-data
  • Labels - I've specified Docker label, so I can limit the builds only to docker-enabled agents
  • Usage - only build jobs with label expressions (as I have other builds like headless designer too which can't run on this agent)
  • Launch method - Launch agent by connecting it to the master - our container needs to tell the master that it is available
  • Tools location - I had to adjust the Git to point to git, without this, the agent tried to execute git.exe, which probably leaked somehow from my master, which runs on Windows.

Once you save it and reopen the agent page on the master you should find further setup instructions and especially the secret. E.g.:

  Run from agent command line, with the secret stored in a file:

  echo xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > secret-file
  java -jar agent.jar -jnlpUrl http://host:8080/computer/DockerAgent/slave-agent.jnlp -secret   @secret-file -workDir "/home/jenkins/agent" -failIfWorkDirIsMissing

Copy the secret (xxxx... in the sample) as we'll need it later.

Preparing the docker host

My docker host is a plain Centos8 VM, only running docker. Since I need to use images that I've prepared elsewhere, I need to connect the host to my local repository. First, we need to log in:

   docker login

After providing the credentials, I got a warning that the credentials are being stored in ~/.docker/config.json. This is important to know as our Jenkins agent may need to pull images too, so it needs to have access to this file too. The easiest way was to add read access to everyone on that file and bind it when starting the agent. 

The second customization was changing the docker group id. I've mentioned that I've created the image with gid 1001 (which originates from my WSL2 environment). Checking this machine, the gid was 982, so I've decided to change it.

The last step is to prepare the volumes

  docker volume create jenkins-data
  docker volume create jenkins-m2-repo

Running the agent

We should have everything prepared to run our Jenkins agent and start the build. First we start the agent using

  docker run -d --name jenkins-agent --init \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v jenkins-data:/home/jenkins/agent \
    -v ~/.docker/config.json:/home/jenkins/.docker/config.json \ -url <secret> DockerAgent

As mentioned above, we bind 3 "volumes"

  • docker.sock - to give the container access to docker daemon
  • config.json - to give it access to the credentials for our repository in case it needs to download an image
  • jenkins-data - volume to share the data easily between containers

The container then needs a base URL for the Jenkins master, secret that we've saved after preparing the configuration for this agent and agent's name.

If all works fine, our agent should be connected to the master.

Running the build

Our infrastructure is ready, so we just need to configure and execute our build.  First, we create a new Pipeline project and use Peopeline script from SCM. Then we specify access to your git repository and branch that you want to build.

 Once you save it, you should be ready to go. Click on Build now and hope for the best.

(the very first build will run longer, the screenshot is with pre-populated maven cache and images already available on the host)

You can review the Console output to see the details. Interesting are commands that create the builder container and execute the build

You can then just download the nsf and use it. The whole build ran in less than 15 seconds, cool.

Further details

If anyone wants to start with a fresh environment and build similar infrastructure, it may help them to know what I'm actually running while building this:

  1. Main work machine - Windows 10 
    • WSL2, Docker Desktop, Ubuntu-20.04 subsystem for building of images
    • VMware Workstation with Windows-based VMs for Domino Designer
  2. ESXi lab machine
    • Jenkins master - Windows 8 VM, headless Designer and a local Domino server
    • Docker host - Centos8 VM with docker
  3. Synology NAS - with Docker
    • GitLab container - as both Git server and image registry


I plan to start to use this setup for all new Domino-based builds. I hope that NSF-ODP tooling will not cause any problems for apps that worked well when I used the headless designer. I hope the V12 one-touch setup will allow me to switch to the official image, but it's not that important as I'm not running the Domino server at all and I doubt that HCL will support issues in my build environment anyway. 

I will also integrate the CD part of CI/CD later and push the result directly to a test server. This should be pretty easy now.

Maybe I'll try to replace docker with podman and check again some of the shortcuts and simplifications, primarily related to the security of the setup.


Many thanks to Jesse Gallagher for NSF-ODP tooling and Daniel Nashed and Thomas Hampel for the domino-docker project. Without them, this setup would not be possible. I hope this post will inspire more people to improve their Domino build processes as it now can be pretty flexible and even based on industry standards.


Each part of the setup is in its own repository, so it can evolve independently if needed



Popular posts from this blog

XPages EL/class-loader memory leak (now with solution)

 We have recently experienced OutOfMemory crashes of XPages app server. The server was recently upgraded to 12.0.1FP1, but we were getting some panic crashes in HTTP even before the upgrade (it was 9.0.1FP10). Our hopes were that the upgrade would stabilize the server, but it's not the case. At least now I start to see what's the problem.  update 8.12.2022 There were actually 3 different leaks. I have rewritten the article to be a bit more clear. I also re-run some of the tests on 9.0.1FP10, so I assume the problems are also in earlier versions. Problem 1 The server is hosting over 1000 NSF sharing the same design + some other custom apps. Not all NSFs are used via web as the app still has classic Notes UI in parallel, so it's a bit tricky to estimate the load. By using tell http xsp show modules I usually see around 350 NSFs active. We kept the default application timeout that should provide reasonable application recycling if it's not used continuously.  We started to

HCL Domino 12.0.2, Engage 2022 and HCL Factory tour Milan

 I haven't published my recap after Engage this year and the recent HCL Factory tour in Milan is a great opportunity to write a summary about what's happening in HCL (mostly Domino) space. It's a mix of news about 12.0.2, future directions, and my impressions, so it can be a bit chaotic, but I got the impression that many people see it similarly.  Engage 2022 Engage 2022 was great (as always). I love the atmosphere in Brudges. I visited it once after Engage a few years ago and I was happy to come back. This was also the first time I had the opportunity to speak at Engage, which obviously made it a bit more stressful, but also more fun. Together with Domino Jams, HCL continued conversations with customers and partners about the future of their products at Engage. Many of these ideas were now discussed in greater detail in Milan, some of them were even demoed.  My main takeaways from Engage were: Nomad (web and mobile) are a great addition to Notes family Restyle is a great g

HCL Domino SSO with Microsoft Teams

 Microsoft Teams is probably one of the most used tools this year, it was already quite popular before the pandemic started to spread across the world this spring, but now most of the businesses I work with use it. After using it just like a chat/conferencing tool, many start to explore further capabilities of the platform. When working with Domino data in apps that are web-enabled, it can be quite easy - just add a web tab anywhere you want. The problem is, that you need to deal with user authentication.