Tuesday, April 15, 2014

Testing your Java code in XPages (part 1)

This topic will lead to a series of articles. It depends how deep I will get, but I will try to stay as much general as possible and avoid solutions that work just in my current case.

It's all inspired by nice hack that Maks Zhuk showed in his blogpost http://mzhuk.blogspot.com/2014/02/unit-tests-for-lotus-domino-applications.html . Since I want to automate everything, I'll take it step further with Gradle.

XPages run on top of a OSGi runtime and so they use system of features and plugins. This can lead to some dependency troubles when you just want to do plain Java work. Christian Guedemann showed how to make this work nicely with Maven http://guedebyte.wordpress.com/2014/04/07/automated-build-with-jenkins-some-progress/ . Sadly Gradle can't read p2 update sites, which can Maven do using Tycho plugin. Even when you try to Mavenize it in some standard way like using eclipse:to-maven, I couldn't use it afterwards since some plugins contain jars as libs in them (probably all plugins that are not stored as jar, but as a folder in notes installation) and this resulted in nested jars.

So if you want to get really serious about building and testing your code with provided XPages runtime classes, you might be better of with Maven (or please let me know how to for Gradle to use it). I my case I just want to test my code and as theory of testing says, I should minimize dependency on other code using mocking or similar approaches. In that case I won't be probably calling XPages code at all, so I just need it to compile my code.

In Notes world it is really hard to separate yourself from Notes API, because big part of logic usually depends on view lookups etc. You could of course create own layer that hides this, but then you are probably just increasing complexity, cost and all other bad aspects of your code and it'd be probably better to switch to different platform that doesn't provide such nice and powerful infrastructure.

Testing your code from separate project

Maks's article showed that when you change nature of your on disk project to Java, you can use it as classic Java project afterwards. Which means that Eclipse will compile it when needed and you can use it as project dependency.

I won't repeat his article here. But in case his blog goes offline, the trick is in adding
  <buildCommand>
    <name>org.eclipse.jdt.core.javabuilder</name>
    <arguments>
    </arguments>
  </buildCommand>
</buildSpec>
<natures>
  <nature>org.eclipse.jdt.core.javanature</nature>
</natures>

into .project file of the on disk project.

Good thing about this is that Eclipse knows about all other dependencies, including plugins, so you don't have problems described above.

One small warning. If you run your tests and change your code from a link stack trace, you are changing it in your on disk project. So you have to synchronize it with the nsf to get it there. And you know what happens if you change code on both sides, ... .

If you have your on disk project configured this way, you can easily create another Java project in your workspace that has this project as dependency.  And then you can do any testing you want, but since we are in Notes environment, we might have few problems that you have to address and I'll get to them later. Some problems are:
  • Initialization of NotesThread - test are running under seperate JVM, so you have to act as in standalone Java program
  • You no longer have current database - again, test are running as standalone Java program
  • You can't rely on XPages infrastructure, because there is none when you run your tests (you should test your code, not your app on domino)
It's really hard to define straight line between unit testing and integration tests in Notes environment, but if you can get to real unit testing level, so with no dependency on external code, you are fine. Otherwise you have to work your way through the issues and even might be better of running your tests on Domino and drive them using selenium (or create some framework to run them directly, which should be feasible).

Example application

I'll continue to use same sample application from previous posts that is hosted at https://bitbucket.org/pradnik/gradledominosample . For this demo I'll move my simple computation from XPage to a Java class. At first it'll have no dependency on Notes or XPages classes, to keep it simple.

Then I'll create new project in same repository to use it for tests.

Adding Java to XPages

I'll describe it quickly, because you probably know how to do it if you are reading this :). I like to follow approaches that Jesse Gallagher uses in his XPages Scaffolding . So I have a class called SimpleController that does simple computation:



 I can use it from a XPage for example in this way:

 and

 But I want to test my class directly, not using XPages. Adding it to the XPage on other hand will test it using my Selenium test from previous post too.

Creating Java project for your tests

Create plain Java project and put it in same Git repository. I use layout of source code that is used by Gradle so I had to reconfigure source folder to /src/test/java.

Then add your on-disk-project as dependency

Remember add odp version (that you've modified to show as Java project), not the version with .nsf at the end. That's actual nsf version that uses Eclipse virtual filesystem and your test classes wouldn't see your code.

Then you need to add library that you use for your tests. In my case I have bundle of TestNG, Mockito and Powermock (that I had to use to get arround some issues noted above). But for this example TestNG would be enough (or JUnit if you preffer).  I've also installed TestNG plugin into my Domino Designer.

My test class looks like:

OK, it's just for demonstration and I probably could test more even in such simple case, but it's for a different debate.

Since I have TestNG plugin in my Domino Designer, I can right click on that class and run it as a TestNG Test and I get:








If I break my Java class and multiply for example by 3 (remember, if you change it in nsf, you have to sync with odp), I get an exception (this time from TestNG view, but same is in console too):




Never trust a test that you haven't seen to fail :)

But wait, I was talking about Gradle not Eclipse. So let's switch gears.

 Building Java in XPages with Gradle

We want to automate all of this proces, so at first we have to teach Gradle to build our Java file.  We already have a build.gradle file in on-disk-project, but it doesn't tell Gradle that we have some Java in the project too. We have to enable it using:
 apply plugin: 'java'
and then tell Gradle where to look for sources
sourceSets {
    main {
        java {
            srcDir 'Code/Java'
        }
    }
}
That's all.

Building test project with Gradle

In our test project we have to tell Gradle that we need TestNG and we work with our XPages app. I used standard project layout, so it know where to look for source code of my test classes. Complete build.gradle is:
apply plugin:'java'

repositories {
    mavenCentral()
}

dependencies {
    testCompile group: 'org.testng', name: 'testng', version: '6.+'
    testCompile project(':xpagesApp')
}

test.useTestNG()
test.outputs.upToDateWhen { false }

I also added testJava  in settings.gradle in root of Git repository.

It's that simple because my code currently doesn't depend on Notes or XPages classes. Also it downloads all required libraries from Maven repository.

Running test

You can now run your tests from console using for example gradlew :testsJava:test
Z:\git\Testing\SampleDomino>gradle :testsJava:test
:testsJava:compileJava UP-TO-DATE
:testsJava:processResources UP-TO-DATE
:testsJava:classes UP-TO-DATE
:xpagesApp:compileJava
:xpagesApp:processResources UP-TO-DATE
:xpagesApp:classes
:xpagesApp:jar
:testsJava:compileTestJava
:testsJava:processTestResources UP-TO-DATE
:testsJava:testClasses
:testsJava:test

BUILD SUCCESSFUL
 If you run it with same command as in previous post gradle buildToServer test , it does all steps. It builds nsf from odp, pushes it to your server, runs your local Java tests and runs Selenium tests on that nsf. All with this one line.

Conclusion

With this infrastructure you can easily split your code in multiple projects and make them talk to each other. One of my biggest reasons is to keep NSF project clean, with no test code or resources. In such case you should also watch what you commit back to your repository and what gets synced back to nsf. Many files are created for tests and you may carry some garbage that you don't want.

In next part(s) I'll focus on issues that we have when calling Notes API and XPages classes.

Sunday, March 23, 2014

Building XPages with Jenkins video

Few people told me that all this automated building is definitely nice, but they don't get why they should bother. I decided to record my test Jenkins machine, so it might show them what is this all good for - it's doing work for them (so you can show this to your boss and tell him/her you will be more efficient if you use such tools).

Anyway, this video just runs project from previous post using Jenkins.
  • At first I committed my code with an error that is later detected using Selenium test (it returns 300 instead of 200)
  • Then I fixed it (using notepad) and committed again

 
I did few cuts in the video, because often not much happens. I hope this will bring more XPages projects to Jenkins or any other CI tool.

final note - I really haven't touched that machine. No keyboard shortcuts :)

Gradle Domino testing

I mentioned in previous post that once you have db on server, you can easily test your app using Selenium. Here is proof for that:

And let the magic happen.

What it does?
  1. Creates local nsf from on disk project
  2. Copies nsf to server
  3. Runs Selenium test using TestNG on this app
App is really simple. Just one calculation using viewScope, but good enough for demonstration.

More Gradle goodness for Domino

I created new release of plugin for integration of headless Domino Designer into Gradle builds. Biggest changes are:
  • Rename to Gradle Domino Plugin (most projects around are named after Domino not Notes, so I wanted to keep it aligned)
  • Added basic error checking and output from HEADLESS.log
  • Task for NSF copying
  • build-in tasks in plugin for building nsf and copying to server
 All changes follow Gradle convention over configuration paradigm, so usage is now even easier.

Since I renamed the project I also renamed repostiory to https://bitbucket.org/pradnik/gradledominoplugin (sorry for any troubles) .

 

Access to Domino API

One of most important improvements is introduction of DominoAPITask that can wrap any other task that call Domino Java API. For these operation you need to add Notes.jar to your configuration, so it can call required libs. Also you may need to add Notes programm directory to system PATH to find required DLLs.

In future I'll probably add OpenNTF Domino API too, because it's much easier to use when doing more complex stuff (now there is just one createCopy call) .

 

Simplest usage

 I also added basic sample and I'll probably add more samples later, instead of more complex documentation. So you might just check Samples/Basic directory

If you have gradle installed on your machine, you need just 2 files to add to any NSF on disk project to get this running.

First is gradle.properties to set path to libs directory. In previous post I set properties from command-line, but then the command is to long to type, so you might want to add those properties somewhere else. You can add this file to the project, or your home directory. There is resolution mechanism in gradle that will take care of it.

Content of the file is simple:
dominoLibDir=C:\\gradle-lib
It is just used in build to tell Gradle where to look for our jars. This folder should contain Notes.jar and GradleDominoPlugin jar

Now to main build file. Since we now full obey convention over configuration, our build file is really simple.
 buildscript {

    dependencies {
        classpath fileTree(dominoLibDir)
       
    }
}

apply plugin: 'domino'
That's all. Just tell Gradle that it should use our plugin.

Now from command-line run gradle buildNSF and you should get NSF in your local Notes data directory. Name of the NSF is taken from name of Gradle project, which is name of the directory.

Customization

If you don't like any of defaults, you can change them. buildNSF tasks has following input parameters:
  • odpPath - change path to On Disk Project
  • nsfName - name of target NSF
If you check sources, there are more parameters to change wait time, location of data directory (which is read form notes.ini), location of notes program directory (which is C:\IBM\Notes by default)

If you want just create nsf called "myCool.nsf" add following lines to your build.gradle
buildNSF {
    nsfName='myCool.nsf'
}

Copying to a server

If you want to get your nsf to a server, you have 2 options. First is to use provided task buildToServer, sencond is to use CopyNSFTask and configure it.

Domino plugin adds extension that currently only contains name of server that you plan to work with. You can set it using domino.dominoServer name, e.g. domino.dominoServer = 'dev.pradny.com'

So your complete build file would look:
buildscript {

    dependencies {
        classpath fileTree(dominoLibDir)
       
    }
}

apply plugin: 'domino'
domino.dominoServer = 'dev.pradny.com'
Now just run gradle buildToServer  and you should get first created nsf locally and then copied to your server.

What's next

There are two features that need to be added. Integration with test framework and filling nsf with test data. UI testing can be added now using standard Gradle tools, data manipulation may require some DXL processing or OpenNTF Domino API.

Notice

Sometimes Designer get stuck during load. I don't know why and since it did that even when I didn't change build logic it's probably some bug that is caused by being just Tech-preview. Usually killing designer.exe and nlnotes.exe helped. If this gets too common, I might introduce mechanism in build file that would do that automatically

Saturday, March 8, 2014

Build XPages app from Git with Jenkins

Since introduction of Headless designer tech-preview in Domino Designer 9.0.1 many people started to think about build automation. Now is time to try it with Jenkins - popular free build automation server.

I've used Gradle as a build tool, which uses Groovy language. So some G-stuff which all the cool kids use in their projects (I always wonder why so many project names start with G even when they are not in direct relation to Google).

*disclaimer*
Most of these tools I haven't used in production, so I might break some best practices and take other shortcuts.
Also all code is written just as a proof of concept, nothing more.
*end of disclaimer*

Required tools:

  • Windows machine
  • Java JDK
  • GIT
  • Domino Designer 9.0.1
  • Jenkins
  • Git repository - bitbucket in my case

Jenkins setup

I did use Jenkins installation with GIT and Gradle plugin. If you are new to Jenkins, there are many tutorials online how to setup Jenkins. Recent tutorials for Android app builds use Gradle too, so you might want to check them.

If you just want to get it up and running, following steps did it for me:
  1. Install Jenkins
  2. In windows configure Jenkins service, so it can interact with desktop

    This might not be necessary, but egmar had some problems when running as a service in his TeamCity integration and I had problem when my ID file was stored on VMware shared folder mapped as local drive, which couldn't be accessed by services, and without this I didn't know why the build was stuck.
  3.  In Manage Jenkins set-up security (it should ask you to do it using small banner on top). I just used settings that allowed all logged-in users do anything.
    This is probably necessary only if you want to access private Git repositores.
  4. Install plugins - in Manage Jenkins / Manage Plugins / Available plugins just check Gradle and Git plugin and let Jenkins install them
  5. If your GIT and Java installation are not in system PATH, you might need to add whose in Manage Jenkins / Configure System
 Now your  Jenkins should be ready.

Create job to fetch project

I hope you have all your projects in GIT (or any other SCM) repositories, so you can pick any to try. I recommend small ones first, since for example IBM Teamroom template took about 8 minutes to build in my test VM. And this is not good for experiments.

If you just want to follow this article, repository I used is public, so you can get it from Bitbucket too.

  1. Click on New Item and create your first job
  2. Give it a name (ideally without spaces to avoid troubles later with my GradleNotesPlugin)
  3. Select a free-style build and click OK
  4. Configure Source Code Management to fetch you repository

    I used my repository bitbucket.org/pradnik/testing.git and develop branch. It contains one NSF project and one Java project for tests that I want to integrate later.
  5. Optionaly configure build triggers.
    My Jenkins is not publicly available, which is why I couldn't trigger the build from Bitbucket, which is recommended way, so I used just Poll SCM option.
Now you should be ready to start your build and fetch the repository to local file system.

Save it and click Build Now. When done, you should find your repository locally under Jenkins/jobs/ProjectName/workspace/


Build the Notes app with Gradle

You might notice that my project contains files like settings.gradle, build.gradle and gradlew.bat. These are used for Gradle build.

Gradlew.bat together with gradle directory is just a wrapper arround gradle. It allows anyone to run Gradle build on machines even when they don't have Gradle installed, because it takes care of everything.

This is good for our testing, as we can try to launch the build from our on-disk-project without using Jenkins and then get back to Jenkins with configuration that works.

Gradle Notes Build plugin

To keep our builds flexible I decided to create plugin that hides all the calls to Domino Designer. You may as well write it in build.gradle file, but this is better for reuse.
Originally I wanted to write the plugin in Java, but I had problems with task parameters, so I had to switch to Groovy (this is first piece of code I have ever written in Groovy, so don't blame me for any mess, please).

Again, plugin can be build with Gradle. But if you don't want to do it, just download it here as a jar.

Repository
Project for this plugin is publicly available on Bitbucket  https://bitbucket.org/pradnik/gradlenotesplugin

I won't cover details how to build a Gradle plugin here as many tutorials exist. I just want to show main code of task that we will use



All it does is same routine as in batch file form my previous posts. It has 3 parameters
  • filePath - path to NSFproject (actually .project file)
  • fileName - name of NSF file created
  • notesDir - path to Notes directory
 It creates command in syntax from Headless designer wiki runs it and then waits in loop for notes to finish.

You can see that there is currently no error handling and also the time limit can be low (I probably almost hit it with Teamroom). But this will be improved in future releases :-) . Also when you run the build, it doesn't show output from designer, which might be usefull.

If you want to modify it and build it yourself just run gradle uploadArchives and it will save jar in locally created maven repository.

Build structure

In my repository I have currently 2 projects, so I need to structure my build in some way. This is quite common in Gradle world and Gradle can do magic with it. For now we can forget TestingTestTests directory as it is not used.

Our root project Gradle file just tells gradle to build what needed and setting.gradle tells what subproject is should check.

build.gradle
apply plugin: 'java'




dependencies {
    compile gradleApi()
    compile localGroovy()
}

settings.gradle
include ':testingTest'

Main work is done in build.gradle in testingTest directory (which is my NSF project)

build.grade
buildscript {
     dependencies {
        classpath files(gradleNotesPlugin)
    }
}
apply plugin: 'notes-plugin'

import com.pradny.gradle.NotesBuildTask

task createNSF(type: NotesBuildTask) {
    filePath=file('.project')
    fileName=nsfName
  
} 

All lines are important in this case. Gradle, unlike Maven and Ant, doesn't require huge ammount of lines just to get basic stuff done.

  1. buildscript block - tells us where to find our plugin. We will set this as a command line parameter.
  2. apply-plugin - tells Gradle to use our plugin
  3. import - imports class of our Task (might not be necessary if you use full name on next line)
  4. task - runs our task with parameters (filePath is taken from current project location and fileName is passed as a command line parameter)
In real-world scenario you will configure those parameters in more sophisticated manner, but this also shows that Gradle can do simple things in very simple way.

And this is all.
Now just run:

gradlew createNSF -PgradleNotesPlugin=c:/gradleNotesPlugin.jar -PnsfName=blogTest.nsf

note: parametres are passed using -Pname=value syntax

When done, you should have blogTest.nsf in your Notes data directory.

Calling Gradle build from Jenkins

Last piece in our puzzle is hooking Gradle into Jenkins build. So return to Jenkins web page and open configuration of your job.

  1. At bottom click Add build step button and select Invoke Gradle script
  2. Select Use Gradle Wrapper
  3. Add parameters that we used during test launch
  4. As Task use createNSF
  5. Save it
Run your build and enjoy power of automation.

You can experiment with  build triggers and other Jenkins options, but from Notes - Jenkins integration it is all for today. In future I want to and task to move created nsf to server (for now you can use batch file method from my previous posts, run test including Selenium, etc.

------------------------------------------------------------------------------------------------------------------------

Troubleshooting note - sometimes Designer didn't want to start. Currently I don't know if it was caused by calling it from local service and as a user during experiments, or was it something else. After reboot it was fine again.