Monday, May 25, 2015

XPages and Java security troubles

This is more a rant than classic post, but I hope it may help someone or someone could prove me wrong, which would be even better.

In recent app I use  XPages Scaffolding from Jesse Gallagher. He does some magic in his code that relies on Java reflection to access classes that you can define in your NSF as your model. As my dev server had Java AllPermission grant in global block of java.policy everything worked just fine. But just until I moved the app to production server, where this setting is not possible.

I started to get Exceptions like:
java.lang.SecurityException: not allowed to access members in class 
class model.Problem
lotus.notes.AgentSecurityManager.checkMemberAccess(Unknown Source)

The stack trace shows that  the exception is thrown from internal Notes class that tries to enforce security.

I still thought that is is not a big deal, since I'm allowed to modify java.policy on this server, so I would just grant correct permission to my app using grant codeBase "xspnsf://" syntax. But it just don't work. I checked XPages Portable Command Guide and other sources to see if I spelled it correctly. I tried to debug the code and validate the permission in code, it just looked OK. But it had no impact on the exception. Than I found a note in release notes of Threads and Jobs project.
Note that the following does not work since the Java code is put as class in NSF as opposed to a jar file in the/lib directory:
grant codeBase "xspnsf://server:0/threadsjob.nsf/-" {
I'm not willing to put my code into a jar as it would make development much harder, so I had to look for another solution.

The problem is caused by using some reflection calls between classes that are loaded by different classloaders. To kill this check you have to play games with SecurityManager and currentClassLoader() . So I decided to use AccessController.doPriviledges and wrap all code that caused my troubles into PrivilegedActions.

It is not the nicest solution for code readability, but it solved my problem and I can continue to focus on my app and not fighting with the platform. 

It took me couple hours to solve this issue and if anyone uses Scaffolding and has similar problem, you can try to use my fork, where these changes are implemented - (code probably needs some refactoring, since this is still first version that worked)

If anyone can show me that java.policy setting for a nsf could solve this problems, let me know. I have seen many posts on Stackoverflow where people couldn't get it to work and probably just ended up with AllPermission in global block.

Monday, April 20, 2015

My slides and video form ICS.UG

Last month I had opportunity to speak at ICS.UG user group together with Martin Jinoch. It was great event as I mentioned in previous post. Our slides are available on ICS.UG session list, but I also want to publish demo video and GIT repository that I used for the demo.

I wanted to do some cleanup first, but as time didn't permit me to do so, I publish it just as it was for the conference. It really just a proof-of-concept that put together pieces that I would use separately otherwise.



I had to move mymouse all the time to get correct recording, so please ignore it. It's no voodoo. 


Next plans are to all of this on current project that also involves Open NTF Domino API and XPages Scaffolding, which probably will make some parts more complicated. 

Friday, April 3, 2015

ICS.UG and Engage recap

Last week I had an opportunity to visit and speak at ICS.UG in Bremen and then continue seamlessly to Engage user group in Ghent. Both events were amazing and transport between them organized by We4IT was more than special, a beer-bus with actual bar inside. I still wonder whether it is legal, definitely would not be in US.

I’ve been to several user groups in past, so I can compare them to recent events. Here are my thoughts:

Venues and organization
Every year events are better organized and take place in nicer venues. This is especially true for Engage where Theo is raising the bar for all conferences every year. It’s going to be hard to keep this pace, but everyone is doing their best.

This year there were many new names I never heard of, some of the classic ones were missing. It’s still good sign about the market as there are definitely people interested in IBM collaboration space. Also giveaways from sponsors are getting better every year.

People are what these events are all about from my point of view. It’s amazing to meet most of the best people from the community at one place. This time even some people I never met before like Nathan Freeman came to Europe. It was nice to talk to anybody at both events.

All these events started as Lotus user groups, but as product strategy, market and also experience of people who organize those events change, focus of these events change too. And it is good. It keeps us updated what real world needs and not just our yellow bubble.

I was able to attend most of development track sessions and the message is clear. It shifted a bit from last year, where we were trying to bring modern frameworks to Domino world like Bootstrap4Xpages, to integrating Domino into general software ecosystem. With IBM focusing on Bluemix it will be more clear in future that everyone has to find own way to build systems and solutions according to their needs. Systems are getting disassembled into services that can be integrated with any other third-party services in similar fashion. Biggest hints about future are:

  •  IBM is separating XPages runtime and Notes data store for Bluemix
  • From Bluemix XPages runtime you can connect to on premises Domino data
  • You can consume Domino data from almost any tools using e.g. REST, or even directly from for example Websphere Liberty using CrossWorlds
  • Not everyone is comfortable with JavaScript

Whole container based approach to application management has many similar characteristics with Domino running classic Notes apps. It was also easy to bring an app to a customer and just ‘push’ it to existing server where defined set of services were running. Nothing new for us, right?

Everyone should just find tools they feel comfortable with and deliver best possible experience for end-users. It’s probably not going to be still the best tool in 10 years, but if we only keep looking for the best solution, we probably never create any solution. The experience we have with Notes being stable ecosystem for 25 years is probably not going to return with any tool. We should be grateful for what we had, but that’s all.

There were 2 special sessions at Engage. Session from Paul Mooney and Gab Davis about life of a consultant summarized thoughts I also noted in previous paragraph to people who still don’t get it. It was nice to see Paul again as he is one of the first Lotus UG organizers. As I’ve written already, those events are about people and as we all enjoy just to meet even when career path have taken us elsewhere. But this brings also external knowledge to the community.
Second session was about future of ‘Lotusphere’ or similar user group events as Engage. Everyone would love to share their social experience with as many a people as possible, doing many similar events in every country or region. But we all know it is not possible since we all have to do billable/real work some time and we have limited time for such social events. I’m not saying that there is only place for one Lotusphere, but there always will be high profile events like this Engage and then rest of more local events focused on information delivery. All the speakers are also attendees of the conferences and they come too enjoy them as well, remember they are not paid for the presentations, they go to these events to have fun.

Thank you
I want to thank to all the organizers, speakers and attendees, many of them I consider friends. It’s always nice to meet everyone in person. I also know that the work is not done the minute conference ends. There is always huge paperwork to get done which seamlessly goes into planning for next event. So I thank everyone for the work they do for the community.

See you all at another event somewhere around the globe.

Thursday, January 15, 2015

Fun with POI4XPages and Maven

It's been long since I wrote my last post. For part of that period I completely left world of Domino and XPages. During that time also many interesting XPages technologies and project appeared or got more popular. One of them is build automation using Maven, so my plan for holidays was to check recent posts from Christian and Jesse about mavenization of XPages libraries.

Before I got to them, a friend of mine called with a request for PDF creation on Domino. It was not for an XPages app, but anyway it seemed like a good fit for POI4XPages, which could be built by Maven. I agreed to create small demo for him. Main goal was to generate an order document which also contains table of items. It will be called by a button on classic notes form, so the plan was to use  URLOpen to download it from the server. Problem that comes with this is that users won't be authenticated, so custom security and session as signer is needed.

I had some prior experience with POI4XPages and FOP, which is used for PDF conversion. But not big. Next chapters are more about fun of learning and experiments rather than instructions how to code. Goal was just a proof of concept anyway

First attempt

So I went ahead and downloaded latest 1.2.6 release of POI4XPages for Domino and put it on my dev server and designer. Easy as usual. Then I had to check the sample database to see how a PDF can be generated. Since I needed some flexibility of getting my data using session as signer, my first attempt was with PoiBean. But problem of PoiBean is that it can't generate PDF.

Second attempt

Now I focused just whether I can just generate the document, no matter how. So I took docx sample and started to fill some fields. It was quite easy, but as the document contained some tables and a logo, it didn't go as easy as I would expect. 

First problem was that sometimes the export worked OK, sometimes not. It took me some digging in docx file itself, before I got to a "pro tip" that is in POI4XPages documentations. DISABLE SPELLCHECKER. If you don't do this, your text is sometimes split by spell-checking marks and search and replace feature of POI4XPages won't find your marks. It took most of my time in this phase.

Once this was working, I just had to get my document using sessionAsSigner and fill wgpoi:document component like this:

I know it's quick and dirty :)

Next problem was a table of items, which are saved in a text field and separated using #.  Here I just resorted to Java and created a class with a helper method that returns List<MyEntry>. With this in place, table generation was easy:

There was one more problem with the table. First two data rows got mixed together, so I had to create fake empty second row.

With all this in place, I could generate my document. But the result didn't look good (btw. original logo is replaced with openntf).

Problem was that my dev server is on Linux. When I created docx and not pdf, it was OK. Problem is caused by FOP font mechanism. It couldn't find some Czech characters, so it just replaced them with #. I used to deal with such problems is past, but luckily customer is running on Windows and there it was fine.

 (If you need to do similar platform test and don't have Domino on Windows, just run your app in Notes as XPINC. I assume you have Notes on Windows if you do development and have Designer. If not, please let me know, I would love to have designer on different platform)

Last hack I did was that I called generateDocument in afterPageLoad event to get immediate download from the URL. It was really a hack since it breaks standard processing and returns an error to Domino console. But basic flow for a demo was working and I was quite happy with that. In just few hours I got a good looking PDF with possibility of easy changes in the template (alternative was to write it by hand in iText, but this would have to been done by someone else).

And then it came.

Image challenge

After demo to the customer, friend called with 2 bad news. First was easy, he forgot one column in the table, which is probably addition of 8 lines in my code, fine. Second one was not. Customer need to add images of signatures to the order. And those images are stored in RichText fields.

OK. I knew it was too easy till this point.

After brief twitter and skype conversation with Christian I decided to continue further. He already did some progress with RichText processing, but his code is not integrated into POI4XPages yet. So I had to do it myself. Also his goal is much bigger. He wants full RichText processing. I just needed 2 images.

It's opensource

I needed to find a way how to hook my code to exiting process of POI4XPages, so I cloned the repo from GitHub . Also Christian did some bug fixing, so I could start on development branch. One of the fixes apparently fixed my issue with table noted earlier.

At this point I finally could check Maven configuration for building of XPages libraries and update sites. It worked well and I was able to build it in few minutes.

After some digging I found that wgpoi:document has an attribute for postGenerationProcess. It allows exactly what I needed - do some processing after the POI document is generated and before it is handed to FOP. Awesome.

Getting an image from RichText

It's not that hard to get image embedded in RichText. I used DXL, where it is Base64 encoded. Now I just have to hope it comes out in form that I can use with POI.

Fail one

I created simple code that took the image and called addPicture() on XWPFRun and put it in my NSF. It failed. It just raised a SecurityException, so I knew I hit a dead end here. Changing java.policy on production server was not an option. 

Clean solution is to put my code into own plugin that runs in XPages runtime, so it has higher permissions. 

Even cleaner solution would be to use POIPowerAction from POI4XPages, but Christian told me about it few days after it got here. But anyway, it wouldn't work because of Fail two later.

One of my goals was to play with OSGi plugin development and Maven anyway, so it wasn't big deal as it was on my learning list for holidays anyway.

 Small problem was that I couldn't find my plugin dev VM and I had to install new one. This didn't go as easy as I would expect. Domino and Notes installation were fine, but fixpacks failed. I use Win2k3 as base of my VMs, so it is unsupported, so this could be the reason. Anyway, if I remember correctly, what helped was to revert from JRE7 to JRE6 before Notes fixpack installation and increasing JVM memory using JAVA_OPTS for Domino fixpack installation.  

Image Exporter plugin

Goal for the plugin was really simple. Just get an image from passed Domino document and add it using POI. I decided that I need to get this working first and then I will play with Maven.
Using XPages SDK for Eclipse I got my plugin easily deployed to Domino, so I was ready to test.

Fail two

Everything seemed OK, except the document was broken. Even when I tried just to generate docx document, word complained that it was corrupted. I did some googling and found a bug report for addPicture. It's not resolved and also POI can't be easily upgraded on Domino anyway. But between comments was a post with workaround that did manual XML inserts.

This manipulations required access to schema classes that were already in place in poi.library plugin. But as we are on OSGi and those classes were not exported by the plugin, I had no access to them. Even worse, since POI used them I couldn't just add them to my plugin. This resulted in an error that classes were already loaded by the classloader. Only solution I found was to add required packages to export list of poi.library plugin. This means I need to have custom built library on the server, but I can live with that for now as I assume my plugin won't be required with future release that will support RichText.

It still didn't work. Even in the plugin I got Access denied exception. I went back to code of POI4XPages code and found some AccessController.doPriviledged magic. Yes, it is in to PowerPOIAction that I didn't know about earlier. It did the trick.

Last request was to align numbers in the table to right. I didn't want to mess with code in POI4XPages because of this, so I used another hack. I set default font for the document to Courier and filled the strings with unbreakable spaces. It looks quite nice. Here is final prototype (signatures are fake :-) ):


Great, I got my prototype, so now was time for some clean-up. First I finished "Mavenization" of my library, so it can be built from command line. Since I started with pom files from Christian, it wasn't that hard. Only problem was with BASE64Decoder. It is an internal JVM class and shouldn't be used. I tried to replace it with Base64.decode(string).getBytes(), but even when I played with character sets, I couldn't get same result. Luckily there was another option with existing classes - DatatypeConverter.parseBase64Binary. It worked as expected.

Second I needed to clean my way of triggering file download. I again found inspiration in POI4XPages demo database and moved it to a REST service. It was quite easy

Now when I call  "objpdf.xsp/docdownload?docid=B3980D1E0D88E207C1257DBC004E8181"  I get back my PDF.


It was both fun and educational, so it fulfilled my needs. I hit some blockers, but since all the code was open source, I could help myself. Also Christian helped when I needed, so many thanks to him for his help and for POI4XPages in general. 

If you want to check my code, my plugin is available on BitBucket . It's not clean, but works for me. Feel free to use it and if you need me to add there some license files, let me know. 

XPages app is quite simple. As it contains customer related info, I didn't share it. But all that is added is:

public void addSignatures(Document doc, XWPFDocument xdoc){
ImageExporter ie=new ImageExporter(doc);
  HashMap<String,String> signatures=new HashMap<String, String>();
  signatures.put("SIG01", "sig01");
  signatures.put("SIG02", "sig02");

<wgpoi:document id="docContract" downloadFileName="order.pdf"
pathInfo="download" pdfOutput="true"

I still need to add some kind of security mechanism to my app, but it will probably just pass some insensitive info from the document to the URL, maybe even just noteid, and compare it while opening.