Engage 2018 / Domino App dev

After missing last year event, I’ve returned to Engage conference, which was this time on decks of SS Rotterdam (unsurprisingly in Rotterdam, Netherland). While many things were familiar, like smiling faces of people from ICS community from all over the world, many things were different this year. From personal perspective, biggest difference for me was that for first time I took my family with me. It was our first trip abroad with our one-year old daughter, along with the older one. This required some compromises, including staying quite far from the ship, which saved me from drinking too much beer at captain’s/conference bar, but also missing some small talk that goes along with it.

I haven’t written down any notes during the conference and every session added another piece to the puzzle of new ICS landscape that is just unveiling, so I hope I can describe what I think is current situation in community/ecosystem/partnerships, you name it.

Last event I’ve attended was Social Connections in Vienna last year. This was the event, where some rumors about offloading of Notes/Domino from IBM started to spread across the community and most of session at Engage tried to explain where we got so far. If I compare content that was presented at IBM Think this year, presentations at Engage moved roughly by 2 IBM years (using progress from last few years). This can be both good and scary for some, but we are getting to a point where partners will finally understand role of IBM and HCL in their partnership. Next step will be to explain it to existing customers, which is also something that marketing departments in both organizations are working on. Message wasn’t communicated straight at OGS but was given in content of sessions and also in one-to-one conversations.

OGS

Keynotes contained good balance of vision, roadmap update and entertainment. It set the stage for rest of conference. I expected more announcements during OGS, but these had to wait to individual sessions. Only official announcement was about Domino Applications on Cloud available from IBM Marketplace https://www.ibm.com/us-en/marketplace/domino-applications-on-cloud#product-header-top. Everyone was waiting for release date of V10 Domino version, which we didn’t get, but we got timeframe for invitation-only Beta 1 (June, subscribe to newsletter at https://www.ibm.com/collaboration/ibm-domino  if you want to be notified about invitations) and open Beta 2 (later this summer). Together with “Golden Ticket tours” to labs in July we should be able to get more information soon.

Domino and Cloud

This session directly followed OGS and was an eye opener for me and probably many other people. While expanding on topic of IBM Domino Apps on Cloud, second half of session was about HCL Domino offering for Cloud, running on Azure and Amazon cloud. So, if you don’t want to host your data on SoftLayer, you have a choice to ask HCL to host this for you on Azure/Amazon. Both use similar stack based on Docker, but in future there seems to be possibility of using different tools and actually providing different offerings by IBM and HCL. At this point we started to realize that HCL want to do more with Domino and can do more within legal boundaries of their contract with IBM. And more was coming.

Notes on iPad

Native Notes app for iPad was announced at Think and there were many demos of this during Engage. Currently it works only in online mode, but replication/offline capability is coming in next Betas. HCL Nomad (I hope I got the spelling right) will be offered probably directly by HCL, but IBM Notes/Domino customers will be entitled to use this in their V10 licenses.

Once iPad is done, next logical step is iPhone support, which has some challenges due to limited screen size. This may require additional tooling added to Domino Designer, which would allow easy reorganization of form elements for that screen size. This is still something that needs to be clarified as while sounding easy, it can bring nightmares if code reuse is not properly managed.
Next on the list is support for Android and later direct support for web browsers using WebGL and other cool technologies. This would effectively remove need for ICAA.

Domino and Node.js

Another announcement from IBM Think was addition of Node.js ecosystem to Domino, or vice versa. More details were shown at Engage, while biggest change for me was replacement of NRPC with gRPC as communication layer between Node module and Domino. This will still allow turning off HTTP task on Domino, but reduces requirements on client side, making it easier and more portable for deployment scenarios.

There were some performance stats shown during the session, but as it was first iteration I don’t want to jump to any conclusions. There is always a tradeoff between flexibility and performance, so I assume it will be slower than NRPC, but easier portability and consumption. Currently there is no plan to open this communication to third-party integration directly, but HCL may do this in future.

Whole idea of integration of Node and Domino could be quite easy if Domino is used just as CRUD data store, but in that case, Domino would be reduced to just storage level, with not much of added value. HCL is aware of it and their goal seems to be leveraging as much of possible from Domino apps using those APIs, including calendar and scheduling APIs in future.

To make this work nicely in secure way, there will be integrations to Passport and other IDM Node modules to handle authentication and LoopBack support for API definition and management. We’re yet to see in real world how complicated this will be to setup and manage.

Notes client updates

For V10 HCL already demoed some mail enhancements in previous webcasts, like forwarding messages as EML or deferring sending of a message. During sessions here, they also demoed revamped workspace with menu option to change background image, preference page for changing of some default colors in Notes client and some more tricks. I think HCL is going for some quick wins in this area as many of these changes were doable in current versions, but not easily accessible. This could also make some users a bit happier.

There will be some enhancements to LotusScript. Most notable for me is support for HTTP calls with focus on JSON data manipulation. There also will be new functions to leverage mobile device capabilities on iPad and other mobile devices like GPS location or camera access.

Domino updates

V10 Domino updates seem to be mostly related to automatic problem resolution and easier management of high-availability environments. This includes automatic matching of databases in a cluster, more robust of streaming cluster replication after crashes/restarts, new options to export performance data from Domino to reporting systems, etc. On other side some limits within NSF have been pushed to new levels, like database size to 256GB or document summary information to 65K. Interesting change is replacement of attachment parsers in full-text indexer with Apache Tika, which should provide better support for more file formats.

Some already announced features have been postponed to V11, including better Active Directory integration and Full-Text search indexer replacement by ElasticSearch or Solr. This will be implemented together with Domino Event Publisher (or whatever the name will be) that should allow anyone to subscribe to Domino Events and be notified about them in real time. As it’s V11 feature, we’ll have to wait for more details. Another change that may come in V11 is replacement of iNotes by Verse, but all functions will have to be ported to Verse before doing that.

No Code/Low Code

Another interesting session was about Low Code/No Code options for Domino. Notes started as Low Code solution and created whole ecosystem around it. Keeping the balance and different tooling for No Code/Low Code/High Code scenarios seems to be on HCL radar as market for such solutions that allow “citizen development” is still growing and we all know that this was market that Notes used to dominate in past.

Challenge is to do this right. There is high risk that attempt to make everyone happy will result in something that will make nobody happy (and we have seen some low code attempts like composite apps in past that failed terribly). To embrace these different levels of tooling would require more stable implementation of code sharing and versioning than we have now and probably also completely different tools, not just different perspective in Domino Designer.

HCL is currently even playing with an idea to completely kill Domino Designer and replace it with Visual Studio plugin (not Visual Studio Code to be clear).

There was even Lotus Workflow mentioned during the session, which was first time in probably 10 years when I heard that product name from an IBMer.

HCL Places

This was probably the most controversial announcement of whole Engage. It was kind of a bomb that Jason Gary dropped not just on the audience but also HCL team (not sure about IBM on this one, but I guess some people there are still scratching their heads). It should be a desktop app that will use Domino infrastructure to provide activity stream like interface for communication, allow AI integration, support audio/video messaging and still open Notes apps. All on-premises.

At first it just remined me of IBM Workplace, yet another client that had to die, but after some thinking it started to make more sense as it gives answer to Why? question that HCL developers may ask around improving Domino/Notes platform. Honestly, I think that one of the reasons why we haven’t seen any updated in Domino in past few years was that IBM killed all products that they could sell on top of it.

HCL Notes/Domino

Most important message that I take home from Engage is that HCL is allowed to do their own modification to Notes/Domino code stream, even their own products around it and they are not even afraid to build solutions that can be considered direct competitors to IBM offerings.

Domino was for long time just enhanced as messaging platform as IBM had this need internally, but with HCL running their production mails on Office365, I think we’ll see a shift towards more integrated solutions with bigger focus on apps, which is frankly currently more and more common situations at many customers as they also move their emails away.

Partners

Similar to position of HCL, traditional ICS partners have built solutions for other platforms, or even build solutions that directly compete with original IBM ideas. Best example of this for me was AppFusion Aloha that is evolution of their solution from providing integrations of external data into IBM Connections to complete product/platform that can create central location for employee engagement.

Connections/Sametime/…

There was so many sessions about Notes/Domino that I couldn’t attend any session about other products. There wasn’t much new about Connections Pink and since the man in pink suite is now on HCL side working on yellow products, I can understand some challenges on IBM side.

For Sametime, IBM Mobile Connect, Wispr and probably bunch of other products that were moved to HCL, there should be dedicated teams working on those. Currently whole engineering organization around those products has roughly 420 people with 70 new open positions (probably mostly for new cool projects like Places or Low code tooling). I don’t know exact structure of HCL engineering around these products and to be hones I didn’t even pay attention when job titles were mentioned during sessions, because in past at IBM these usually changed till next event. Maybe this will be more stable at HCL and once we get first V10 releases out, we’ll know more about allocations and priorities for individual products.

More information to come

There weren’t typical IBM disclaimers at beginning of each session or in slides about confidentiality and subject to change notes, so I assume slides will be posted online. Also, HCL tries to do things differently with playbacks open to external partners and different communication channels used during whole development process. I was never part of IBM Design Partner program, so I can’t compare old processes with new model, but I think at the end this will be always about people who are responsible for the communication and their willingness for information sharing.

HCL also announced that important customers can get direct access to assigned technical expert from labs to discuss the strategy, planning or current issues (not replacement of PMRs). This could open to partners too in future, but currently it’s just for customers.

Where does this leave us?

HCL tried to demonstrate that they have a vision for Domino ecosystem. Many things are in early stage, just on Powerpoint slides or on Jason’s Mac, but at least they were told out loud and clear. Now they have to show if they are able to deliver on these promises/ideas.

App dev strategy could be probably simplified to couple ideas:
• Support On-premises deployment but allow easy move to cloud when possible. Provide added value in the cloud
• Open Domino app ecosystem to developers from Node.js world, open Node.js world to Domino developers
• Improve the tooling for low-code development, while keeping it possible to enhance the app with more complex code
• Use all previous points to build HCL Places client that will eventually run on all platforms (and replaces Eclipse Notes client)

There were many topics there were not mentioned that much anymore. One of them was XPages, that probably got last feature updates in FP10. I don’t see it as a priority anymore. With focus on open source software, we may get to the point (finally) when the runtime is pushed to OpenNTF, but I think there will be parts of new code committed first.

It’s hard to make any business decision based on information that we got. For existing development projects, nothing changes until V10 is shipped, or some stable beta with Node.js is out. For new projects, I wouldn’t probably start new project on Domino now, unless it’s a Notes client app for existing customer. There are many technologies that were mentioned over and over like React.js, Vue.js, Angular, Electron, PWA, so there are things to learn until Domino V10 matures enough to be used as backend for such app.

Wrap up

There was more valuable content than I expected and ICS (or whatever the community name will be now) has many topics to discuss. We should see by end on 2018 what HCL is able to deliver and we should see even more in 2019 with V11 release.

The conference overall was great. Theo knows how to impress attendees and I think I ran out superlatives that I can use to describe events that are organized by Engage team. I can’t wait to see what’s coming in 2019. I have some ideas, but I think organizing conference on the Moon or Mars will take few more years to put together, but if someone is able to do this, it’s Theo.
Thank you all for great time.

Using JAX-RS inside NSF

Last week Christian Güdemann published new release of SmartNSF on OpenNTF that contains cool new feature that Christian tweeted before. With new CUSTOM strategy it allows direct execution
of Java code from REST API defined in router configuration. It's even better than it sounds as it initializes facesContext and XPages application if needed, so even access to beans works.

I needed to start to build new REST APIs for few databases, so I decided to test new SmartNSF option and also other available options for REST APIs on Domino (there are several, check references at the end for more info). Since CUSTOM strategy requires dependency on SmatNSF in NSF project and also implementation of CustomRestHandler interface, it'd force me to do more changes in my code that I wanted to. If I need to change my code, why not adjust it for JAX-RS spec anyway.

Existing Domino JAX-RS options had to packaged as plugins, which make it hard to call code that is currently in NSF. I could make it work using pieces from SmartNSF, more classloading hacks and java reflection, but calling all code using reflection isn't best for development productivity. This way I was able to build JAX-RS API on top of existing NSF without changing anything in NSF itself, which would be perfect for my current use case, but I didn't like it.

I decided to take different approach. Use code from SmartNSF to take care of facesContext initialization and create XSP library that would allow direct usage of JAX-RS in NSF. ExtLib already contains AbstractRestServlet that wraps Wink implementation that is available on Domino for long time,  so at the end this solution required far less code than I expected.

Using com.ibm.xsp.adpater.servletFactory I was able to register own factory that takes care of new servlet creation for each servlet/nsf.

This factory also passes properties map to the servlet, so it knows where to look for Wink configuration and JAX-RS resources/providers.

Those paths are evaluated using module classloader, so it looks for files inside NSF.

CustomWinkServlet then just overrides getContextClassLoader method to point it to module classloader and takes care of facesContext and app initialization if needed.

I experienced issues with SessionAsSigner access, which I need for some configuration loading, so I had to move super.doInit() call to first doService call. Problem is that if something is loaded form NSF using module classloader when NotesContext doesn't have current session assigned or first resource is not a class (at least I think), it's not possible to get sessionAsSigner after that point. I got around by loading plugin.Activator first, which I know that should exist in any NSF.

With this infrastructure in place, wrapped as XSP library and installed on my server and Designer, I'm able to call existing XPages/Java code, including existing beans. For example:

For automated mapping of JSON data I included Jackson mapper in my plugin, so it can be just registered in wink.application along with resources.

In wink.properties I kept wink.defaultUrisRelative=false which was mentioned od Jesse's blog, but I really don't know if it has to be there. I'll have to test it. 

That's all what needs to be done. Now I can just access my new REST services: e.g.

or

If you want to try it yourself, sample database also contains showCounter.xsp that displays value from appBean, which is a proof that there is no secret double-life, which may happen when you use different approach for class loading. 

Now I can just wrap existing model and controller classes with JAX-RS annotations or thin wrapper layer when needed, so I can easily use same code from XPages and REST API. 

Source code is available on Bitbucket https://bitbucket.org/pradnik/pristo_rest .

I can imagine that combining this approach with OpenNTF API can make creation of JAX-RS services quite easy as ODA wrappers already take care for mapping proprietary Notes structures to standard Java classes. This is something to try next. 

Happy coding

Resources

Many good articles, series, sagas and samples were published around Domino and REST topics. Here is list of those that I know of:

SmartNSF - https://www.openntf.org/main.nsf/project.xsp?r=project/SmartNSF

Paul Withers- http://www.intec.co.uk/from-xpages-to-web-app-part-sixteen-osgi-jax-rs-rest-access-with-oda-revisited/

Jesse Gallagher - https://frostillic.us/blog/posts/87267DB72A55133F85257E380073495F

Sedar Basegmex - https://github.com/sbasegmez/RestAssuredDemo

Sven Hasselbach - http://hasselba.ch/blog/?p=2413

Toby Samples - https://tobysamples.wordpress.com/2015/07/09/jax-rs-or-the-way-to-do-rest-in-domino-part-4/

Eric McCormick - https://edm00se.io/servlet-series/

Extension Library REST Services - https://www.openntf.org/Projects/pmt.nsf/36B7CD129ED7357A86257AC6005523E7/$file/Extension%20Library%20REST%20Services.pdf

My presentation from SUTOL 2016 - Automation is developer's friend

Last week I had a session at Sutol conference about automation for developers. It covered several samples, where I started with Reat.js front-end app, stored it in Domino nsf database and later did also complete build on Jenkins server with Selenium tests.

It was a lot of fun to put this together and even more fun, as always, was to meet all those great people from ICS (or should I call it Watson Workplace now?) community.

Sutol 2016 - Automation is developer's friend from mpradny

Repository that I used for the demo is at Bitbucket.org - https://bitbucket.org/pradnik/todoapp_full/branch/develop . If you want to try it, you need to adjust some hardcoded values as I didn't make this build parameterized and it also depends on my Jenkins configuration. Job itself that I used was simple Pipeline from SCM, just with repo address.

Let me know if you want to try it and get stuck.

Many thanks to all sponsors,  organizers and attendees, who made SUTOL 2016 great event. 

SUTOL conference 2016

It became a tradition that every autumn SUTOL organize technical conference that focuses on topics around IBM ICS portfolio. After successful switch to English as primary language last year, this year the event will be extended to 1.5 days.

Event will take place at PARKHOTEL Praha on 10th and 11th November.

More details are at conference site 8th SUTOL Technical Conference. Registration will open soon.

Two day program of conference should give more room for attendee interaction and community discussion. We hope that more international attendees will find their way to Prague as autumn is really nice time to visit. Weather is usually still warm and it's less crowded than during summer season.

If you need any reason, why you should attend, let me know. Call for abstracts is open till 9th October and sponsors are also welcomed.

See you in Prague.

XPages ${} risk of code injection possible workaround

I wasn't happy with findings in my previous post, because it can lead to security issues, but also can have performance hit when you actually need dynamic evaluation of injected code (I use it for app localization and few other use cases). After some digging I came to conclusion that it can't be easily changed/overridden because getBindingValue simply turns into createValueBinding when value is evaluated to a String with #{} inside.

Only solution I see is to wrap binding with code that checks possible injections or runs the evaluation in case I really need it. Another benefit is that I can easily log/notify when possible unwanted injection happens.

For the most simple use case that I used in demo I added two beans that implement DataObject to the app, so I can use following syntax ssan[..]/seval[..] (it's not possible to pass arguments in EL method calls in XPages, so this is a bit hacky way of doing this).

Now when I repeat my test I get:

Partial refresh to text2 doesn't update time in second text, because it was pre-calculated using seval bean.

Here is code of those beans
ssan - StringSanitizer:

seval - StringEvaluator:

Now you can have complete control. It'd be nicer if ExpressionEvaluatorImpl could be somehow replaced with custom implementation, so developers can get this level of control without such wrappers, but I haven't found any way doing so.

XPages ${} risk of code injection

While working on app optimization I experimented a bit more with 'Compute on page load' vs. 'Compute dynamically' behavior. There have been several discussions in past about possible combination of ${} and #{}, for example posts from Marky Roden, Sven Hasselbach and Paul Withers . What struck me today was risk of code injection.

In this app many elements are read from configuration documents that are loaded into beans and later used using ${} binding. This is recommended way as it is static information, so it's efficient. It works nicely until you insert expressions into your data. This way I realized that a lot of code is prone to code injection that can be contained either in configuration documents or any string that is stored and later read this way.

To simulate the issue I created simple page with one field, one button and one text:

All it does is saving entered value into applicationScope and then displaying it. Since the text uses ${} Compute on page load, I have to reload whole page to see the result immediately.

So now to the results. Normal user would probably enter something like 'Hi there'

But more advanced user can try 'Now is #{javascript:new Date()}'

(Also note that if you do partial refresh to the text, date gets updated as it's computed every time)

If he stops being nice to your app, he will switch to

 'You #{javascript:database.getAllDocuments().removeAll(true);'had'} data'

Once he gets bored, he can finish his job with

'You had #{javascript:sessionAsSigner.getDatabase('','names.nsf').getAllDocuments().getCount()} documents in address book, but 

now you have 0.  #{javascript:sessionAsSigner.getEffectiveUserName()} did that.'

(dev/pradny is name of the server as I signed the db with server ID as many admins do)

(one part is missing in previous example as I wasn't brave enough to run full version on my server to take a screenshot, you get the idea...).

The problem is not how you get the data, but how you use it. It can come from field, configuration or computation. With great power comes great responsibility. Just be aware that ugly things can happen, which reminds me of a question. Is your son's name really Robert';) Drop Tables?

Update (15.8.2016)
 Possible workaround in next post

XPagesPreloadDB more evil than good

While doing optimization of application load time I found that XPagesPreloadDB notes.ini parameter didn't work in way I expected. With quick google search I realized that I'm not the first one to hit this problem as John Dalgaard wrote about the issue few years ago https://www.dalsgaard-data.eu/blog/caching-in-xpages-not-as-straightforward-as-you-would-believe/. My goal was similar. Just preload configuration as it's loaded from several places and even worse it's loaded using SessionAsSigner.

First of my issues was caused by my stupid mistake. I copied parameter in syntax for Notes client, so it contained server name. It worked, kind of. So if you want to try it, just check the URL from request that's processed by XPages and you get:

With Notes.ini setting:
XPagesPreloadDB=dev/pradny!!test/appload.nsf/entry.xsp
result was:
http://localhost:80/dev/pradny!!test/appload.nsf/entry.xsp

Which is different context than you'd normally use, so it's actually completely different instance of your application.

So next step was to test correct path to application:
XPagesPreloadDB=test/appload.nsf/entry.xsp
result was OK:
http://localhost:80/test/appload.nsf/entry.xsp

Application scope was correctly initialized during preload and stayed for first access. This was what I needed.

I was happy for few minutes, until I started to see strange things happening. It looked like that the app sometimes stopped working. After few tests I added ApplicationListener to the nsf and found the reason. No matter what I did, the application was killed after 30 seconds, so new request after this time hit clean application, which resulted in strange behavior I observed.

I tried to change xsp.application.timeout parameter, but with no luck. It was correctly obeyed when I started the app using normal HTTP request, but the pre-loaded instance was always killed even when requests were hitting the app.

Conclusion is simple. Don't use XPagesPreloadDB as it can cause you troubles. It didn't matter if I loaded just the nsf or an XPage. Application was killed in both cases.

[26853:00002-1723520800] 08/02/2016 04:05:07 PM  HTTP Server: Started
[26853:00013-402384640] 08/02/2016 04:05:17 PM  HTTP JVM: applicationCreated()
[26853:00013-402384640] 08/02/2016 04:05:17 PM  HTTP JVM: ID: 1
[26853:00002-1723520800] 08/02/2016 04:05:37 PM  HTTP JVM: applicationDestroyed()
[26853:00002-1723520800] 08/02/2016 04:05:37 PM  HTTP JVM: ID: 1

Problem probably won't have much impact on production environment as users probably won't be using the app 30 seconds after restart, but it makes this feature useless.

Also note that when XPages are processed during preload, session user name is anonymous with lowercase a, instead of normal Anonymous. So if you have some conditions in your code, make sure you use equalsIgnoreCase.

(problem was tested on 9.0.1FP4 on Windows and 9.0.1 without FP/FP6 on Linux)