Skip to main content

XPages and RichText (part I) - ACF

 This post is long overdue. I originally planned to write it to document the issues and solutions while working on the app, but as the problems kept appearing I wanted to make sure that I have all the answers. Now I know that I probably won't have them in the near future as I'm seeing new issues even after having the app in production for some time. 

The project was about a webization of an existing app that served well in Notes client UI for many years. It's a classic support/helpdesk app that once used to be a great example of Notes flexibility and integration of mail functions with app code. After all the mail database was just another app using the same APIs (mostly). Unfortunately, this is not the case for XPages as iNotes or Verse has nothing in common with XPages, so the APIs are not battle-tested for all the scenarios. I remember doing a similar project during the early days of XPages and I must admit that it was more painful, but mostly due to a lack of community resources and experience. Some of the issues that I'll be mentioning are fixed in the upcoming 12.0.2 version, so it may get easier to maintain the app eventually. 

There are still many great blog posts about XPages, RichText and MIME issues, so I'll try to refer to them if possible. Unfortunately, some of the old blogs are no longer available, which means that links e.g. from StackOverflow answers are no longer helpful. 

Before starting the project we did a proof-of-concept to see if it was even possible to implement all the mail-related functions in a way that will work for the users. The app uses all classic mail functions like reply to all with history (and without attachment) and needs to be backward compatible, so the code must work for existing RichText items and new data stored in MIME. The PoC was successful, so we started the implementation. 

I won't be mentioning the issues in the order as we have discovered them, but rather based on their impact on the final (currently) solution.

Problem 1 - To ACF or not to ACF

Active-content-filtering is a really important feature for security, especially when we are working with emails from the Internet. We had ACF enabled (in the default) settings during the development and testing of the application. It worked well. Unfortunately, once we deployed the app to production, we started to see issues with some received emails.

ACF crashing on semi-invalid messages

The first problem that we encountered was caused by invalid inline image data in emails. The details are not that important, simply the image claimed to have inline base-64 value, but there was none. Something like:

<img src=3D"data:image/png;base64,cid=
:116c8444-683e-423d-964a-0b914d8e4b01" style=3D"position: static !importa=
nt;" title=3D"Call: " /> 

 The result was ACF completely crashing in a way that the page was not loaded:

Exception
Error while executing active content filter
Bad character exits in Base64 array
Array index out of range: -1

com.ibm.xsp.FacesExceptionEx: Error while executing active content filter
com.ibm.xsp.acf.ACFProcessor._processMarkup(ACFProcessor.java:106)
com.ibm.xsp.acf.ACFProcessor.processMarkup(ACFProcessor.java:92)
com.ibm.xsp.context.FacesContextExImpl.filterHtml(FacesContextExImpl.java:928)
com.ibm.xsp.util.FacesUtil.convertValue(FacesUtil.java:1152        

 So the users were not able to even see what was in that email.

Workaround

Custom ReadOnlyRteRenderer with improved error handling. This way we can adjust HTML if there is an error - but it's really a try-and-guess approach to find the problem.

public class ReadOnlyRteRenderer extends com.ibm.xsp.renderkit.html_basic.ReadOnlyRteRenderer  {

	@Override
	protected String encodeText(FacesContext paramFacesContext, UIComponent paramUIComponent) {
		try {
			return super.encodeText(paramFacesContext, paramUIComponent);
		} catch (Exception e) {
			MailUtil mailUtil = new MailUtil();
			Object v = FacesUtil.getValue(paramFacesContext, paramUIComponent);
			String str2 = v.toString();
			return mailUtil.sanitizeHtml(str2, true, e);
		}
		
	}
}   

and MailUtil extract that does the sanitization - I needed it also when we are copying the message content between documents, so it's called in different scenarios, triggering the ACF internally. It's trying to remove the invalid data and if even this would fail, it removes all base64 encoded images.

public String sanitizeHtml(String html, boolean errorCase, Throwable e) {
	FacesContext fc = FacesContext.getCurrentInstance();
	if (!errorCase) {
		try {
			html = ((FacesContextEx)fc).filterHtml("acf", html);
			return html;
		} catch (Exception e1) {
			return sanitizeHtml(html, true, e1);
		}
	} else {
		try {
			if (e!=null && findCause(e) instanceof ArrayIndexOutOfBoundsException) {
				//check for a known problem with base64 followed by cid
				String pattern="<img.*[\"'].*base64,cid[^>]*>";
				html = html.replaceAll(pattern, "");
				html = ((FacesContextEx)fc).filterHtml("acf", html);
				return html;
			}
		} catch (Exception e2) {
			//remove all base64 imgs
			try {
				String pattern="<img.*[\"'].*base64[^>]*>";
				html = html.replaceAll(pattern, "");
				html = ((FacesContextEx)fc).filterHtml("acf", html);
				return html;
			} catch (Exception e3) {
				OpenLogItem.logError(e3);
			}
		}
	}
	
	return "ERROR loading message content due to invalid structure - please report this issue, including link. Message still can be accessed via Notes client"; 

}
	
private static Throwable findCause(Throwable throwable) {
	Objects.requireNonNull(throwable);
	Throwable rootCause = throwable;
	while (rootCause.getCause() != null && rootCause.getCause() != rootCause) {
		rootCause = rootCause.getCause();
	}
	return rootCause;
}
       
 

then just register the renderer

<renderer>
  <component-family>com.ibm.xsp.InputRichText</component-family>
  <renderer-type>com.ibm.xsp.InputRichText.ReadOnly</renderer-type>
  <renderer-class>custom.theme.renderkit.ReadOnlyRteRenderer</renderer-class>
</renderer>

We needed it only in read mode as all editable documents would contain the already sanitized version of the data.

The app was not crashing anymore, but another problem was reported.

Unreadable emails after ACF processing

Many systems rely on formatting within the message to provide a nice rendering. Especially when data is mostly stored in a table, it can be really hard to read when ACF stripped out all the styling, except some basic inline styles. It resulted in tables with no borders, so for example, if you have some request forms with many cells, it made them almost unreadable.

Standard mail clients provide for such scenarios buttons like "Allow external content" or "Open the message in a separate window". But this did not make much sense in the app as it would potentially cause all the security issues that ACF was supposed to solve - the page would have been loaded from the same host.

We could possibly try to tweak ACF configuration, but as it's not part of the db template and would have to be deployed (and versioned) separately to all the servers, we decided to not follow this path.

Workaround
Disabling ACF - not the nicest solution, but with we did a few additional steps - see below.

Conflicts between message content and the app

One of the nice things about ACF is that it removes all the active content that can potentially interfere with your app. All JavaScript code, all globally applied styles. Once we disabled the ACF and tested many emails that are processed in the app we started to see weird issues - menus to rendered correctly, buttons not working on some emails. 

When disabling ACF we already thought about some options for isolation. Moving the content into a separate iframe would solve many of the problems, but doing it manually is quite messy. Luckily this is exactly what CKEditor is doing (together with other things like disabling injected JavaScript). 

Solution
Use CKEditor even in read mode. So instead of using a custom renderer we've switched to the standard RT renderer even for read mode.
<renderer>
  <component-family>com.ibm.xsp.InputRichText</component-family>
  <renderer-type>com.ibm.xsp.InputRichText.ReadOnly</renderer-type>
  <renderer-class>com.ibm.xsp.renderkit.html_extended.RichTextRenderer</renderer-class>
</renderer>        
We already had some toolbar customizations, so to make sure we don't show any useless buttons, we had to adjust the customization.
<xp:dojoAttribute name="toolbar">
						<xp:this.value><![CDATA[#{javascript:if (pageController.isEditable()) {
 return '[["Format", "Font", "FontSize"],["Bold", "Italic", "Underline", "Strike", "-", "TextColor", "BGColor", "-", "JustifyLeft", "JustifyCenter", "JustifyRight", "JustifyBlock", "NumberedList", "-", "BulletedList"],["Indent", "Outdent"],["Subscript", "Superscript"],["RemoveFormat", "-", "MenuPaste", "-", "Undo", "Redo", "Find", "LotusSpellChecker", "-", "IbmImage", "Table", "Link", "Flash", "-", "PageBreak", "HorizontalRule", "SpecialChar", "Blockquote", "Smiley", "ShowBlocks"],["Maximize", "Source"]]'
}
return '[["Maximize"]]'}]]></xp:this.value>
</xp:dojoAttribute>
Now we have a nice editor (with maximize button) even in read-mode. It provides content isolation and together with antivirus and antispam filters on mail gateways we can be pretty safe. 

But it's still not the end of the ACF story.

Content not being uploaded after switching to edit mode via partial refresh

Emails were working nicely, but when the main document was opened for reading and then switched to edit mode via a partial refresh the changes were not saved. There was no error or warning, nothing. Then the document was opened directly in edit mode, or reloaded with a full page reload, everything worked fine.

As I've already briefly mentioned above the CK Editor is an iframe, so to submit the changes to the server some client side code must take the value from the iframe and put it somewhere into the current form. This is handled by XSP object querySubmitListeners.

There is a different querySubmitListener rendered when the RichText is rendered in read mode. Unfortunately it uses the same id and the XSP object code does not update/replace the listeners.

Snippet from XSP object source:


Workaround
The only way to fix this was to remove the existing listener during the partial refresh before XPages try to add the new one. Another option would be to change the behavior in XSP object, but I did not want to fight with it this time...
<xp:scriptBlock id="scriptBlock3">
	<xp:this.value><![CDATA[
	try {
		if(XSP) {
			if (XSP.querySubmitListeners) {
				var listeners = XSP.querySubmitListeners;
				var listenerId = '#{javascript:getComponent(compositeData.inputId).getClientId(facesContext)}_rteSubmit'.replaceAll(':','_');
				for(var i=0;i<listeners.length;i++) {
					if (listenerId == listeners[i].scriptId) {
						//remove the listener and return as there can be only one
						listeners.splice(i,1);
						break;
					}
				}
			}
		}
	} catch(e) {
		console.error(e);
	}
 ]]></xp:this.value>
</xp:scriptBlock>
The snippet uses compositeData.inputId that is used as the id for inputRichText control.

Conclusion

With these fixes and workarounds, the message content is displayed mostly fine. CK Editor in read mode provides good isolation of message content. There were other problems with images and attachments, but I'll save this for another post.


Comments

Popular posts from this blog

Microsoft Word black box in numbering issue

This is awkward post, primarily to save the solution for future me. I have seen many people mentioning this problem over years and as I've struggled with it several times, I needed to find final and permanent solution. All editions of Microsoft Word from time to time suffer from bug in numbering. Instead of a number, black box is displayed. Sometimes it happens right after document is opened, sometimes during editing. Probably some internal structure of document gets corrupted, so based on level of corruption, different fixes could help. Many of them are listed at  https://answers.microsoft.com/en-us/office/forum/office_2010-word/ms-word-header-styles-are-showing-black-boxes/c427b21c-dcda-46ce-a506-b9a16c9f2f3f I took different approach. Since docx is just standard zip package with xml files, I decided to try if I can fix it manually. And it worked. When I extracted the docx, there was file called numbering.xml in word folder. When I examined that file, I found strange se

HCL Domino SSO with Microsoft Teams

 Microsoft Teams is probably one of the most used tools this year, it was already quite popular before the pandemic started to spread across the world this spring, but now most of the businesses I work with use it. After using it just like a chat/conferencing tool, many start to explore further capabilities of the platform. When working with Domino data in apps that are web-enabled, it can be quite easy - just add a web tab anywhere you want. The problem is, that you need to deal with user authentication. 

Quick Tip - SSJS Error line numbers

 Recently, I had to work on an app with a pretty huge server-side JavaScript codebase. Several developers with different levels of XPages knowledge worked on that project in the past, to the code is quite hard to follow. My suggestion to rewrite all the code to Java was not accepted, so we have to deal with SSJS for now. One of the most annoying things, when you work with SSJS, is that in many cases you don't know where an error is happening. If you keep things simple and allow redirection to an error page (default or custom), you get a lot of information, and usually, you don't want to scare users with that or you may even want to do something useful in a catch block to recover from the problem. If the problem is thrown directly in the XPages JS engine, it's usually an InterpretException, which contains the context information, but if you have to deal with standard Java exceptions, you don't have it. Imagine a simple scenario with a function in a ssjs lib. When you run