Serialization, viewScope and Exploding XPage after Upgrade...
This is not a new thing. I experienced this problem while working on my custom controls. Some history:
I have designed a Captcha Control a while ago. I were using this control in LUGTR site (8.5.1) and it was working fine. One day, I tested this on my development server (8.5.2) and it failed. I somehow corrected the issue with some stupid modifications and put it under the pile to be explored later.
After a time, I were designing my xProperties control for OpenNTF contest. When it's all done, I moved all components to a fresh database and BAM! It wasn't working. I couldn't solve this for a while and then saw a tip on Dave's "XPages Gut Check" post. I contacted him and we did a couple of tests. Finally the great master, Tim gave us a long explanation of the problem :)
Before the explanation, I will show you what's going on.
First, create a test database on Domino 8.5.2 server. A new feature came with this version. I'll explain what it means in the next step. For now, we will disable this to simulate 8.5.1 behaviour. Open Application Properties and change the following setting and restart the HTTP task on the server for convenience:
Now create an XPage as the following:
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
<xp:this.beforePageLoad><![CDATA[#{javascript:viewScope.testJS=function testJS(parameter) { print("got this: "+parameter); }}]]></xp:this.beforePageLoad>
<xp:inputText id="inputText1"></xp:inputText>
<xp:br></xp:br>
<xp:button value="Click Me!" id="button1">
<xp:eventHandler event="onclick" submit="true" refreshMode="complete">
<xp:this.action><![CDATA[#{javascript:viewScope.testJS(getComponent("inputText1").getValue());}]]></xp:this.action>
</xp:eventHandler>
</xp:button>
</xp:view>
It's a cool trick. What we did here is to create a Javascript function, put it into viewScope. We can use this inside a custom control for instance. So we can communicate between custom controls and their mother pages. But be patient before implementation :)
Now running this code:
It works and prints some string into the console:
Now, get back to the XPages tab under the application properties page. Normally, when you create a new database in Domino Designer 8.5.2, the setting we just changed will be like the following. So you change it as "Keep pages on disk" and restart HTTP task, again for convenience:
Now, run our sample page again:
Oops! WTF!
We didn't change anything except accepting the default performance option in the application properties. It gives "java.io.NotSerializableException: com.ibm.jscript.std.FunctionObject" error now.
Let's change our code as the following. This time, we will use sessionScope, instead of viewScope.
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
<xp:this.beforePageLoad><![CDATA[#{javascript:sessionScope.testJS=function testJS(parameter) { print("got this: "+parameter); }}]]></xp:this.beforePageLoad>
<xp:inputText id="inputText1"></xp:inputText>
<xp:br></xp:br>
<xp:button value="Click Me!" id="button1">
<xp:eventHandler event="onclick" submit="true" refreshMode="complete">
<xp:this.action><![CDATA[#{javascript:sessionScope.testJS(getComponent("inputText1").getValue());}]]></xp:this.action>
</xp:eventHandler>
</xp:button>
</xp:view>
It works now! Seems funny huh? In fact, sessionScope and viewScope are coming from the same HashMap implementation in Java. However there is a problem with the setting that XPages team changed.
Let's explain the problem.
First, you have to be aware of the Serialization issue.
Serialization is the ability to convert some data structure to a type that can be stored to a file/memory buffer and/or transmitted over a stream. I knew this issue from my final project in the university. I were dealing with an SIP application and I were using socket streams to transmit lots of information between clients. One day, I implemented an additional property to my objects and my class became 'non-serializable'. Just two weeks before, I found myself in the darkest and strangest tunnels of Java before tuning my classes to implement serialization interface.
How are we affected with this complex notion?
Tim has explained this issue in a very deep detail. It is related to the JSF page lifecycles. When XSP server serves a page, it constructs a 'in-memory' tree structure. This tree-structure, we can call it as the page, contains many objects like div elements, viewScope, etc. The page is being used to operate between states and changes.
In 8.5.1, this page was being stored in memory. Think of hundreds of users requesting the same page. Memory will be filled with hundred 'page's in memory for each request. That's why XPages consumes so much memory but it feels very fast after the initial launch.
XPages team creates an additional method to increase scalability. After 8.5.2, XSP server has the ability to store those pages in disk. It decreases the performance because of the increase in disk activity, but it improves the scalability which means you may serve more and more users with the same memory space.
However, XSP server is using serialization to store the page into the disk. So if you have objects that are not serializable in your viewScope, it will fail writing these pages into the disk and throws the error above. I have experienced this issue with my custom controls. My custom controls can work with multiple copies. To optimize this methodology, I were putting all necessary functions and fields into a JSON-style SSJS object and using them for custom control interactions. If I put this global object into requestScope, I would not be able to use it in partial refresh events. On the other hand, applicationScope or sessionScope will be waste of resources. So I were using viewScope to store my global object. It was a realy practical method for my job
Any object containing javascript functions will fail when put into viewScope; because XPages team has not implemented serialization for SSJS functions.
Is it a bug or regression? This is an ongoing discussion. IBM created a feature, a method working previously stopped working in a later version which is the formal description of the regression bug. However, IBM claims that storing SSJS functions is not a good practice and not accepting this as a bug (I don't know the recent situation).
This is also valid if you store standard Domino objects like NotesDocument, NotesView, etc. Because They cannot be serialized as well.
However, it is not adviced to play with these objects. Tim advices using primitive references instead... Because these objects are actually based on C and C does not have an automated garbage collector. That also means those objects cannot be serialized in any future because garbage collector occasionally deletes those objects' memory pointers and this worst-practice will become toxic (term is nicely taken from Tim) for the application.
I have tried to explain the issue. Hope this helps.
Comments (8)
To add to the argument of regression / bad practice, the obvious response to 'you were doing it the wrong way' is 'we did it that way because the documentation didn't tell us we shouldn't, because the documentation was lacking.' Enough bitching, though, a compromise would be to allow the setting at server level in the xsp.properties. So many other Application Properties can be set there, it would be nice if this could be too.
The main reason I've moved to Java is because of the limitations of the SSJS editor and error logging. A number of times I have struggled because error logging doesn't give an error line number. Java improves that, but has it's own challenges, for example, how it's possible to get a result from database.getFilePath() even after database is recycled. David is right, there is a big learning curve and coding is slower (even taking into account the ability to have code templates in Java).
Sorry for just echoing Nathan's sentiments verbatim, but this isn't Google+, so I can't simply +1 his comment. :) Something used to work... now it doesn't. That's the very definition of a regression.
I'd actually be tempted to agree that this is simply a matter of new features requiring new best practices, were it not for two glaring problems:
1. As Serdar illustrated, "Keep pages on disk" is now the DEFAULT. So, if a developer had been creating JavaScript "classes" in script libraries, then creating instances of these within the page, and, where appropriate, storing these instances within the viewScope (which, until last summer, I would have considered to be a best practice approach), under 8.5.1, it would have worked. So the developer upgrades to 8.5.2, creates a new app, and continues developing as he always has. Now it doesn't work. He gets the serialization error... and doesn't know why until he asks someone who does, as Serdar did. If the default were to use the server setting, then a given server could be set to store all pages in memory, and this could be overridden in specific apps that need to be more scalable by overriding that default. But IBM, as a large company, tends to focus on their largest customers, and larger companies are more likely to create apps that need massive scalability, so "keep on disk" is the default. So the developer has to remember to change this setting in EVERY application. While annoying, this would be almost acceptable, were it not for my second concern:
2. We weren't warned. If there was any warning at all that this would be an issue, it must have been on some Design Partner call I was unable to attend. There certainly was no clear communication to the customer base as a whole indicating that this style of development conflicts irrevocably with the new features being introduced, no documentation on recommended alternative patterns that play nice with component tree serialization. The new feature simply arrived, complete with toxic side effects, and those of us who stumbled upon the problem earliest were simply told - after asking why our code no longer worked - "well, duh, you shouldn't be storing functions in a scope map anyway".
Serdar points out yet another issue: the instinctive way for a developer to respond to this change is to find a way to continue developing as he previously has without consequence. As it stands, this means bumping up the storage to a higher scope, such as the session or application. But there are two problems with this as well. As he mentions, to store something in the session scope that used to be in the view scope just because that doesn't blow up means that you've promoted something that was more appropriate to store with the view to the session, and are therefore cluttering that scope with objects that need to be periodically cleaned up manually, because they didn't conceptually belong in that scope to begin with. Without this manual cleanup, these orphaned objects consume unnecessary memory, reducing scalability... ironically, defeating the purpose of changing the serialization setting to begin with.
Even more worrisome, however, is that even this adjustment is only a temporary fix. XPages currently do not support true clustering. If a user is accessing server A at initial page load, but an IP sprayer redirects him to cluster partner B when sending the POST data for a button click event, for example, server B doesn't know that page instance exists, because even though the view scope for that page was serialized on server A, that serialized output is not replicated to B. Because a serialization process is already in place, however, it now becomes at least feasible for IBM to include this information in cluster replication events in subsequent versions. Trouble is, if the same page is contributing to, and/or relying upon information stored in, the session or application scopes, then cluster redirection still fails unless those scopes are also serialized and replicated as well. Hence, if IBM were ever to implement this, then non-serializable information such as SSJS functions are no longer safe even in the higher scopes... we'd be back in the same proverbial boat: a new feature has been added, so we must adapt our development practices to prevent that "feature" from behaving as though it's a "bug".
One final thought: this issue does not only affect SSJS. This affects any object stored within a serialized scope. So the solution isn't "just write everything in Java"... one must understand what serialization is, how it works, how the application properties determine whether it is a factor in application behavior, which built-in objects and object types are safe - and which are unsafe - when serialization is a factor, and how to develop one's own objects (regardless of language) to ensure they are serializable. This is a lot to ask of any Domino developer... and, frankly, IBM forgot to ask. Suddenly folks are just getting bit by this with no way of knowing why, and they're reaching out to others for an explanation. To be told, "you were doing it wrong to begin with, but now it matters" is not easy for a developer to swallow.
Isn't the definition of a "regression" where the product breaks working code? And isn't the write to disk feature the new DEFAULT in 8.5.2 / 8.5.3? So it's not actually turned on by the user. Existing apps might just break after an upgrade but there happens to be the "workaround" where we can turn off "best scalability"?
Because that's what we love to do... turn off "best scalability". :)
@Stephan I'm saddened to hear you repeat that IBM party line about it not being a regression. "Don't turn on the write to disk feature" might as well be "don't install the new version."
@Stephan,
Beyond the bug discussion, I think what IBM should do is to enable serialization of SSJS scripts. I agree with David about this.
Prototyping javascript classes for different purposes is a very useful practice and we should not sacrifice scalability for that.
Furthermore, if we are allowed to use these objects in other scope variables, we should also be able to use them in viewscope. I didn't see any documentation that warn developers not to use those in viewscope. If there, please somebody refers...
Just think of a developer's point of view. I experienced this situation. You are planning a whole logic according to the approach above and BAM! It fails and you find yourself redesigning all of your structure.
About regression discussion, it would not be regression for instance, if XSP server notices an object that cannot be serialized and prevent serialization for that specific page with a nice warning in the Notes Log. This is not a database specific issue. You transfer your custom control to a fresh database and it fails. With all my respect, it's a regression.
Great post! If IBM is listening I still think this is a big deal.
Here's what I THINK this means to the Languages that are available in Domino.
LotusScript - All Purpose, great for putting business logic in custom classes
SSJS - All Purpose, but contrary to best JavaScript practices, you can't fully use objects within the scoped context.
Java - Great for business logic, but a MUCH higher barrier of entry to get started.
So I'm still REALLY bummed about this. I admit I and others need to learn Java if for nothing else, but to better integrate packages like iText and the debugger and such. But this "Feels" like another attempt at IBM to push us towards Java.
I don't want to extend the product - which would require Java. I just want to create business applications to solve problems. I SHOULD be able to do that effectively without needing Java.
I would love to be able to use SSJS like LotusScript and be able to create objects that can be stored in scoped variables regardless of the performance settings.
Just my 2 cents.
Nope, not a bug. You set "keep in memory" and it behaves (including the memory limitation) like 8.5.1, so your applications run as designed. To take advantage of the new capability you need to adhere to its constraints. You can argue, and I would agree, that the function leaves room for improvement. So not a regression (old code works with the settings that match the previous server), but an unpleasant need to update your code to take advantage of new capabilities.
I have a related problem that I haven't seen discussed anywhere yet and thought I would mention it here.
Starting with 8.5.2 Fp3, the 'Server Default' option on the Server Persistence option may cause you problems with some things. I saw it manifested when using the "Keys" of a view Panel would cause the page to return a Null Pointer Exception.
Turns out IBM moved the 'Server Default' settings to the <install path>/data/properties/xsp.properties file on the Domino server. (Specifically, it is value of the xsp.persistence.mode property in that file, in the JSF PERSISTENCE section) . The issue is that with that version, the file does not exist (only a sample file is included).
The Domino server has no fallback value if the file doesn't exist.
So the workaround is to create the file or choose one of the other three options for that setting in the nsf files.
Like mentioned above the Server Default would have been to 'Keep pages on disk' option.
This problem has been issued SPR #SRKM8KJ7Q2.
I haven't head of any time frame for a fix but development seems very concerned and intends to fix it quickly