HTML5, CSS3, jQuery, JSON, Responsive Design...

Getting around Domino’s Ajax lookup restrictions

Michael Brown   June 13 2011 06:41:07 PM

Ajax and Domino

By default, and Ajax lookup to data in a Domino view will return only 30 rows (i.e. document entries) of information.  (At least that's true in ND7.  Maybe it's different in 8.)  You can raise this limit by using the count parameter in your server URL, e.g. to up the limit to 100 rows, you set the count parameter to 100:

var serviceURL = '/path/db.nsf/viewName/?ReadViewEntries&Outputformat=json&count=100'
// now make the Ajax call using the serviceURL variable


If you don't want any limits on the number of lines returned by your Ajax lookup, you set the count parameter to -1, like so:
var serviceURL = '/path/db.nsf/viewName/?ReadViewEntries&Outputformat=json&count=-1'
// now make the Ajax call using the serviceURL variable


No you'll get all of the rows in your view returned to your Ajax call, right?

Sorry, no you won't.  There's a server limit too, and that is determined by a server variable called Maximum Lines Per View Page, which you'll find set on one of your server config documents. It may be the server doc itself, or it maybe on one of the web config documents, depending on your setup.  When you do find the server variable, you'll likely find that it's set to only 1000.  That means that Ajax lookups to your Domino views are never going to return more than a 1000 lines of info, no matter how many lines are actually in the view.

Assuming that you can't find a friendly admin to raise that limit higher or remove it altogether, how do you get around these restrictions?  Here's how I approached it.

It's fairly obvious that I was going to have to make the Ajax call numerous times until I had all the data that I needed.  You can use the start parameter of the Ajax URL to instruct Domino return a chunk of data starting from that particular row number.  By advancing this parameter by a set number each time, you can make your way down the view, returning its data in chunks until you've got it all.  But how to calculate how many times to call make the Ajax call?

Well, Domino does give you the @toplevelentries property the data that's returned by an Ajax call.  At least, it does if you're using JSON, via the Outputformat=json parameter.  (Sorry, the whole XML thing passed me by!)  The @toplevelentries property tells you how many rows there really are in the view, and using that, you can calculate how many Ajax calls you'll need.  I didn't like this approach, however.  Apart from anything else, it's not particularly transferable: other server systems may not have any equivalent of @toplevelentries for you to hook into.


Recursion

So, I went for a recursive approach.  Recursion is a programming technique where a function calls itself, and keeps calling itself until a particular set of circumstances has been fulfilled.  To be honest, it's something that I've taken great pains to avoid in the past, because I didn't really understand it!  Well, maybe I do now.  What I want to do is keep calling my Ajax lookup function so that each iteration of the function returns a different chunk of the view's data.  We'll start at the top, and work our way down the view in chunks until we hit the bottom and there's no more data to return.

Here was my first attempt.  (Note I'm using jQuery's $.ajax() function to make the actual Ajax call.  You'll also see that I pass parameters into functions as a single object, rather than a list of individual variables.  Although that's probably overkill in the examples below, it becomes a life saver when you start passing in more than three or four parameters, and keep messing up their order, like me!)



// the actual function that makes the call to Ajax
function ajaxLookup(parametersObj) {
    var serviceURL = '/path/db.nsf/viewName/?ReadViewEntries&Outputformat=json&count=' + parametersObj.countSize + '&start=' + parametersObj.startPos + ';

    $.ajax({
            url: serviceURL,
            success: function(returnObj) {
                    callBack(returnObj, parametersObj);  // returnObj is the JSON object returned by the Ajax call
            },
            error: function(request,error) {
                // do error stuff, if required
        },
        complete: function() {
            // do complete stuff, if required
        }
    });

}

// The callback function that is called by a successful ajaxLookup() call.
function callBack(returnObj, parametersObj) {
    var resultsCount =  $.isEmptyObject(returnObj) ? 0 : returnObj.viewentry.length;                        
    if(resultsCount > 0) {
            doStuffWithThisChunkOfData(returnObj); // returnObj is the JSON object returned by the Ajax call

            // This is the recursive bit.  We keep calling ajaxLookup until it returns no more data
            ajaxLookup({"startPos": parametersObj.startPos + parametersObj.countSize, "countSize": parametersObj.countSize});
    }
    else {
            doStuffNowAllAjaxIsComplete();
    }
}


The ajaxLookup() function makes the Ajax call that gets the data from the view.  If successful - meaning that it's returned something, even if it's an empty object - it will call its callback function.. errmm... callBack()!

The callBack() function checks to see if there's any rows in the JSON object returned by the Ajax call, and if there is, it does two things.  First, it calls a function to process the current chunk of data that we've returned.  This is the doStuffWithThisChunkOfData() function, which I've not bothered to define here because what it actually does isn't important at this point.  The second thing it does, is call the ajaxLookup() function again.  This is the recursive bit, where a function is calling itself.  (Even though we have two functions here, we can think of ajaxLookup() and callBack() as two halves of the same function.)  The important thing it does when calling ajaxLookup() again, is that it moves the startPos parameter on by the number of lines specified in the countSize parameter, so the new Ajax call will start where the previous Ajax call left off in the view.

The callBack() function will keep calling ajaxLookup() until the latter fails to return any rows, in which the resultsCount variable will evaluate to zero.  In that case, callBack() will call doStuffNowAllAjaxIsComplete() to do whatever - again, not defined here because it's not important - and will then exit.  It will make no further calls to ajaxLookup() because we're at the end of the view now, so we're done.

Now we just need to make the first call to ajaxLookup():
ajaxLookup({"startPos": 1, "countSize": 300})


Note that the startPos parameter is set to 1 for the first run because, obviously, we want to start at the first row in the view.  We're going to be retrieving the data 300 lines at a time.


Ajax and Global Variables - No!

So far, so good.  What of the doStuffWithThisChunkOfData() function though?  Well, in my particular case, this function was updating the rows of an HTML table with the data returned by the Ajax call.  This turned to be not a particularly clever thing to do.  Basically, it crawled on IE, espcially IE6 and IE7. Because of the asynchronous nature of Ajax, it meant that one JavaScript function was updating the HTML table while another JavaScript function was pulling more data with which to populate it.  In fact, I might even have had mulitple iterations of the doStuffWithThisChunkOfData() function trying to update my HTML table at the same time!  (I'm not entirely sure just how asynchronous all this stuff really is.)

So, it occurred to me that it might be better to pull in all the Ajax data to variable somewhere, and then run the doStuffWithThisChunkOfData() only at the end, using all the data that we got from Ajax.  Easy enough, I thought.  I just push the returned returnObj data objects into a global array variable, and then I can have doStuffWithThisChunkOfData() read this global variable when the Ajax calls are finished.

Only, it didn't work.  Whenever I tried to read the global variable at the end, it was always empty.  This appears to be a problem with Ajax and globals, and I think I can see why this is.  Basically, the Ajax functions, being asynchronous, can't really ever know what the state of a global variable is at any given time.  Perhaps a better way to say this is that the Ajax functions can't know what you, the programmer, is expecting the global variable to be at any given time, if that makes any sense!  You can't guarantee what order Ajax calls will run in.  Nor can your Ajax function program know what other Ajax functions have updated a global variable before it.  So, my guess is, they simply ignore the whole thing!  If anybody has a better explanation, I'll be glad to hear it!

Anyway, here's how I got around that problem.  Rather than trying (and failing) to store my results in a global, I store them in another parameter of the ajaxLookup function.  Here's my revised functions, using this technique:

// the actual function that makes the call to Ajax
function ajaxLookup(parametersObj) {
    var serviceURL = '/path/db.nsf/viewName/?ReadViewEntries&Outputformat=json&count=' + parametersObj.countSize + '&start=' + parametersObj.startPos + ';

    $.ajax({
            url: serviceURL,
            success: function(returnObj) {
                    callBack(returnObj, parametersObj);  // returnObj is the JSON object returned by the Ajax call
            },
            error: function(request,error) {
                // do error stuff, if required
        },
        complete: function() {
            // do complete stuff, if required
        }
    });

}

// The callback function that is called by a successful ajaxLookup() call.
function callBack(returnObj, parametersObj) {
    var resultsCount =  $.isEmptyObject(returnObj) ? 0 : returnObj.viewentry.length;                        
    if(resultsCount > 0) {
            parametersObj.viewEntryObjArray.push(returnObj);


            // This is the recursive bit.  We keep calling ajaxLookup until it returns no more data
            ajaxLookup({"startPos": parametersObj.startPos + parametersObj.countSize, "countSize": parametersObj.countSize});
    }
    else {
            doStuffNowAllAjaxIsComplete(parametersObj.viewEntryObjArray);
    }
}



The only change is that the doStuffWithThisChunkOfData() function is now removed from the callBack() function, and its place is line that updates a new parameter called viewEntryObjArray.  This array variable holds an array of all the JSON view data objects that are returned by the Ajax lookups.  This array is passed as a parameter doStuffNowAllAjaxIsComplete() function, when all of the Ajax lookups are done.

All that remains now, is to ensure that the viewEntryObjArray is included in our first call to ajaxLookup() and is instantiated as an empty array, like so:
ajaxLookup({"startPos": 1, "countSize": 300, "viewEntryObjArray":, []})



Update October 2015

While this approach works, it does have a serious drawback: it's slow.  That's because with my recursion method, the Ajax calls had to run in sequence; the second Ajax call was triggered by the .success() callback for first Ajax call.  Then the third Ajax call would be triggered by the .success() callback for the second Ajax call.

Please see my post Parallel Ajax Calls, where I explain a much better way of doing this by using promises.















Comments

1Mark Leusink  06/15/2011 2:32:42 AM  Getting around Domino’s Ajax lookup restrictions

Hi,

Nice explanation!

You're right that the server returns the number of entries in a view in the XML/ JSON representation of a view. The method you're describing is therefore correct, but if you're a lazy developer, you could also add &count=99999 (or some other large number of which you're pretty sure that the number of documents in your view won't exceed).

If I'm not mistaken the &count parameter overrides the limit set in the server document.

Mark

2Mike Brown  06/15/2011 7:20:04 PM  Getting around Domino’s Ajax lookup restrictions

@Mark,

I tried setting the &count parameter to a huge number. It made no difference for me. I still only got only 1000 lines returned, as per the server limit.

That's on Domino server 7.0.2 though. Maybe later releases differ in this regard.

Cheers,

- Mike

About