Post: Lift and Data Driven Comet

Daniel Spiewak is one a rant again about Lift's Comet model being the wrong one and being the devil and such. He's had this rant in the past and I've tried to talk him through the positive of what he does want. I've been unable to understand why his goal isn't served by what's already in Lift's Comet support. What the discussion usually boils down to is "in most of the Lift Comet examples, the server manipulates client DOM, but the server should know/care about client state." I agree with that, but the fact that example code manipulates client DOM doesn't mean that this is the only way to work with Lift's Comet. Further, complaining about the examples rather than discussing positive use cases doesn't help me or the Lift team better understand how to build better data-driven (rather than DOM driven) examples and abstractions. With that off my chest, let me walk through Lift's Comet and how one can build data driven Comet apps based on available examples as well as some conceptual discussions about what I've done in non-open source projects.

Let's take a quick tour of Comet and what Lift's Comet implementation offers. HTTP is a client-owned protocol. The client in HTTP makes a request on a server. Based on that request, the server returns a response. After the server has returned the response, the server has no further mechanism for changing state on the client.

It's valuable to have applications that allow server state to be "pushed" to the client. These applications include stock quote applications, sports betting, multi-user games, etc. Comet, or server push, is a mechanism for the browser to keep an HTTP connection open to the server and when server state changes in a way that may impact the client, the server sends data over the connection to the client and the client processes the chunk of data.

There are a fair number of "elegant engineering solutions" to keeping the HTTP connection open so that the server can "push" new information to the client.

Lift's current Comet implementation uses long polling. In the future, Lift will also use WebSockets, but the WebSocket API and implementation isn't "baked" yet, and Lift's current implementation works well for browsers from IE6 on. Lift's long polling implementation opens a single HTTP connection from the browser to the server if one or more Comet components are detected on a HTML page. The request contains a list of the components and the "version" for each of the components. If the version number of any of the components on the server is "newer" than the version on the client, the deltas are sent as the response, the connect is closed, the browser applies the deltas and then re-opens the HTTP connection (with HTTP 1.1 keep-alive this does not imply reopening a TCP/IP connection.) If the version of the component changes while the HTTP "poll" is open, the deltas are sent down as a response and the connection is closed. If no state changes after 110 seconds (this is a tunable parameter), a noop is sent to the browser and the browser re-opens the HTTP connection. The 110 second timeout is to insure that Lift's comet implementation is proxy friendly.

Lift's comet implementation is hidden from the developer. The developer need only focus on two things:

Generating "initial state" that is sent from the server to the browser during a full page load that contains a Comet component
When state changes on the server (state can only change for the CometActor is it receives an asynchronous message in its Actor mailbox), the server generates JavaScript that causes the browser to take appropriate action based on the server state change. Not all state changes on the server necessarily result in state changes on the client. Further, this does not imply that the server must keep track of or keep a mirror of client state.

The mechanism for achieving #1 above is implementing the render and fixedRender methods in a CometActor. The former is required and the latter is optional. The combination of the return values of these two methods define initial state for the CometActor during a full page load. Here's the documentation for render:

/**
   * It's the main method to override, to define what is rendered by the CometActor
   *
   * There are implicit conversions for a bunch of stuff to
   * RenderOut (including NodeSeq). Thus, if you don't declare the return
   * turn to be something other than RenderOut and return something that's
   * coersable into RenderOut, the compiler "does the right thing"(tm) for you.
   * <br/>
   * There are implicit conversions for NodeSeq, so you can return a pile of
   * XML right here. There's an implicit conversion for NodeSeq => NodeSeq,
   * so you can return a function (e.g., a CssBindFunc) that will convert
   * the defaultHtml to the correct output. There's an implicit conversion
   * from JsCmd, so you can return a pile of JavaScript that'll be shipped
   * to the browser.
   */
def render: RenderOut

You can define your render method to be DOM oriented. Here's the super duper simple code for a Comet-based Chat app where the Server defines the DOM:

/**
 * The comet chat component
 */
class Chat extends CometActor with CometListener {
  private var msgs: Vector[String] = Vector() // private state
 
  // register this component
   def registerWith = ChatServer
  
  // listen for messages
  override def lowPriority = {
    case v: Vector[String] => msgs = v; reRender()
  }
 
  // render the component
  def render = "li *" #> msgs
}

This example demonstrates how easy it is to write a Lift Comet component. It is not and full, industrial strength implementation of a Chat component. It is an example that strips the code down to its bear essence. The essence is "set up default state, when change comes in, change complete state and redraw the entire screen real estate associated with the component to reflect the new state."

However, we could just as easily have done an entirely data driven application. This is taken from the Frothy example that integrates Lift and Cappuccino:

class Clock extends CometActor {   override def localSetup() {     super.localSetup()     ActorPing.schedule(this, 'Ping , 3 seconds)   }  override def highPriority = {     case 'Ping =>      partialUpdate(currentTime)       ActorPing.schedule(this, 'Ping , 3 seconds)   }  def currentTime: JsCmd = JsRaw("clockCallback("+(""+now).encJs+");")   def render = {     val str: String = """var f = function() {try {"""+(currentTime.toJsCmd)+"""} catch (e) {setTimeout(f, 50);}}; f();"""     OnLoad(JsRaw(str))   }}

In this example, the server side knows nothing about the client state. The server side sends events to the client and those events call a client side function with event data. The server-side knows nothing about the client state, browser DOM or anything else. It's up to the client side to implement the the function.

More generically, the above could be written as:

class KnowNothing extends CometActor {
def render = Noop // we have no comment on initial component state

override def lowPriority = {
case msg: MyMessage => partialUpdate(callEventHandler(jsonSerialize(msg)))
}

def callEventHandler(in: JObject): JsCmd = // what function do we call on the client side with a JSON blob?
def jsonSerialize(in: MyMessage): JObject = // serialize a case class
}

And voila... we've got a comet component that simply passes events from the server to the client. The Comet component has no knowledge or information about client state. This is in effect an "Actor proxy" between the server and the client. What you get is an implementation that passes messages from server to client.

"What benefit do I get by using the event passing mechanism?" you may ask. Lift's Comet implementation multiplexes all the Comet components on a page through a single HTTP connection. So, you could have 20 or 50 or more event handlers in the browser all handling different parts of the screen and Lift will handle all those components through a single connection. If there are 10 updates between HTTP requests, all those updates will go down the same response. If the client is disconnected for a period of time (a mobile user is on a train going through a tunnel where there's no Internet connectivity), the client will receive all the deltas in order when the client is once again reconnected (note there are certain limits that are tunable as to the amount of time and aggregate number per comet component of deltas that are retained.)

So, we've shown both ends of the "server knows everything about client state" and "server knows nothing about client state" spectrum.

I've written substantial Lift comet code for Innovation Games and Much4.

In Much4, the CometActor in the server kept the authoritative copy of the data. When updates can in the data deltas were calculated and JavaScript that updated the client's representation of the data to match the server's version of the data was sent to the client via partialUpdate along with a command to "redraw" the client. The client redrawing was accomplished entirely in JavaScript on the client... the updated DOM was generated entirely by client-side JavaScript. Thus, the state that the server knew about the client was the state of the data on the client. The CometActor on the server knew nothing about the DOM state on the client.

In the Buy a Feature game in Innovation Games, the CometActor on the server kept a copy of the DOM of the gameboard. The server was authoritative about the client's data state and DOM state. In fact, the client kept no state other than the DOM. Changes to gameboard state were calculated by generating deltas between state and time T and state at time T+1 and those changes were turned into DOM manipulation JavaScript that was aggregated and send to the client via partialUpdate. Because of the rules of gameplay, there was never a case where the places that the user was changing on the gameboard would be updated without the server knowing the deltas.

In the Prune the Product Tree game, the gameplay took place on an SVG canvas for "modern" browsers and some wacky something something in IE6. The server never had any knowledge about the way that the gameboard was being displayed. In this instance, we kept with the gameboard delta mechanism, but the deltaing generated two pieces of JavaScript for each delta between gameboard at T and gameboard at T+1. One of those deltas was the JavaScript necessary to update the browser-side data structure at that point in the node and the second was a call into the node-specific "update the visual display" routine (for example moveItem(itemID, oldPosX, oldPosY, newPosX, newPosY)). In this case, the deltaing routine had to have knowledge of the API structure on the client. This was for expediency sake... it was easier to write Scala code that knew about the client APIs than write JavaScript that could take the delta commands and also forward them to the redrawing commands. In this case, the server knew what client data state should be and generated commands to send the data down. Note, too, that the client was "smart enough" not to cause data changes if (1) the the browser already contained the new state (this allowed for a single client to update its state without having to do a server round-trip) and (2) not updating items that the user was "touching" until the user finished touching the item.

Lift's Comet support has been able to work effectively with the above wide range of scenarios. The place where Lift's Comet support will not work as well is if you try to mix the metaphors of DOM state and data state when there's code in the browser that will modify the DOM state without telling the server. In the case that you only want the server to be partially knowledgeable about DOM state, then you're opening a huge can of worms. Yes, Lift lets you open that can of worms, but just because a web framework lets you make bad design decisions that's not the web framework's fault.

Yes, there are plenty of API improvements we could make for data-driven Lift Comet. I'd love to see a data diffing mechanism that'd take two JObjects and come out with JavaScript representing the commands needed to update the old to the new as well as generate the delta events. We hand-wrote that mechanism in Prune the Product Tree... but having it automagic would be great. Yep, it'd be better if there was more documentation and example code for doing data-driven Comet. I'd love to get beyond the research phase on my Lift Comet to KnockoutJS integration as well as finishing the work I started doing Lift to SproutCore integration. But a lot of that kind of stuff is community and client demand driven.

So, I thoroughly and completely disagree with the assertion that Lift's comet support requires that you keep state or DOM state in the Comet component. Lift's Comet support gives the developer a wide variety of ways to send a wide variety of data from the server to the client based on events on the server.

Lift and Data Driven Comet July 7, 2011