Also fun news:
I’ve added my first round of SimpleTest unit tests for the Listotron server side, and I’ve also added some jqUnit test for the client side.
The cool thing about the jqUnit tests - they run on live Listotron code. So all the client side tests not only test the JavaScript, but also end up testing that the entire app does what we think it does. They end up doubling as functional and regression tests.
awesome!
I just spent quite a few hours this weekend making some changes to the Listotron code-base.
The browser polls the server many many times a second, and the server sends back whatever’s changed in that small amount of time. This means that if an AJAX call is taking about 100ms, then the change 1 user makes to the list will show up on all other collaborators screens after no more than 200ms. Pretty sweet!
So I had all of that working a few weeks ago - multi-user list collaboration and all - but I found that sometimes collaborators lists got out of sync. Not all of the edits seemed to make their way across all collaborators.
The problem: I was doing all of the data manipulation in JavaScript in the browser, and only sending the changes to the server. So when a user indented a line in the list, instead of sending an “indent” command to the server, I would indent locally and send the updated row’s properties to the server.
The problem was further compounded by the fact that 1 edit (that should have been atomic) was split into multiple AJAX calls (due to how BAJAX bundles sets of ajax requests).
The problem manifests itself when 2 people edit the exact same row at the exact same time (not that uncommon in my 6 row demo list). Each user’s JavaScript would update the row differently and send their own results to the server. Each time, the server would accept the changes, modify the server’s definition of the list, and send back an “ok!” So both user’s got a successful response but saw totally different lists.
The solution is to do all list manipulation on the server. so instead of sending a row’s updated properties (like who the parent row is and what row is before it), i just send a specific and atomic command, like “indent row X.”
The server then indents the row if it can, and returns a list of all modified rows to the client. Then the client updates the view. This means that currently the view takes about 150-200ms for a keypress to manifest itself in the view.
This delay will fixed in later versions, and is surprisingly unnoticeable anyways.
I’m very happy to show off the first prototype of Listotron below - complete with multi-user edit capability! The video below shows me editing in one browser and edits showing up live in a 2nd window. You’ll have to trust me when I say that it works with 2 users editing in their own windows. 
Imagine for a moment that you’re building a web application based nearly 100% percent on AJAX. And imagine that you’ll likely need to send bursts of multiple AJAX requests to your server, and you also need all of these requests to be processed in order when they hit your server. You want a near continuous AJAX connection - every action, every edit, every keystroke the user makes - all of these need to be sent to your server for processing. You are building Listotron.
Since we’re designing Listotron to be a 100% AJAX based application, every edit the user makes will need to be AJAX’d to the server. What’s more, we want users to be able collaboratively edit the same list at the same time - Google Spreadsheets style. This means that every edit, every tab, every indent, everything needs to be AJAX’d quickly and efficiently to our server so we can update any collaborators that might also be working on the list. And as I mentioned, all of the edits need to happen in order, since most edits to the list will depend on previous edits to the list.
My goals for speed are to make the real-time editing even faster than Google’s collaborative spreadsheet editing. A large part of the perceived speed of collaboration is the frequency of AJAX calls (the more calls per second the faster the app can update collaborators) - so I’ll need both an incredibly streamlined server-side and an incredibly optimized AJAX front-end in JavaScript. This article will talk about the JavaScript strategy.
The problem with relying on plain ole’ asynchronous AJAX is that it’s, well, asynchronous. We’re not guaranteed that all of our AJAX calls will be received by the sever in the correct order, and we’re certainly not guaranteed that we’ll receive the responses in the correct order.
Imagine sending off 4+ edits to the sever, only to find out in the last response that the 2nd request failed. What if edits 3 and 4 relied on edit 2? How do I roll back the UI? How do I maintain data integrity on the server? on the client? It’d be hell to build, hell to debug, and hell to maintain. No thanks!
Since I need to send all my AJAX requests to the server in order, why not just use synchronous ajax calls? Problem solved, right? Unfortunately, it’s not that simple - from the YUI blog:
XMLHttpRequest can operate synchronously or asynchronously. Many people prefer to use it synchronously. When used this way, the JavaScript engine is blocked until the interaction with the server is complete… Temporal complexity is abstracted away, leaving a very familiar and comfortable programming pattern.
So far so good, but here’s the zinger (emphasis added)…
Because the JavaScript engine is blocked until the request completes, the browser will be frozen. The user cannot cancel the request, cannot click away, cannot go to another tab. This is extremely bad behavior.
That’s very bad news indeed. Synchronous AJAX ends up looking something like this:
Clearly this is not the way to go. It’s simply not acceptable to freeze the entire browser - (even in Chrome’s case: the entire tab) - for every single tiny edit. A different approach needs to be made.
The solution we’ll be using with Listotron is to manually enforce synchronization into what would otherwise be asynchronous AJAX calls. The idea is this: after an AJAX request is made, queue up any other requests that need to be made until a response comes back - then send a single bundled AJAX request that contains all of the requests currently in the queue. The server decodes this bundled AJAX call, processes each request, and sends back a bundled response. All this ends up looking something like:
Since we need to maintain the call-order of our AJAX calls, we can queue them up in-order and then send off multiple requests to our server in one go. The benefits are: (1) fewer http connections hitting the server, and (2) the order of the AJAX calls is maintained, (3) and UI is still very responsive. And, since the server sees multiple edits at the same time, further optimization can be done that would otherwise be impossible. For instance, the server receives 2 commands in a bundle: an indent command followed by an outdent command. The net result to the list is to do nothing!
This optimization comes at a price, of course, and it’s that the server needs some custom code to handle these bundled AJAX requests, so this solution isn’t as easy as dropping in a new JavaScript framework or function.
I’ve cross-posted this post on my personal website, and set up a basic demo and tutorial that bundles up asynchronous AJAX calls to maintain their order. I’ve implemented everything as a jQuery plugin with a lightweight PHP/MySQL server-side. Check it out with full source code and download links here.
JavaScript has allowed us to create dynamic interfaces, adapting to a user and providing context in many ways. In past endeavors, we have developed the entire front-end layer using JavaScript: a vast object-oriented structure to deliver HTML on the fly.
When we started using JavaScript as a serious web-interface tool, JavaScript libraries were young and most too cumbersome to implement. We decided on the cross-browser library for its structure and ease of modular integration. However, in the past two years, JavaScript libraries have matured, each taking a separate approach in programming methodologies. jQuery is the one we enjoy using the most.
Rather than focusing on why we did not choose other libraries, we will keep things as short as possible by providing a few reasons we chose jQuery.
It’s true. jQuery has left out many utility-style functions Prototype has included, and rather focuses on functions that require some cross-browser love. jQuery is easily extensible if we find we need to add functionality to the library.
In addition to the jQuery library being small in size, it allows us to write shorter, more readable code. This provides a significant decrease in development time as well as a decrease in page load time.
A friend of mine once told me “It took me less time to learn jQuery and implement a jQuery-based solution than to implement a Prototype-based solution. And I already knew Prototype!”
This may be hyperbole, but it’s not far off. I’m a designer first, and while I’m trained in the art of programming, I like to spend at little time as possible coding. The jQuery team has done a wonderful job with the documentation, and the library is very well structured.
As a web designer with a full understanding of the DOM, jQuery is a joy to use. Traversing, manipulation and animation has never been so easy.
Indeed, I have been using jQuery enough that coding without it is arduous.
We plan to share a great deal of code with you. Therefore, we plan on building our features as jQuery plugins and including them as such in our product. There is hardly any overhead in turning a bit of functionality into a jQuery plug-in, making it the perfect way to share our code with you.
We are anxious to start preparing prototypes and examples for you using jQuery! Adam will post some fantastic jQuery-centric JavaScript tutorials very soon.
You can check out his extensive front-end Model-View-Controller JavaScript development pattern here. We will be applying this methodology to Listotron, so expect more specific examples and tutorials in the near future.
Today marks the day we begin development on Listotron, an online list maker. No doubt you have many questions about the functionality and features of the product itself; I assure you we will talk more about the product in subsequent blog posts.
However, equally– perhaps more– important is the manner in which we are building the product. We have three rules during the development of Listotron, and we plan to vehemently follow them in every decision we make.
We plan on including you guys every step of the way. We want to share as much as we can: design decisions, interface thoughts, programming methodologies– right down to sharing some code and design files. Expect to see a great deal of tutorials (both in design and development) on this blog.
Creating a product like Listotron requires expanding our horizons in every way possible, and we want to share everything we learn with you.
We are creating this software and maintaining this blog for fun. Simply put, we enjoy building things that people use, we enjoy learning new things, and we doubly enjoy sharing information. Learning and sharing knowledge is fun.
The goal is to let the concept of “fun” permeate our product and our blog. If you send us an email, ask us a question, or post on our blog, expect a personal reply. Talking to people is fun.
There’s simply no point in doing what we do unless it’s the best. We believe we have the talent to create the best online list maker in the world, and we’re going to see it through.
“Best” is subjective, though, obviously; so we’ll be sharing with you what we mean by “best” in future blog posts.
We don’t want you to just use the product we’re building; we hope that you enjoy reading our posts and learn something from our experiences and tutorials.
Be sure and follow the RSS feed, and follow us on Twitter to stay informed on all the goings on.
Thanks for embarking with us, we hope you have as much fun as we do!
- Adam & Buck