Syncing
As soon as you start creating offline applications syncing becomes an issue.
However, if you are not careful, trying to figure out syncing can become an
endless black hole that causes your project to spiral and fail. Syncing can
either be relatively easy or infinitely complex, depending on how you approach
the issue. In order to help, Dojo Offline includes both a syncing framework,
named Dojo Sync, as well as a set of syncing guidelines to help you avoid sync
land mines. We've thought through many of the hard issues and created a path
for you to navigate through the sync process.
Lets begin with the guidelines.
Sync Guidelines
First, user's view syncing as a bug, not a feature. They don't want to toy with sync interfaces -- they just want syncing to happen automatically.
Along these lines, user's don't understand merge interfaces. Developers are pretty comfortable with merging files, such as using Subversion or CVS. Users, however, don't care and don't understand merge interfaces. If you are popping up a complicated diff and merge UI as you do syncing, then you have failed.
Therefore, Dojo Sync recommends that rather
than showing merge interfaces when syncing, the client- or server-side should
automatically make a best guess, without UI intervention by the user.
If a document was modified locally while offline, and also modified on the
server by someone else, for example, they should be automatically merged
without user intervention. If there is a conflict, you should make a best
decision on which to keep based on your application. You could choose to keep
the newest one, for example, and could also make a backup in the document's
history list of the other change in case the user needs to retrieve something
from it. Do merging automatically, and decide
the best way your application can support this.
With all this said, users still need basic UI feedback that syncing is occurring and where in the sync process they are. Syncing should not be completely invisible; just as a browser has a basic network throbber to show you network activity is happening without showing the entire intricacy of the process, syncing is the same way. Users want to know that syncing is occurring; see from a high-level where the syncing is at (uploading, downloading, etc.); whether syncing succeeded or failed; and want the option of finding out why syncing failed or any automatic merges that might have happened. Even though users don't want to do merging, they some times want the option to look at a sync log to see what was merged. This should be done in a way that doesn't clutter the main sync UI (i.e. it should be a sync log link, for example, that would display a pop up window with a sync log of what was merged and what conflicted, and how the server did automatic merging).
Finally, user's don't want to have to manually
always kick off syncing -- they just want the application to work correctly
offline and automatically start syncing at correct times. In light of
this, an application should always sync when it first loads and the network is
present, bringing down a subset of data to make available offline. Users
rarely know when they will be offline, so if the application simply always
syncs then there should be some set of data available offline when the user
loses the network. Likewise, when the network reappears, the application
should automatically sync any local changes back to the server, with the user
not needing to kick this off.
Dojo Sync Overview
Let's look at working with Dojo Sync and how it implements the guidelines laid out above.
The first thing to know about Dojo Sync is that it works in conjunction with the default Dojo Offline UI widget you embedded into your page earlier, doing much of the hard work of communicating the sync process to the user in a lightweight way. When you embed the offline widget you get a sync user-interface for free.
Second, syncing happens automatically, without the user having to initiate it. Dojo Sync automatically kicks off syncing when the web application first loads and we are on the network, and also if we are offline and we detect that network has come back online.
When syncing starts, three things happen in the following order:
-
Dojo Sync downloads and caches our web user-interface -- any files that
dojox.off.files.slurp()
detected are now brought offline - We upload any data that was changed or created while offline
- We download data to make available offline
As you will see in a bit, Dojo Sync does much of the work for you, and provides some specific places you hook into to help it do its job.
Because Dojo Sync hooks into the offline widget, the user-interface for these three steps is completely automated. Let's quickly see what this default sync UI looks like from an end-user's perspective, since this can help you as a programmer understand how to tie into Dojo Sync.
First, Dojo Sync downloads our web user-interface to make it available offline:
Next, our web application uploads any changes that were made while offline:
Finally, we download data to make it available offline:
If everything is successful, the user is notified:
If there was an error, the user is also notified:
The user can find more details about the error by pressing the details link; this will cause a pop up window to appear with details on why the sync error occurred:
>Now that you have an overview of Dojo Sync and what its user-interface looks like, let's see how your application code actually hooks in.
Hooking Into Dojo Sync
As stated before, Dojo Sync does much of the heavy lifting for you around user-interface and syncing. Your application only has to hook into two parts of Dojo Sync to help out: downloading data to make available when offline, and uploading any changes that were done while away from the network. Downloading is the simplest.
Sync Downloading
During the syncing process, Dojo Sync fires sync events that you can subscribe to:
dojo.connect(dojox.off.sync, "onSync", this, function(type){});
There are many different onSync
events, given by the type
variable, but you only need to be concerned with a few unless you are doing something unusual (you can see a full list of the onSync
events here).
When the sync process reaches the downloading stage, onSync
will fire with type
equal to the string download
. You should listen for the download
event and proceed to download your data. In the example code below, when the sync framework wants us to download, we make a network call to some web service that fetches new documents, for example:
dojo.connect(dojox.off.sync, "onSync", this, function(type){ if(type == "download"){ // download new documents dojo.xhrGet({ url: "/documents/download", handleAs: "javascript", load: function(data){ dojox.storage.put("documents", data); dojox.off.sync.finishedDownloading(); }, error: function(err){ err = err.message||err; var message = "Unable to download our documents from server: " + err; dojox.off.sync.finishedDownloading(false, message); } }); } });
In the example code above, we first subscribe to onSync
events. When a sync event fires, if it is the download
event, we then proceed to make an XMLHttpRequest call to some web service that downloads new documents, located at "/documents/download"
, which returns JavaScript JSON as its results. We use Dojo's easy to use dojo.xhrGet()
method to do so. xhrGet()
can take an error
handler and a load
handler, as well as allowing us to specify how the return results should be handled. In the example above, our handleAs: "javascript"
parameter says that the results will just be handed to us as a JavaScript object.
Let's look at the load()
method first. If the web service returned correctly, we simply take the data returned, which would be our list of documents, and store it right into Dojo Storage with a put()
call; we could also have used Dojo SQL here and iterated over the results to write into a relational store if we wanted.
The next line is important:
dojox.off.sync.finishedDownloading()
You must call this method when you are done downloading. Since downloading data tends to be an asychronous process that involves talking to the network, Dojo Sync has no way to know when it can continue the syncing process after downloading is complete. When you call dojox.off.sync.finishedDownloading()
from your network callback, it tells Dojo Sync to continue. If you do not, Dojo Sync will get stuck at the downloading phase and your user-interface will just show "Downloading new data..." forever.
Let's look at the error
callback now. If an error occurs during downloading, you must also call dojox.off.sync.finishedDownloading()
, but with two arguments. The first should be false
to indicate that downloading was unsuccessful; the second argument should be the reason that downloading failed, which will be reported in the sync log at the end of the sync session.
Sync Finished
We will see how to handle sync uploading in the next section, but first there is one more sync event that you will find useful in your code. When syncing is finished a sync event is fired with type "finished"
:
dojo.connect(dojox.off.sync, "onSync", this, function(type){ if(type == "download"){ // download data // ... }else if(type == "finished"){ // refresh our UI when we are finished syncing var documents = dojox.storage.get("documents"); dojo.forEach(documents, function(i){ // update the UI list of documents with this new document // ... }); } });
In the example code above, when the finished
event is fired, we load all of our new documents from Dojo Storage and then use each one to update our user-interface. The finished
event can be useful for updating your UI once syncing is done.
Note that this event is fired whether an error ocurred or not during syncing; check dojox.off.sync.successful
and dojox.off.sync.error
for sync details. These are boolean properties that will tell you whether an error occurred or not.
Sync Uploading
Sync uploading is the trickiest part of syncing in general. Before we can tackle that, let's take a quick look at how Dojo Sync recommends you record user actions when they are done offline.
Offline User Actions
What do you do when a user is offline, but starts doing user actions that would normally cause a network call or which updates data?
Dojo Sync adopts what is known as a transactional model to solve this problem. When a user works offline, creating a new business contact or modifying an existing one, for example, we create a record of this action and update our local data. These actions are added to an action log.
For example, imagine that a user is working offline with a web application that has sales contacts in it. The user first modifies the contact record for "Brad Neuberg," changing the phone number. Since we are not on the network we can't write this change to the server; instead, we create an action object to indicate that "Brad Neuberg" has been updated, and store it in an action log. We also update our local offline data with the new phone number. The user next creates a new contact, named "John Doe"; again, we create an action object to represent this offline action, and store the new contact.
When we go back online, all we have to do is replay the action log, which will simply execute each of the actions while online in the exact same order they were performed when offline. When we replay actions while online, all we are doing is essentially "simulating" that the user just did them right then, but very fast. The great thing is you can call existing web services that you might have to handle these actions once online.
In the example above, when the network first appears and we start syncing and reach the Sync Upload phase, we first replay the user action of updating the "Brad Neuberg" contact. Replaying this action executes doing what we normally do while online, which is to call some network service to update some contact. In our code example above, we might do an XMLHttpRequest call to POST the updated contact data to "/contacts/Neuberg/Brad"
. We then "replay" the user action to create the "John Doe" contact, which might do an HTTP POST to "/contacts/Doe/John"
with the new contact.
As recommended in the Sync Guidelines section above, these web-services should do automatic merge and conflict detection so that the user is not inundated with complicated merge UIs. If you insist on having a merge UI, see the Advanced Syncing section.
Let's see what all of this looks like from a code level now.
If the user is offline and does some action, we create action objects and add them to our action log:
// some new client added by sales person var client = {lastName: “Doe”, firstName: "John", phoneNumber: “555-222-1212”); // create an Action object to represent this action -- // these can have anything you want in them -- as you will // see later, they should have enough data to help you replay them // when we go back online var action = {name: "new contact", data: client}; // add this to our offline action log dojox.off.sync.actions.add(action); // persist the contact into offline storage dojox.storage.put("John Doe", client);
In the example code above, we first have an object that represents the new client that was created, John Doe. Next, we create an action
object to represent this offline action; the action
object can have any values you want -- it must simply have enough data for you to be able to replay this action when we go back online. In general, you will have an action name
, such as "new contact"
, and some data
that goes along with this action, which in this case is the newly created client
.
Next, we add this action to our offline action log. The offline action log is automatically persisted, and records any actions that were done while offline, in the order in which they were performed by the user.
Finally, we persist this new or updated action into local offline storage; even though we can't communicate on the network, we want to keep our local data up to date with any offline changes that were made by the user.
Replaying
When the network reappears, Dojo Sync kicks off and starts automatic syncing. After refreshing the offline UI, Dojo Sync then attempts to upload any changes that were made while offline. Basically, Dojo Sync tries to replay the action log you built up while the user was offline, executing each one one at a time.
However, Dojo Sync doesn't know how to replay actions -- thats the job of your application. You register yourself to know when replaying has occurred, and Dojo Sync calls your replay function for each action, simply handing it the action object you created earlier while offline. You can now use this action object to execute the action, but this time while online.
Let's build this code segment up one piece at a time. First, we register to know when replaying has occurred:
dojo.connect(dojox.off.sync.actions, "onReplay", this, function(action, actionLog){ } );
The function we give will be called over and over for each action that was added to the offline log. The offline log is just a persistent array, or list of actions in the order they were done by the user and added by you to the action log. This function is given each of the actions, one at a time, as the first argument action
, and a reference to the action log itself, actionLog
. Your code should then look at the action and try to replay it:
dojo.connect(dojox.off.sync.actions, "onReplay", this, function(action, actionLog){ if(action.name == "new contact"){ } } );
In this case we just check the action.name
property, which we set earlier when the user actually performed this action. If it is the value "new contact"
, then we try to simply create this new contact:
dojo.connect(dojox.off.sync.actions, "onReplay", this, function(action, actionLog){ if(action.name == "new contact"){ var contact = action.data; // create this new contact on the network dojo.xhrPost({ url: "/contact/" + contact.lastName + "/" + contact.firstName, content: { "content": dojo.toJson(contact) }, error: function(err){ var msg = "Unable to create contact " + contact.firstName + " " + contact.lastName + ": " + err; actionLog.haltReplay(msg); } }, load: function(data){ actionLog.continueReplay(); } }); } } );
There's alot going on in the code block above. First, if we see that the action was to create a new contact, "new contact"
, then we actually shoot off an HTTP POST to create this new contact. We get the contact to create from the data:
var contact = action.data;
And then do the actual HTTP POST using this data. Our URL becomes the first and last name of the contact, or /contact/Doe/John
in this case. Dojo's xhrPost()
method takes the content of the post as an attribute named content
, which we provide. We transform our contact into a JSON string to easily send to the server with the dojo.toJson(contact)
method call.
xhrPost()
calls error()
or load()
based on whether things were successful or not. Notice that we call something called
actionLog.haltReplay(msg)
and actionLog.continueReplay()
. Since most of the things we will do while replaying are asychronous, we must tell Dojo Sync when to continue replaying and whether an error occurred. If the action was replayed successfully, you should call actionLog.continueReplay()
, such as we do above in the load()
method. If an error occurred, you should tell Dojo Sync to halt replaying by calling actionLog.haltReplay(msg)
, providing a message string on what the error was -- this error message will be written into the sync log for users to see if they want.
Remember that these web-services should do automatic merge and conflict detection so that the user is not inundated with complicated merge UIs.
One final property that is useful is dojox.off.sync.actions.isReplaying
. Sometimes you want to share the same network code whether you are online or replaying an offline action, since you don't want to have to rewrite the network call. In this case in your load
or error
callbacks you can see if replaying is occurring, and if so tell the action log to continue replaying or to halt. If you are not replaying, you would just handle the load
or error
as normal.
Speeding Up Syncing: version.js
During syncing, we always refresh the list of offline files. This is somewhat due to limitations in the underlying Google Gears APIs we use (for the technically inclined, we use Google Gear's ResourceStore
instead of it's ManagedResourceStore
). This can slow down syncing considerably, however, since every time we sync we also pull down the offline files.
Dojo Offline has an optional feature to speed this up. Your application can have an optional version.js
file bundled in the same directory as it. During syncing, Dojo Offline will read this file to see if the version of the application has changed. If it has then we go ahead and refresh all the offline files; if not, then we skip this step.
The version.js
file is very simple; it should simply have the version inside of it:
"07-12-2007.9"
This can be a string or number; it does not matter. We simply look to see if the value has changed. We also force an offline file refresh if you are in debug mode (i.e. djConfig.isDebug
is true
), or if you just debugged the prior time the page was loaded.
Simply place this file in the same directory as your main HTML file.
This is all your should need to know to work with Dojo Sync.
- Printer-friendly version
- Login or register to post comments
- Unsubscribe post
Error in this page
// some new client added by sales person
var client = {lastName: “Doe”, firstName: "John", phoneNumber: “555-222-1212”);
Should be read (curly braces):
// some new client added by sales person
var client = {lastName: "Doe", firstName: "John", phoneNumber: "555-222-1212"};