Dev Diary #4 - Moving to Offline-First

Moving on to the fourth in this series of Friday updates, they're inching ever closer to actually being ready to publish on a Friday (it will get there one day!).

While all the updates so far have been focused around visual and interaction design, to explain this week's progress we'll have to dive a level deeper -- to something much less apparent on the surface, but absolutely fundamental for providing a worthy experience for mobile: the move to offline-first. Why & how? Read on to find out.

Over time it's become increasingly obvious to me (both through my own experience and extensive feedback!) that with something as core to one's life as goal and task management, the medium used to manage it should ideally be accessible at any moment.

Whether it's a sudden flash of inspiration on how to move forward that needs to be recorded before it's forgotten, some reference notes that need to be accessed immediately in the heat of the moment, or just an overwhelming urge to review and reorganise one's plans during the daily commute, there are countless examples that disprove the notion that Nach should be solely used on a desktop or laptop with a stable Internet connection.

By definition, a great offline-first experience is seamless. Whether an Internet connection is available or not becomes a detail irrelevant to the user, which the app abstracts away, internally making sure information syncs in both directions when possible, without any need for user intervention. But despite giving this illusion of simplicity, it introduces heaps of new challenges behind the scenes.

The original Nach web-app opted for a more straightforward approach (to be honest, offline support wasn't even on my radar at the time). The server (a beefed up central computer and database, which stores and manages everyone's data) does a large amount of the work in providing the app. When a user visits the web-app, their web browser sends a request over to the server for the most up-to-date version of their goals, steps, trackers and so on. Once the user has their data, the server subsequently waits patiently to receive back any requests for changes.

Shortly after launch I added support for web-sockets, a technology allowing any changes the server receives to be broadcast out to all other instances of the app the user has open, instantaneously. So any version of the app the user has open, across any number of devices, should always remain up to date with the others.

This system works well, but with a significant caveat: it requires a stable Internet connection. As you may have experienced trying to make changes on the app during a period of no connectivity, you'll be greeted with the dreaded red box error message informing you to reload the app because your changes may not have been saved.

This is because, as soon as it's no longer able to talk to the server, the web-app essentially becomes lost, like a child who's wandered into the wrong aisle of the supermarket, who decides its best course of action is to stay put and wait for an adult. More technically: there are several things the web-app can't figure out without the help of the server; both the app and the server need to hear about updates as they happen, or they'll fall into inconsistent states; and the app doesn't know what to do with updates if it can't just tell the server about them.

Setting out to solve this is one of the main influences that motivated a ground-up rewrite for the mobile apps. Yes, our lost-in-the-supermarket web-app could have, with the right parental guidance, have one day grown up to be able to navigate supermarkets on its own. But that would take years, and when the option of adopting a supermarket-born native is on the table, who knows the route from frozen produce to baking supplies like the back of his hand, that becomes an awfully tempting option.

As I mentioned, there have been many new challenges faced when trying to build up a solid foundation for offline support. The fundamentals are straightforward enough: everything needed to use the app will now be cached offline on your phone; the app will gain all the functionality it needs to be able to carry out key actions without a server; and when changes are made, the app will store up a list of these changes, ready to be sent to the server when a connection is available.

But as with anything, the devil is in the detail. There are so many edge cases that require careful consideration to really make the system watertight. How can changes be resolved coming in from a device which has been offline for days, while other changes have been made elsewhere? What happens if a user makes overlapping changes to the same item from 2 devices before syncing? How can a user who has built up years of data keep within the much more limited storage and processing capabilities of a mobile device?

Of course, there's a lot of fun in coming up with neat solutions to challenges like these. I've developed and been playing around with a system which I'm really happy with as a starting point for this, which nicely handles many of these edge cases.

The ultimate goal is for changes to be accepted from any device, with the proper intention derived, no matter how long that device has been offline, or how much has happened since from other sources. There's a long way to go to reach that, but I'm really looking forward to getting out the first beta out, so people can start using Nach quickly and responsively, no matter where they are.