E-mail clients are interesting in that it’s a somewhat religious decision, comparable to choosing a text editor or operating system; everyone has an opinion and their own feature that’s always a deal breaker. I’ve used quite a few clients, and I’ve never been happy with the IMAP implementations, atleast with regards to Mac e-mail clients, there might be some shining stars on other platforms that I don’t know of, (if you know of one let me know). I’ve been told that Mulberry had the best IMAP support, but screenshots of Mulberry were enough to keep me away. Sadly, the company went out of business and nothing more came of the client. There isn’t much in the way of great IMAP clients. Mail.app is slowly getting better, Thunderbird is probably the best out there right now, but it’s not completely OS X native, and most other clients don’t even support IMAP. This brings up the interesting question:

Where are the kick ass clients?

Part of the reason few excellent IMAP clients exist is due to the complexity of the IMAP spec. POP3 has been a lower fruit for developers to pick, and consequently POP3 support has been much more widespread. Not only is it difficult to get IMAP completely right, but it is a totally different paradigm then POP3, so in order to support IMAP and POP3 increased complexity is required. In fact, many e-mail clients start out POP3 only and add IMAP as patch later, which is causes some of the wonky IMAP behavior. However, it’s understandable that developers don’t conquer IMAP once you compare the specifications of IMAP vs. POP3:

IMAP Spec (RFC 3501) 6051 Lines

POP3 Spec (RFC 1939) 1291 Lines

The IMAP spec is far larger than the corresponding POP3 spec, so it’s not surprising that it’s a bitch to get right. Even then, that is just one RFC; there are other RFC’s for IMAP which add extensions like IDLE support, UIDPLUS, the UNSELECT command, and more (check out IMAP Connection if you’re interested).

Those 6000 lines describe just the basic protocol, people also expect stability, speed, and, disconnected support in a modern client. IMAP has proven itself to be stable, and fast, but disconnected support is a curious issue. IMAP wasn’t designed to support disconnected access, but rather an always on connection, but wait…

What’s online, offline and disconnected?

Let me regress for a moment and explain. An offline client uses POP3, you connect to the server, and download your mail to the local machine. Whenever you read your e-mail it’s being fetched from your local computer. An online client is what IMAP was first designed as, when you read your e-mail you always have an active internet connection, your e-mail stays on the server and messages are downloaded on demand. A disconnected client is a fusion of the two mail models. In a disconnected client your local copy of e-mail is synchronized with the server. You can still manipulate your mail and the client is smart enough to copy your changes to the server when you have an internet connection again. Because of the flexibility it’s not surprising that many users have come to expect disconnected mode.

Where does this leave us?

Disconnected mode is very convenient but it’s difficult to get right. The IMAP protocol is very difficult to use in disconnected mode because it wasn’t originally designed for that purpose. In IMAP, every message has a UID (Unique Identifier) and a every folder has a UIDValidity value, the combination of these two is unique across the account. This does make it easier to program a disconnected e-mail client, but the problem is that the UID’s aren’t always static, and some servers are better with this than others. What can happen is that the server might decide that it’s time to change the message UID’s for whatever reason, and then you need to purge your local cache and download everything in that mailbox again. If the developer fails to verify the UID before executing a command on the server, some very nasty things can happen. For example, you might have a UID stored for a message that the user would like to delete, so you send a delete command to the server, but the UID validity is never checked. If the UID has changed, in the best case the delete operation will fail, but in the worst case it could remove a message that is now using the other message’s old UID. This is of course, not cool, and it would be nice if UID’s didn’t change. Some servers are crappy in that they change the UID’s far more than is necessary while others attempt to keep the UID constant for as long as possible. And just as it is hard to build an IMAP client, it’s even more of a bitch to build the server.

Another problem with IMAP is that it’s very time consuming to update all off the message flags. What do I mean by message flags? I’m taking about the flags that mark if a message has been read, replied to, forwarded, marked for deletion etc. There are some tricks out there using UID’s to only download new messages, but to make sure you have the latest flags you have to iterate over every single message, and this can be very time consuming for a large mailbox with thousands of messages.

What next?

What I propose instead of time consuming iterations is a time stamp. You send the server a time stamp, like for example the last time you connected to the server, and it plays back every change it has made to the mailbox since that time. So syncing becomes much easier, connect to the server and send the time stamp, the server sends you everything that has changed since that time. No more time consuming flags updates, or UID tricks. It would be even nicer if we could do away with IMAP and replace it with something simpler, and more generic. Why not write a generic synchronization server like .Mac that you can use to synchronize objects with? Then add support for time stamps and we can start syncing everything on to a central server. Then we can sync all kinds of data across computers just by using the generic protocol. Oh, and let me know if I missed an IMAP detail, or there is some magic trick I don’t know about it.