A new protocol would be ... prettier, I guess. But to pretend that the reason facebook and IM are replacing email is IMAP (and not, y'know, spam) is borderline delusional.

The Android GMail client is a perfect example of what a client looks like in this world. It talks to the (secret / private) GMail API, it does offline mail reading, and queues actions so you can archive / filter / whatever mails while offline and it’ll push changes later. You can read and write mail. It doesn’t try to do anything clever, because anything clever done on one client isn’t reproduced on any other client. And if I don’t have a client on my current computer for GMail, I can use a web browser, and still get all the features of the server. I use the web gmail interface for everything anyway, because it’s better than any GUI client I’ve got.

I dont know - this guy makes for pretty convincing arguments on why email should be done right on the server rather than the client. Which effectively means a new protocol.

(For example, I don't like gmail's flat "conversation" view; I much prefer a "threaded conversation" view, like classic Usenet readers, or HN discussions.)

I think what is meant by a new protocol and server innovation is a change in the data model. I do not believe they are going to dictate views.

Yes, we could be happy with the status quo. But I do believe that one of the things that's holding up innovation around email is the lack of easy access to data. Proposed exercise: Write a script that downloads and displays your Twitter feed. Then write a script that downloads and displays your email. The email exercise will take you an order of magnitude longer.

Spam filtering isn't the problem you make it out to be. I get more spam in my Facebook inbox (mostly viruses) than in Gmail. If you're hosting you're own email or are using a sucky provider, that's a different story.

Email will be hard to replace long-term anyway. I doubt your bigcorp employer really going to let you send work-related Facebook messages. IM is a fundamentally different protocol with different use cases (think async vs. sync, fax vs. phone).

  $mbox = imap_open( $host, $login, $password );
  foreach( imap_search( $mbox, 'UNSEEN', SE_UID ) as $uid  ) {
    $header = imap_fetch_overview( $mbox, $uid, FT_UID );
    echo $header[0]->subject . "\n";
  }
  imap_close( $mbox );

    $status_url = 'http://twitter.com/statuses/user_timeline.json?screen_name='.$user;
    $timeline = json_decode(file_get_contents($status_url));
    for($i = 0; $i < count($timeline); $i++){
        echo $timeline[$i]->text ."\n";
    }

IMAP and MIME are hard because there is a higher level of information that is available. Can you attach an archive over Twitter (not link to but attach)? Can you send HTML? They are not the same thing. If you want something really simple your code will be simple. If you want more use cases than "i can haz cheeseburger" then you need more expressive power and more complexity.

As is believing that it's about spam (or probably believing it all, for that matter, but let's stay on topic).

I don't understand people bitching about spam in 2010. I have probably a dozen email addresses that I publish all over the web and I get maybe one spam message a month. I get way more "spam" through Twitter than I do through my email.

I have had false positives with every single spam filter I have tried. This is unacceptable when running a business, and if we manually have to double-check hundreds of spams each day, what is the point of the filter?

Additionally we spend hours each week working with users who do not receive our emails because of spam filters unknown to them (many believe the fault is ours).

It is 2010 and spam is still a major problem.

It's faster to scan a list of subject lines and senders than to look at the contents of an email, which is faster than actually deleting it. So verifying that the spam filter sorted your emails correctly is faster than sorting every message yourself.

A better protocol will only be "better", analogous to, say, the Mac (c. 1990) being better than Windows 3.1. While true, compatibility pressures aren't going to allow someone else into the market simply for being "better". You have to be transformative, and this isn't.

And in any case what's killing email isn't the lack of apps, it's the lack of authentication and moderation.

I'll only point to some "highlights".

OAuth isn't nearly secure enough (http://hueniverse.com/2009/04/explaining-the-oauth-session-f...) for your e-mail (which, don't forget, gives access to everything via password resets). Let's just ignore the part about third-party access.

Encoding everything as UTF-8 is nice, but breaks the first time some idiot mail client puts Shift-JIS encoded Japanese in the Subject: field. Without any charset marker, obviously. (This goes right back to the "where does the mail come from?" issue.)

IMAP has a THREAD extension (solving #4), a SEARCH extension (solving #9 and parts of others), and all sensible servers support IDLE (http://en.wikipedia.org/wiki/IMAP_IDLE). It also supports downloading MIME parts separately, solving the part of #7 that can actually be solved (again, you'll have to accept mail.)

I think IMAP could be replaced independently of SMTP. You could conceivable replace an IMAP server with a reMAP server and not affect the rest of the mail ecosystem. SMTP will likely not die for another hundred years.

1. I know about the OAuth problems. Do you think we'll see a more secure solution in the next few years?

2. Is there an easy way to validate UTF-8ness? Happy to hear suggestions.

3. I know about THREAD and SEARCH and IDLE but find them clunky. Under the assumption that we're replacing IMAP with something RESTful you might as well get those ones right.

Happy to take this offline as well - my email is on my HP

We agree SMTP will be around forever; but how do you plan on abolishing MIME then? Your system will have to handle it, no matter what. Do you want to push MIME handling to the server? IMAP already does that, there's nothing revolutionary about that.

Due to the special structure, it's usually possible to distinguish UTF-8 from other 8-bit text. The tricky part is what to do with unmarked non-UTF-8 8-bit text - it could be pretty much anything. Mutt, for instance, allows the user to configure this according to what the user expects to see. Quoting the manual,

    For example, Japanese users might prefer this:
    set assumed_charset="iso-2022-jp:euc-jp:shift_jis:utf-8"

I probably wouldn't trust my e-mail to a protocol that was only a few years old. I don't anticipate any new security protocols with significant uptake either; "serious" stuff already uses SSL/TLS plus an authentication method (passwords/certificates/SASL), and the web crowd is unlikely to replace that (OAuth is about weakening security in a controlled way, not replacing SSL.)

And yeah, parts of IMAP are clunky. But that's not enough reason to rip-and-replace it.

Regardless, I would like to see public key encryption of all email. There's no reason why messages should be plain text anywhere except at the two ends.

I realize key management would be a problem, but it's probably a solvable problem. The solution would have to allow most people to be practically unaware of encryption issues.

What is important is that the IMAP protocol isn't difficult, and I don't think that is what limits "email innovation." It is more so that it has been around since the 70s and email clients haven't been able to remove themselves from their past. It isn't the protocol limiting them from moving forward, it is the shear number of other clients that wouldn't support any of the "innovations" one would try to bring to email.

Project wiki: http://letters-wiki.heroku.com/

1. HTTP / HTTPS - please don't... There are stupid/broken transparent proxies in many places that will effectively block this protocol. Also custom TLS connection gives some hope for domain-specific certificates instead of ip-specific - there is enough problem with them in HTTPS as it is.

3. People still don't know how to set the encoding (or what the encoding is) and will happily send you Greek in latin1 or something equally silly.

4. Would it really work for mailing lists where conversations naturally fork into many branches?

5. IMAP supports labels... even if it's crazy number-based labelling where every client needs to set its own names for the labels. We'd just need to make it not suck this time ;)

So it's not all that bad in IMAP - there are many areas where we can do worse by accident. But yes - it would be nice to have server-side text searching...

I agree with this. The idea that messages must be accessed by conversation is a bit much. Support for conversation group could be nice, but ignoring that feature would have to be allowed so that clients could implement their own views.

For example, the following structure would be crappy to force on clients:

  Label
  `-> conversation 1
      `-> msg0
      `-> msg1
  `-> conversation 2
      `-> msg0
      `-> msg1

1. I'm not convinced that http/https is the best protocol. I agree that it's the defacto standard, but if you're going to go through this, there might be a better way. (Push, in particular, is a problem).

2. Stateless is wonderful, agreed.

3. I agree with JSON over XML, but what about binary data? That is the weakest part of JSON, in my opinion.

4. I like where you're going with this, but I'm not sure this is the exact right way to handle it.

5. Agreed.

6. how about SHA-1 hashes of content? Done properly, that would eliminate any duplicate messages, and could make it very easy to include other messages. This might also allow content to be transfered between accounts only once, dropping bandwidth and speeding delivery.

7. Don't know enough about MIME to comment.

8. According to your comments on this page, you're referring to long polling. I agree that this use case needs to be addressed, but I'm not sure that this is the way to do it. Perhaps multiple options here?

9. Good idea, but do you think that stemming/language differences is going to be a problem? What about other types of content? attachments?

The biggest thing missing from this list is some sort of Public Key Encryption. Anyway, it looks like a good start.

What binary data? All attachments in emails are Base64 encoded. That's what MIME is all about.

> 6. how about SHA-1 hashes of content? Done properly, that would eliminate any duplicate messages, and could make it very easy to include other messages. This might also allow content to be transfered between accounts only once, dropping bandwidth and speeding delivery.

Developers should start to shy away from SHA-1 since it's already started to get broken.

>> 6. I wasn't intending it as a security mechanism, but as a way to know which bits you have and which you don't. Do you think SHA-1 is still a bad idea in that case?

IMAP might do some sort of MIME-handling, but MIME is part of the email spec, not the IMAP spec. Hating IMAP because of MIME is like hating cars because of the invention of the wheel (i.e. thinking that the wheel was invented as part of the production of the initial automobile).

http://en.wikipedia.org/wiki/MIME:

  Multipurpose Internet Mail Extensions (MIME) is an 
  Internet standard that extends the format of e-mail
  to support:
    * Text in character sets other than ASCII
    * Non-text attachments
    * Message bodies with multiple parts
    * Header information in non-ASCII character sets

If you use Gmail, try the 'open original message' option on emails that have attachments or multiple text parts on them (or even emails that have both a text and an html version of the content).

Dealing with MIME is part of dealing with emails in general, regardless of if you're interacting with an IMAP server, or composing an email to send to another user on the same mainframe (i.e. just moving files around).

That is a security mechanism. If I can inject a message with a hash that collides with something else in your queue, I can at least prevent you from seeing the original, or potentially spoof a message from someone else.

I know it depends on knowing hashes of other messages in the mailbox, but if the hashes are supposed to persist, that might not be a problem.

Duplicate suppression can be tricky. The overhead may not be worth the benefit, and the risk of message corruption is increased. Besides, if the mail is encrypted by the user, it may not even be feasible to do server-side.

Maybe BSON?: http://www.mongodb.org/display/DOCS/BSON

I don't need my protocols to be human readable. I need my computers to read them and know that I've got good, bug-free, reusable, efficient, and easy-to understand implementations.

This isn't an argument for XML over JSON -- I've made good use of both, but I don't want to consider how easy it is to type up communication protocols when I'm defining stuff the replacement for something I expect to take over the world.

It's by far easier to implement the memcached binary protocol as a client or a server than it is to implement the text protocol. Easier to get right, more regular, and just generally more pleasant.

it'll require an RFC or some other kind of standard, which will take a long time to finalize. then someone will have to write a reference spec server and client, then "real" versions in "real" languages, then get them both to the stability, security, and scalability of current imap clients and servers. then once big email providers (isps, google, etc.) start supporting it on the server end (gotta factor in all those different operating systems, authentication systems, storage systems, etc.), popular clients can support it. then work out all of the minor implementation details that some servers or clients get wrong (i think almost every smtp and pop3 server has hacks in it to support the stupid things outlook express does wrong), and then once everyone complains long enough, the iphone will finally get support for it (while a small but vocal minority shouts that android already supports it).

there is a reason why smtp, imap, pop3, dns, http, and all of the other core internet protocols are still around after 20 years and it's not because there aren't any faults in any of them.

If the benefits turn out to be worth it, maybe it'll go somewhere.

That said, it's easy to skim this article and nod, "Yeah! Yeah!", but when we're discussing these, laymen may need to know a few of these seem over-simplified.

> "TCP connections are great, but for transferring large amounts of email securely, HTTP is the way to go."

1. http://wiki.answers.com/Q/What_is_the_difference_between_tcp... - HTTP uses a TCP connection. For that matter so does SMTP and IMAP and the proposed REMAP.

2. I generally have at least 5 devices using IMAP (on Google Apps) at once, and interleave my interactions with these devices arbitrarily, yet expect each to show me the exact same state when I look at it. Keeping them all in the apparently same state is the sort of intractable software problem bedeviled with implementation details, and I agree throwing out IMAP and starting over with lessons learned could be easier than, say, the RFC linked below.

3. http://ajaxian.com/archives/json-vs-xml-the-debate - Seems this point is something like GIF vs JPG vs PNG, with a bit of "Markdown vs HTML" thrown in. Meanwhile, both XML and JSON are missing an important concept given labels/folders and conversations: multiple references to single objects.

4. Conversations or threads exist today as In-Reply-To: GUID and References: GUID headers, among others, but see the JSON problem above. We don't realize these headers are there because clients generally ignore these and try to group on Subject instead. Gmail grouping respecting existing headers is much better, but from usability point of view, Gmail's approach makes it challenging to bulk delete individual matching messages across a broad set of conversations. For example, to bulk move or mark all messages on a particular date results in the full conversations being moved or marked. So I like this idea as long as I can still optionally operate on sets of messages without affecting the rest of the conversation.

5. http://googlesystem.blogspot.com/2007/08/organizing-chaos-fo... - It's not yet clear that people prefer tags to folders. After all, monkeys don't expect a banana to be in two boxes at once. I don't want to label anything with keywords. I want semantic search.

6. Excellent point. Changing GUIDs out from under us is just lame. Having had to resort to http://github.com/rgrove/larch recently to move a decade of email onto Google Apps because Thunderbird was putting new IDs on each moved message, I fully agree this problem is evil.

7. Seems we're stuck with MIME until we get rid of every legacy email server in the world, or control what servers our recipients use. Between the new server and new clients, sure. But the server will have to know it to send it. Granted, IMAP is for receiving, and SMTP is for sending, but something in the chain has to know MIME.

8. "Call" an HTTP endpoint that "returns" when messages arrive? Just nomenclature, since IDLE is a "call" that "returns" when state changes. Either way, the socket is held open, so neither of these is "push". Maybe Gabor meant "posts back" instead of "returns"? In any case, under the hood, iPhone push and Microsoft ActiveSync are exploiting out-of-band channel such as SMS to notify the phone it should poll (or in case of ActiveSync, maybe doing a "long pull"). See http://msexchangeteam.com/archive/2004/04/26/120520.aspx versus http://tools.ietf.org/internet-drafts/draft-maes-lemonade-p-...

9. Will the RFC specify a search results algorithm, or will the same search return different results depending on the implementation of mail server the client is connected to? We manipulate a "client" tool to perform the search and access the email, so our natural mental model is of the client at hand, not of the remote server. This lets the brain wrap itself around Gmail's web search versus Apple Mail's search box vs OS X Spotlight vs Gabor's reMail all returning different results. Practially speaking, search at the server does make far more sense from both a data retention and security standpoint. I have 509 MB of index in Gabor's "reMail" app on my iPhone, and would rather not.

Thanks for brainstorming this, and thanks for a kickass iPhone app.

Yes, of course. He is not claiming that TCP is bad, but rather that it is too low-level. i.e. One wants an application protocol (e.g. HTTP, SMTP, IMAP) rather than a transmission protocol (e.g. TCP).

The HTTP verbs should be sufficient for email, so why not leverage them?

It's not yet clear that people prefer tags to folders. After all, monkeys don't expect a banana to be in two boxes at once.

Heh, dare I reply? Tags can be used as pigeonholes, and this can be enforced if desired. But why? Tags are so much more useful because they are not pigeonholes.

Here is a classic example:

Does my app config go in a system directory or an app directory or a config directory? the executables? the daemon? the log files?

The classic approach is to put system-level config files under /etc, log files under /var/log, executables and daemons in various locations, etc. Some newer approaches (HomeBrew, GoboLinux, etc) keep everything organized per-application.

Instead, keep every file in the global folder / namespace. Tag the executables as exe, the log files as log, config as cfg, and also tag everything with the app's name. Possibly the version, etc.

A file is uniquely identified by its tags and its name. Any combo can be used, so long as it doesn't already exist.

You don't have to make painful decisions about the unique place in the tree that a piece of information should exist. Just tag it appropriately and move on. The tag system could even create an optimal concrete hierarchy based on tag and tag association metrics if needed.

But the server will have to know it to send it. Granted, IMAP is for receiving, and SMTP is for sending, but something in the chain has to know MIME.

I'm not sure if this is GoF, but this sounds like the classic gateway pattern. You put MIME gateways at the edges of the new email network. MIME-only servers must go through the gateways.

You need to be able to specify both parent and child tags to be able to facilitate search and give users something that isn't too confusing re: organization.

I have a script that monitors my desktop. Any file not modified within the past 8 hours is moved to a folder dated for the end of this week. That gives me 52 folders a year, granular enough to find a file manually if I know roughly when I dealt with it.

My desktop stays clean except for what I'm actively working on, and I don't look for files hierarchically or by category/tag. To find something, I use Spotlight search. That's faster than navigating a folder hierarchy would be even if I knew exactly where the file was.

That's also why I love Gabor's reMail on the iPhone.

I am having trouble understanding what the problem is, here. I also suspect this may be a Gmail problem rather than a tagging problem.

I frequently put different emails from the same conversation in different folders, because they contain different types of information that I need to do different things with.

GMail will not let you do that.

What version? This doesn't happen to me on 2.0.0.23