Hacker News new | past | comments | ask | show | jobs | submit login
Rim/Blackberry tales – reply all (awadwatt.com)
159 points by cloudedcordial 55 days ago | hide | past | favorite | 79 comments



Some nice nostalgia for me

-In high school we walked to that exact KFC for lunch and would discuss the previous nights antics playing StarCraft broodwar.

-I used to fix computers (professionally) at a store on the same street as that gas station as an after high school job

-In Dec/jan 2010 I worked 18 hours a day laying floors in the new RIM buildings at Philip/Colombia. A friend’s dad did a lot of the furniture moving. Both of us made over $4000 a week in our early 20s

-Now out of those 4 buildings I think black berry only has two floors of one building

-Waterloo has seen serious decline since the death of RIM

-Not sure it will ever come back, most people including myself left years ago.

-there has been a serious condo tower boom, but that sucks for “walkability” and it’s radically changed the area

-if you attended university in Waterloo in the 2000s and lived off campus, wherever you lived is likely gone and there is a condo tower there now.


>-Waterloo has seen serious decline since the death of RIM

>-Not sure it will ever come back, most people including myself left years ago.

I think you overestimate Waterloo's decline just a tad, perhaps your perception being coloured by leaving it. I assure you, it's thriving and in many ways better than the early 2000s when I went to UW. Including the condo boom you mention, though I'm puzzled why you think this somehow hurts walkability.

But yes, Lester street is unrecognizable and every single house I lived in between 2002 and 2007 is gone.


Agreed 100%. Waterloo has grown substantially over the last 15 years, and is generally thriving. The condo boom and general increase in density has only increased walkability. Most of these condos have ground level retail too. The light rail is also a nice addition.

As RIM/Blackberry declined, a whole ecosystem of startups emerged started or staffed by ex-RIM folks. The universities have also grown substantially.


The wild thing about the condo boom in Waterloo is 77% of units are owned by investors. It truly exemplifies the mess that is the housing market. Rentors can't break into it and homeowners are doubling, tripling up on properties.


The Canadian housing market is apocalyptic. It's hard to think of a greater regulatory failure.


That's a lot of SW Ontario now. I've moved back to the US due to the high costs of the GTA, but a lot of my friends in Toronto own multiple condos.


I was in the area around 2010 working for someone else. Adding to your bullet points:

- The pool business near the single-digit RIM buildings had more business than they could do. Many folks wanted swimming pools at their homes.

- Various eateries such as the sandwich shop mentioned in the article made decent money during the height of Blackberry.

- People skipped starter homes and bought single houses as their first homes. Some real estate agents waited outside of some buildings during bonus was announced.


> People skipped starter homes and bought single houses as their first homes.

This is a terminological distinction I am not familiar with; what is a "single house", and what is the difference between a starter home and a first home?


Maybe they consider a townhome or apartment condo as a starter home, which is true in the current market, but 25 years ago it wasn't uncommon to buy a detached as your first home.


They mean a single family home which might be described as a "detached house"

In the GTHA (including Waterloo) there is no such thing as a starter home any more, which in the past meant "small detached house, probably needs some work". The only thing they build now is very small 500 sqft condos and very large 3000+ sqft houses.


Your username is nice nostalgia for me. :-)


Density has hurt walkability?


I have no specific experience with Waterloo, but sometimes towers follow a Corbusian ideal of a tower surrounded by nothing; or worse, a tower surrounded by high-speed roads/highway — essentially a stacked bedroom community with no walkable amenities.


FWIW, I live in the region and disagree with OP's characterization of "serious decline" and "most people have left".

I went to school here from 2003-2008, moved away and moved back in 2011.

The area's population has increased by ~20% since 2012 (~the death of RIM, according to its stock price). In 2011, it got regional train service to Toronto. In 2019, it got a local light rail train.

The university area that the OP seems to be referring to is, IMO, more walkable and bikeable now than before. Some of the towers are mixed use, with ground floor retail.

The city is definitely quite different from the early 2000s, though.


A dense housing boom could hurt walkability if it replaced mixed low density retail and housing and if there was no compensating retail boom near housing.

First floor mixed use retail can address this, but sometimes those spots sit vacant because of cost or other issues with rhe space.


I guess my definition of walk ability includes seeing interesting things and being able to walk towards them. The condo towers block the sun and you can’t see beyond them walking down the road.

My post was a bit too negative I suppose, some people probably like the tall buildings surrounding you when you’re walking, I mean it works for New York City right.


I'm glad to see WCRI is still there, though it's hard to claim it's fully "off-campus".


Did you go to WCI too? This is all so nostalgic for me.

Though I grew up in Waterloo and lived at home, yeah, the city sure has grown a ton. I moved away to raise a family.


Username checks out haha!


Yes. Grade 9 in 99 :)


Cool. One year behind you. So we went at the same time.

I want to say small world but after enough time it’s bound to happen, eh?


That random “J” at the end of the messages brings me back to mail circa 2010. As I recall, iOS also didn’t render Outlook’s smileys right, leaving a bunch of Js in mail from my Mom.

(For the ones who ‘missed’ it: https://devblogs.microsoft.com/oldnewthing/20060523-10/?p=31...)


for a long time, I thought people were just really into giving me a single letter nickname


Lots of email clients still don't.


same here J


!!!1one


> Blackberry at the time was kind of an “everything goes, whatever it takes” free for all. And this damn the torpedoes full steam ahead attitude was pervasive everywhere.

> RIM Job

I love that this was actually the URL for their careers page in this era: https://web.archive.org/web/20101122175558/http://rim.jobs/


I don't know why these articles (and many comments here) always make it sound as if the people replying were idiots. For many, or I'd rather say most in this case, it's clear that they know exactly what they're doing - having some fun given the opportunity that presented itself.


Reply all storms are some of the most amusing things to witness and joke about with your colleagues. Sadly I haven’t seen a good one in a very long time :-(


Poor Sumit B. He never did anything - it was his manager!


Sumit B forever has a story to tell at dinner parties.


This happens often enough (even MS itself broke their internal Exchange servers).

And every story seems to end with admins having to improvise. Am curious: (why) isn’t there a “kill reply-all chain” button as a feature?

(The article explains that this didn’t work for RIM because of BB’s architecture, but for Exchange?)


In theory, mail should be stateless, so what would they use to do that? Some clients and servers understand headers for thread identifiers, and then it would be possible to (in theory) zap that id, but all it takes is one client not supporting it for that to be out the window. The article kind of points this out - Exchange had a mitigation, but the clients got the notification and were able to reply because by then the mail was on the client (pushed).


The search term for MS for is bedlam 3.


I worked at MS in 2012 and we had another email storm that shut down almost everything. I remember not being able to do any work at all because the network was so slow I couldn't even edit files (the internal source control system required getting a lock on the file to be able to edit them).


The standard fix is to massively limit who can send emails to DLs - which is an Exchange config option.



> it’s estimated that over 15 million emails were generated in the space of about an hour.

wtaf?


Email storms and nuclear meltdowns have the same rough physics.

If 1% of emails are blind replies from people out of the office, but are tracked by the originating sender (not the DL) since OOO messages in Exchange get sent to the DL (as well as the originating sender) and soon enough you have Exchange sending a new OOO email per reply from a new reply-er. Your typical company the size of Microsoft is going to have probably... 3-5k people out of the office at any one given time (the lingo is "OOF", or "out of facility")

OOOs can also trigger another OOO, which can in turn cause a new OOO to spawn.

Soon, you've got OOOs reply-all'ing to OOOs and drenching a DL... and an exchange forest.


The world needs a driving licence for email. It would mandatorily include use of plain-text and bottom-posting.


Europe has one. I’ve got one. I am qualified to use 2004-era Microsoft software, and have a certificate to prove it… somewhere.

(Edit: since I got mine it’s acquired the word ‘international’ but lost the word ‘driving’. Swings and roundabouts.)

https://en.m.wikipedia.org/wiki/International_Computer_Drivi...


I used to think the same but I caved when I started using web interface email clients. Let’s face it, the world has moved on, new generations are online, and we are the wrong ones now.


> I caved when I started using web interface email clients

It's one of the things I like about Gmail. It does plain-text and bottom quoting just fine.

> Let’s face it, the world has moved on, new generations are online, and we are the wrong ones now.

NEVER GIVE UP! NEVER SURRENDER!

https://mygeekwisdom.com/2014/03/15/never-give-up-never-surr...


I don't understand why this isn't configurable. Why can't the basic data structures (of one email after another in serial) be displayed in either order depending on mail client settings?

Admittedly I haven't looked into it because I'm perfectly fine with top posted emails. But I routinely sort files in my directory. Why not emails in a displayed thread?


It's people quoting text, not threads of messages.

The ability to semantically parse text to determine what order paragraphs should be displayed in to suit the tastes of the individual reader is a very recent development. Or, rather, will be soon. Maybe not very soon.


Quoting the prior message(s) should be off by default. When paper letters were still written, did you enclose a copy of the original letter when you answered it? No. And nobody looks at the growing trail of the entire message thread that's copied below your reply. Just leave the subject line intact and anyone that doesn't have a brain-dead email client will see the messages threaded properly.

If you need to address a specific point in a message you're replying to, quote just that bit.

We are emailing TBs of data around daily that provides no value to anyone.


> When paper letters were still written, did you enclose a copy of the original letter when you answered it?

The point of bottom posting was you never left the original text intact, but trimmed it to show the relevant details you are replying to and so enhance readability. Exactly as I am doing now in this reply.

> If you need to address a specific point in a message you're replying to, quote just that bit.

Precisely. Just like this. We are on the same page!


As the other comment mentioned, the email body contains the entire quote chain. The way clients accomplish threaded display is a combination of:

- parsing the unstructured email body and looking for quote levels, html formatting and printed email heads

- parsing certain headers like message-id, in-reply-to, dkim sig

- looking for sections of the message body in the inbox

This is done because there is nothing in the protocol to cleanly accomplish what you want. Even if there was, you could not rely on it at all. Doing anything with email is a gigantic PITA, you sometimes get emails where the msg-encoding header doesnt match the body's encoding, html in the plaintext section and other fun things.

Since nobody really cares about the RFC and just does their own thing, there is no chance at improvement.


This is true. OTOH, I do think the problem is solvable.

I came up with a routine to parse and translate about 2-3GB of saved emails into MBox format once.

The official delimiter is unbelievable, IMHO.

« the exact character sequence of "From", followed by a single Space character (0x20), an email address of some kind, another Space character, a timestamp sequence of some kind, and an end-of-line marker. »

https://datatracker.ietf.org/doc/html/rfc4155

https://en.wikipedia.org/wiki/Mbox

That's it. An email is a section of text beginning with

From $something

That's the spec.


Certain software used to add a > before any line starting with From in an email body because of this.


If the entire email is being replied to, I can just read that email, which is displayed immediately before the current one in a threaded display. Why should I have to scroll past another copy of that email?

However, the original email is included as a convenience in case my MUA doesn't support threaded display or it's a mailing list I joined after the original email was sent or any other reason why I might not see it. That's why there is quoting at all.

Nobody top-posts when using selective quoting because obviously it's different.


I hate both of those things. I guess this means war, please look for my envoy bearing the formal declaration of war.


You are wrong.

https://useplaintext.email/

The biggest and most successfull FOSS project of all time is coordinated entirely by email. These are the rules:

https://subspace.kernel.org/etiquette.html

Again: you are wrong.


I don't give a shit what the Linux kernel does. It doesn't change the fact that I think plain text is ugly, HTML mail has no drawbacks I care about, and bottom posting is just plain weird.

> Again: you are wrong.

No, you.


I'm teaching the chapter, "Why is everyone signing off with J? A crash course in email from Windows users"


> bottom-posting

Immediate and indefinite suspension of email license.


A: Because it messes up the order of things.

Q: Why is top posting bad?


You jest, but I've noticed that the conventions of "newest on top" vs "newest on bottom" is _seriously confusing_ for some people that I help navigate tech stuff. I don't know how to describe the heuristic for:

- New text conversations show at the _top_ of the list of conversations - New messages are at the _bottome_ of a conversation - New emails are at the _top_ of your email client (?) - and now you remind me that email replies can be both at top and bottom (:

It feels arbitrary, but I suspect this is due to the heritage of paper, where newer things are on top of the pile, but in a given document, newer text tends to be added at the bottom/end. (it's a stretch :))


No.

Bottom posting replies was the default in all early email and USENET clients.....

Then MS Outlook came along, which was the first email client to break convention and default to top posting (i.e. putting the cursor at the top when replying to a message). Thence forth, "office" users began top posting, and the confusion began.

To this day, the old guard (like me) bottom posts and always trims the above quoted text of irrelevant details (!). Anything else was considered not only lazy and sloppy, but the mark of a noob with bad netiquette.


A significant part of weirdness involving outlook/exchange is that it is not an SMTP-based email client

Even though X.400 is no longer officially feature of Exchange, the entire data model of MAPI is based on it, shared between Outlook and Exchange, with somewhat lossy translation when it has to go outside of X.400-over-RPC that MAPI provides.

Sometimes you can get burned by vestigial parts of the model, like how MS MAPI implementation as provided by Outlook/Exchange (there used to be others!) does not actually support HTML email, and crashes with corrupted message errors if given an email object containing a HTML body.

Now I can hear you going "but Outlook does HTML email!".

Outlook converts HTML email body part into RTF-wrapped HTML and stores the resulting message in RTF body field of message object. In fact, before Outlook got changed to convert RTF to HTML, Outlook users were infamous for sending RTF formatted emails (at least RTF was always documented, as it was supposed to be interchange format).

But MAPI messages do also contain a HTML message body field... But if you put HTML there, MAPI.DLL explodes - or at least did every time I did it.


Only if you ignore that I just read the previous message.

In reality it's:

Q: Why is bottom posting bad?

Q: Why is bottom posting bad?

A: Because it repeats itself.

Q: Why is bottom posting bad?

A: Because it repeats itself.

Q: Isn't this also a problem with top posting?

Q: Why is bottom posting bad?

A: Because it repeats itself.

Q: Isn't this also a problem with top posting?

A: No, because the reader skips to the next message when they get to the quoted part.


A local technical college has a dedicated course on "Slack at workplace" (or a paraphrase of the title).


Hopefully they’re discouraging students from sending lots of fragmentary messages interspersed with superfluous nonsense like “hey” “i mean” etc


I vividly remember this incident and more like it. I was coached by my manager to never reply all to an email addressed to a larger distribution group because of this exact reason. Not only were you annoying people, but everyone's email ended up getting delayed for an hour or two.

Can someone explain to me why the backlog would happen? Why they didn't have systems to protect from such a basic DOS attack?


from the first Exchange Reply-All email storm, a dev who worked on the Exchange server, https://techcommunity.microsoft.com/blog/exchange/me-too/610...:

    An Exchange email message actually has TWO recipient lists – there’s the recipient list that the user sees in the To: line on their email message. This is called the P2 recipient list. This is the recipient list that the user typed in. There’s also a SECOND recipient list, called the P1 recipient list that contains the list of ACTUAL recipients of the message. The P1 recipient list is totally hidden from the user, it's used by the MTA to route email messages to the correct destination server.

    Internally, the P1 list is kept as the original recipient list, plus all of the users on the destination servers.  As a result, the P1 list is significantly larger than the P2 list.

    For the sake of argument, let’s assume that 10% of the recipients on each message (130) are on each server. So each message had 100 recipients in the P1 header, plus the original DL. Assuming 100 bytes per recipient email address, this bloats each email message by 13K. And this assumes that there are 0 bytes in the message – just the headers involve 13K.

    So those 15,000,000 email messages collectively consumed 195,000,000,000 bytes of bandwidth. Yes, 195 gigabytes of bandwidth bouncing around between the email servers.

    ...


    So what did we do to fix it? Well, the first thing that we did was to fix the MTA. And we tried to scrub the MTA’s message queues. This helped a lot, but there were still millions of copies of this message floating around the system.

    To prevent anything like this happening in the future, we added a message recipient limit to Exchange – the server now has the ability to enforce a site-wide limit on the number of recipients in a single email message, which neatly prevents this from being a problem in the future.

It didn't fix the problem completely from what I recall, there were smaller versions of Bedlam at MSFT. I've heard that some branch of the US Dept. of Defense created their own Bedlam storm a few years back. So they had to layer in a few more guardrails to prevent another reply-all from getting out of control.

Here's one reference, https://www.theregister.com/2023/02/14/us_army_reply_all_sto..., though I thought they had one back in the 2010s.


Another storm hit MSFT around the start of the pandemic, https://forums.theregister.com/forum/all/2020/03/26/microsof...


Maybe the last fix for the reply-all email storm problem (at least for Exchange servers)?

see https://www.theverge.com/2020/5/10/21253627/microsoft-reply-...


> Blackberry at the time was kindof of an “everything goes, whatever it takes” free for all. And this damn the torpedoes full steam ahead attitude was pervasive everywhere. Red tape and beuaracracy was loathed by everyone.

Ironic as RIM became known for its cripplingly dense beuaracracy and red tape.


Not surprising though that the negative consequences of one extreme can lead to an overreaction in the opposite direction.


Every company has lore like this; I remember when I joined Amazon hearing how someone would create SEV-1 tickets because their phone didn't work (Jeff B. would get personally paged for all SEV-1 at the time)


I was working there that fall as one of my co-op work terms!

I recall the onboarding tour around the testing rooms which were essentially giant Faraday cages. There was a print-out on the door exhorting employees to CLOSE THE DOOR! when you come or go. Apparently it was a semi-monthly occurrence where someone would accidentally leave the door propped open and the nightly tests on upcoming devices would make real 911 calls to the local dispatchers as the E2E tests on physical hardware were running.


I love reply all events and have fond memories of the adventures of the Fluke Meter at GlobeSpan (long defunct.)

It started when one good soul sent out a worldwide email asking "Who has the Fluke meter?" and after the first person replied "It's not here!", the rest of the world reacted in kind.

It took about a day for the storm to die down.


Haha I was working there at the time. I don't remember how many emails came in but well over 100.

Emails were so abused there though. I would get over 100 a day that were work related. Think Slack over email.


Yeah, I'm pretty sure this isn't the only time it happened - I don't recall Sumit but I do recall others.

I worked in the NOC and you quickly learned to basically ignore every non-personal email sent before you were on shift that you weren't directly copied on. If it wasn't your shift and it wasn't handed over, it wasn't important.


Ya this definitely happened a few times.


I once had that happen to me with spam. A spammer had set up full-fledged mailing list software to send spam. With the reply function active!

I'm sure you can imagine the rest.



A few years ago we got the same thing: a guy sent a email to finance showing his bonuses and somehow sent it to everyone. Then everyone knows his bonus and salary.


this is one of the funniest tech stories ive heard!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: