It's been years since I last did some serious email hosting, but it looks like the SPF rule is the problem.
Google is forwarding mail for dnalounge.com but the SPF rule doesn't allow Google's SMTP servers to do that: "v=spf1 a mx ptr ~all". That could explain why the email gets in the spam box; failing SFP should increase the "spam score".
Besides I think SOFTFAIL shouldn't be used in production; and I also doubt that Google should be taking seriously a SOFTFAIL anyway; so please take this comment with a pinch of salt.
SPF is easy to get wrong, and it always backfires at you ;)
"A@dnalounge.com is logged in to GMail Web Client as A@gmail.com and sends a message to B@dnalounge.com. Google's SMTP servers deliver that to cerebrum.dnalounge.com with an envelope sender of A@gmail.com. (THIS IS WRONG ON SO MANY LEVELS.) cerebrum turns around and bent-pipe forwards back to Google's SMTP servers, who determine that Google's SPF record doesn't list cerebrum as a designated sender for gmail.com (given the preserved envelope sender of A@gmail.com)."
In my >5 years experience running a mail server through which gmail users send authenticated mail, that is not accurate. To confirm that nothing has changed, I just looked at the mail logs, and in the last 24 hours I don't see a single instance of gmail doing what is described. In every single case, the envelope sender address is user@mydomain, not user@gmail.com.
Are the users' gmail accounts 'gmail.com' or 'mydomain'? Because I've sent mail from servers using a google mailbox to authenticate, and the from field was overwritten with the name of the mailbox used.
The users' gmail accounts are at user@gmail.com, but they're sending mail as user@mydomain, via mydomain's authenticated SMTP server. This is exactly the scenario described in the comment at jwz.org.
I read it as the opposite of jwz. You say your users are sending mail via "mydomain" SMTP. jwz says his users are sending mail "from their phone up to Google's SMTP server".
The goog then sends mail to the receiver at 'dnalounge' MX, where it's then shuffled back to the receiver's gmail mailbox. I don't read his setup as having the sender involved with the dnalounge SMTP server at all.
SOFTFAIL is only contributing to the anti-spam score (in theory) as it is supposed to allow the mail pass, but the host is still unauthorized [1].
Using "~all" means that you can only tell which hosts are definitely allowed to send mail for the domain, and you're unsure of anything else.
IMHO that reduces the effectivity of SPF. SOFTFAIL is useful as a debug method when you're testing rules and you don't want mail to be rejected by mistake; but I think it should be transitory and finally replaced by FAIL ("-all").
If you configure SPF to allow mail being delivered by Google's SMTP servers for that domain, you're again reducing its effectivity (Google's SMTP servers are used to send spam); but still better than a "SOFTFAIL all" I think :)
It's been some time since I ran my own mail server, but from what I recall, some spam filters actually give more negative weight (i.e. message is more likely to be spam) to a SOFTFAIL (~all) than a FAIL (-all), even though this contrary to the spec.
Spamassassin's rationale was that many of the tutorials online never explained the difference, so the majority of mailservers were just using SOFTFAIL everywhere. More paradoxically, messages that had a hard FAIL result were statistically more likely to be due to a misconfiguration, based on an empirical analysis.
Like I said though, it's been quite a while since I've had to deal with this. I'm not sure if it's still true or not.
Obviously good email (deep in a conversation) being declared "spam" has been a problem with my business for some time also.
I figure Google takes all email in a 2 step process (either intentional or accidental)
1: get a good fraction of the world on GMail
2: intermittently declare any non GMail mail spam (
greatly lowering the utility of non GMail).
This isn't about any mass mailings or even mailing lists. This is about directly replies from me to people I have been emailing for months all of a sudden ending up in their Google Spam folder. Combining that with Google's good reputation and it comes off like "my dog ate my reply" (even though the mail is in their spam folder, the point is it is easier to convince a Yahoo user something is wrong on their side, than to convince GMail user something is wrong on their side). And it isn't that I all of a sudden decide to include a lot of links, attachments or zip files.
And for the "I don't like conspiracy theories crowds." From the first this said "either intentional or accidental". It would be enough for this to be a fortuitous bug for the effect to hurt non GMail users disproportionately. Google could even fail a fraction of GMail and the overall image would still be "email is flakey in general, we'd better all switch to the service provider for safety."
I known spam filtering is hard- but GMail has some really strong signals that I would think could dominate here (like incoming email contains text unique to a recent outgoing email). Or email is from a sender you clearly have a relation with.
(note: "Google takes all email" is a possible outcome, not a plot/conspiracy. A bug like this on a system of this size can have a big impact. Of course that does mean somebody running a system of this size might have an extra responsibility to look out for such things.)
I've run my own mail server since before Google (yes, there was a time...), and I'm having similar problems. Right now I'm in the middle of a job hunt, and email from me (even to people who have already emailed me) is ending up in spam folders. It is infuriating.
It is also apparently unfixable. Regular people cannot inquire as to why mail is not getting through. And amazingly, even having friends at Google does me no good here. I'm told that the anti-spam team is so mysterious and opaque that they won't even talk to co-workers about this.
It's maddening. I get why Google grew up with absolutely no conception of customer service. (It's hard and expensive, and not necessary for their business model. Chicken farms don't have customer service windows for the chickens to lodge complaints.) But their apparent utter indifference to the problems they are creating for other people is maddening.
Do you have the various auth headers setup on for your mx records? I have to go through that ordeal occasionally when setting up outgoing emails for sendgrid or mailchimp campaigns. There are some things you can due to reduce the likely hood of mail being auto flagged as spam although in a lot of ways it is out of your hands.
I know this may sound extortionist, but one solution is to use an email-as-a-service, like Mailgun, Sendgrid, or Mandrill. All of these services offer the first few thousand emails free. They perform spam filtering on outgoing mail so they are trusted by Email Service Providers (ESPs).
Ideally, we'd all be able to send and receive email on our own servers, but due to the fact that the vast majority of email traffic is indeed spam, it is an unfortunate reality that the big ESPs have to be extra-cautious about it. Additionally, they need to make the decision whether email is spam or not instantaneously, so mining customers' previous emails is not an option without dramatically increasing the size of their infrastructure for this task.
It is possible to send email from your own server; you need to follow onerous industry-standard practices, like using a dedicated, known IP that has a certain amount of increasing email volume, known as "IP warming". After a few months of constant interaction with an ESP, your sender score improves. Unless you're sending massive amounts of email, the cost-benefit ratio isn't in your favor to use this method.
Having had delivery problems to gmail for years, I couldn't agree more.
However Yahoo are an order of magnitude worse when it comes to delivering email. If they decide you aren't their friend, they send back an error code with a URL in it to go and visit and identify yourself via a web form that is reviewed rarely by some humans. Then if you fuck that form up or if they don't like you, blackholed for 6 months and they stick their fingers in their ears. Try telling a yahoo user about this as well and it's "meh".
Email is broken unless you have a small provider or DIY and it's fucking frustrating.
I had that experience with Yahoo. I sent a single email to my brother whom has an account on Yahoo with SPF, DKIM, and not in any blackhole files. I received their 421 response "All messages from x.x.x.x will be permanently deferred; Retrying will NOT succeed." and my only recourse was to fill in a "bulk email prioritization form" I was not sending bulk email and was thus denied for that program (nobody read my explanation that I am NOT a bulk mailer). It took over a month of contacting various channels to get them to take my mail.
I understand spam is a problem but email is perhaps the last communication identity you can own, and have interoperate with others. I have had my own domain and own email for 16 years now, and I am not going to give that up. Free email services come and go, but I own my identity.
I think the problem with DIY/small business email is that it is surprisingly difficult to set up a proper mail server. Sendmail book is about 4 inches thick, and my current setup is using 3 projects to achieve a simple mail server with SSL auth/IMAP. (Dovecot, postfix, SASL). I am a software dev, but even very good sysadmins I know do not want to have anything to do with email anymore and will often farm it out to Google.
I really think this used to be a problem. Why anyone in their right mind would even consider using Sendmail these days, is beyond me. I use exim - because I've used at work, and it's default in Debian -- but I'd generally recommend either postfix, or probably openbsd's opensmtpd (it's in Debian unstable). It's not really that hard, AFAIK no sane distros will set up an open relay out of the box anymore, and that's really all there is to it. Assuming you're serving less than a thousand users or so (eg: one user ;-).
Might want to set up greylisting, though, to curb incoming spam.
Yahoo are an absolute pain in the ass. My last dealings with them were from shortly before Marissa took over, I don't know if they've improved or worsened.
What succeeded for one client was my posting pflogsumm stats showing delivery rates and times for Yahoo vs. numerous other large mail providers, along with proof of SPF and DKIM support, to all the Yahoo C-level (and several lower) execs, indicating that they had a problem, and I'd been attempting for months to resolve it.
Apparently there's a Yahoo "concierge", and we managed to get onto the company's approved senders list shortly after.
That said: email's a pain in the ass, and I do know for a fact that that particular client was failing to scrub known dead recipients (as in: the domains no longer existed). Some fights you just concede....
When I was being recruited by Google I was frustrated by having great phone calls with recruiters, doing well in my phone interview and never hearing a followup.
Turned out all the emails from @google.com were going into my Gmail spam. I think they still do (no matter how many I've marked as not spam).
I've been waiting for an email from a company that I'm in the recruitment process with and your comment inspired me to check my spam folder. If it wasn't for that, I wouldn't have seen an email from a government agency looking to recruit me. So thanks!
"This is about directly replies from me to people I have been emailing for months all of a sudden ending up in their Google Spam folder"
Yup same exact thing. Just the other day I sent my wife a few links from my business email account @company to her @gmail and it ended up in her spam folder. And I have been mailing her for longer than a few months as well.
In other words google does not differentiate it appears on whether you have an ongoing relationship (email wise) with the person that you are sending an email to as if they are just looking at the mail message and nothing else.
For some reason, gmail classifies nearly all of the transactional email I get from Amazon (order acknowledgements, shipping notifications, etc) as spam, and no matter how many I catch and un-mark, it won't stop.
Spam classification is a hard problem, but I don't think they've put the false-positive/false-negative tradeoff in the right place, and they really need some safety valves like offering to whitelist an address when you un-mark it as spam.
It's as if they intentionally want to force their filter on you. Not that there's anything wrong with a server-side spam filter. One that puts a polite X-Spam-Score header so you can adjust the sensitivity client-side.
Looking at the contents of my spam folder, and knowing that they likely reject many more e-mails in the background, I'm pretty sure I'd drown if I ever tried to use such option...
Why are you doing this? If you subscribed to something, do the responsible thing and unsubscribe from it. Don't fuck up spam filtering just to be lazy.
I get a ton of unsolicited email from Amazon. It tends to be of the form "order confirmation for product XXXXX" or "your order of XXXXX has shipped". I don't view this as particularly unreasonable, though.
No, emails that you ask to receive are the definition of solicited email. When was the last time the grocery store emailed you confirming that you'd just paid for your groceries?
The last time I had Facebook emails before I marked Facebook as spam every action which somehow concerned me on Facebook resulted in me getting a email and consequently flooding my inbox. Admittantly this was a few years ago when I finaly had enough of it so they might of gotten better but at one point they did.
Yes, I picked facebook as an example specifically because of their reputation for emailing you about everything. But I think you'll find that they tend to email you when other people do something that, in some sense, concerns you, not when you do something. (Remember the ancestor comment defining "solicited email" as email that responds to an action you take.) "This person just liked your comment" emails are plausible. "You just liked this person's comment" emails are self-evidently absurd, but they fall perfectly within the (spurious) definition of "solicited" that nitrogen wants us to believe in.
To all the people that say "don't do this" - I don't do this with Amazon, but I do that with LinkedIn and any company that starts to, well, spam me with e-mails. And I think it's totally fair. It's a way for people to send a message to the companies that don't respect their customers time, because some bean counter realized that they can wast X hours of people's time and get Y amount of additional sales from it. I'm sure that without such feedback, we'd get spammed by useless newsletters even more.
(And keep this discussion in mind next time you read a HN story teaching people how cool e-mail marketing is for your startup - what it's cool at is often pissing off your users.)
Also Google took a good approach to that issue - when an e-mail has an unsubscribe link and you try to mark it as spam, you get asked if you'd like to unsubscribe instead. And yes, I sometimes do exactly that.
This is a difficult balancing act, to be sure. I firmly believe two things would help:
1) Unsubscribe should be one click, no confirmation. In the case of accidentally unsubscribing, if it's really that valuable, the person will re-subscribe, and I guarantee it's easy to subscribe because that's already optimized, for obvious reasons.
2) GMail shouldn't give you the option to choose; if you mark something as spam, it should silently attempt to unsubscribe in the background, then keep track of which services don't actually support unsubscribe, and mark those as spam. Far too many people are vindictive and lazy and will intentionally mark things as spam either because they don't want to go through the process of unsubscribing, or they forgot they subscribed, or think they've been "wronged" by getting an email they opted in to.
For the record, I too despise the oblivious email marketing stories on HN, eg the "cold emailing" C level execs whose addresses aren't publicly available. Dirty, dirty, dirty.
If so, it seems like an uncharacteristically clunky way of doing it -- sure, clickthrough rate has a some effect on PageRank, for example, but there's a ton of content-based filtering going on too. I don't really want my spam folder full of false-positives because folks don't want to unsubscribe from things.
Don't do this. Seriously, don't. It's one thing if you never asked to get additional emails or you can't opt out, but if you signed up for it, at least put in the same amount of effort to get out of it first. If that fails, sure, then start marking as spam. Otherwise, you're feeding false positives into the system, thereby degrading it on purpose.
And that's why I said: "but if you signed up for it, at least put in the same amount of effort to get out of it first."
If you signed up, then you opted in. Don't be lazy and mark things you signed up for as spam. I'm not understanding how people get email from Amazon that aren't customers, and how customers can't turn off everything but confirmations, which I know is possible, because I have.
> how customers can't turn off everything but confirmations
I'm a customer, I consider that opt out. You don't? I signed up for amazon, not their bullshit emails. That you consider the situation opt in shows they have you trained well.
Nice, ad hom, and I can't downvote your insulting comment! How about this: are you too lazy to uncheck a few check boxes? Because that's all it takes, and then you'll only get order and shipping confirmations.
Most Amazon email is spam. They send tons of promotional email to everybody that buys something there, with not even an opt-out option, and after you receive the emails, go to the opt-out link and uncheck it, they create new categories with you signed in by default. The GP is correct in marking amazon emails as spam, and I'll ask anybody that's considering stopping to keep marking it.
That said, this is no excuse for Google to mark their transactional emails as spam for people that clearly want to receive them. Google is just being lazy and doing a bad job. Even Yahoo gets that right, Google can do it too.
Speaking as a person who frequently buys from Amazon, I've never experienced this. (No, it's not that promotional email from Amazon is going into my spam folder. I have to check my spam folder semi-regularly because gmail keeps flagging perfectly legitimate email as spam.) The mail I get from Amazon is order confirmations and a newsletter that I specifically signed up for. What's this promotional email that you're talking about?
I can't help but notice that all of those come from amazon.co.uk. As evidence-free speculation, perhaps there's a problem at a lower level than "Amazon the company". This sort of thing would certainly help explain why some people are extremely upset about spam from "Amazon" and other people are taking the position "you're crazy".
You signed up to be their customer, and you can turn these off. That they do this by default isn't ideal, but it's not spam - you have a business relationship with them.
just because i signed up for amazon.com, doesn't mean they get to spam me crap because they've buried a 2pt font unsubscribe feature that may or may not work over the course of a few weeks.
I call bullshit. I'm a very regular Amazon customer, and I neverever receive anything but order and shipping confirmations from them (and the occasional interview offer). Can't remember if I did anything special (like uncheck a few boxes), but I'm very sure it's possible to not be spammed by them as my inbox is proof.
This seems more like a bug than something intentional. What I find more interesting (and am somewhat saddened by) is that a bug such as this causes people to leap to the conclusion that it is part of a malicious, anti-competitive plot.
Sadly, it doesn't have to be malicious to be anti-competitive. Back in AT&T's monopoly days they were at their most destructive not when they were scheming, but when their size let them not give a fuck what they were doing to other people as long as it was good for them.
It sounds more like a side effect than a bug. Both domains have conflicting policies. He wants any Gmail user to be able to impersonate a sender from his domain, but not in a way Gmail supports.
Are you asserting that Google has not engaged in anti-competitive behavior or that they should be given the benefit of the doubt despite previously engaging in anti-competitive behavior?
Agree. It is time to start boycotting Google ecosystem until it is too late. It is definitely turning into a corporation of evil. I enjoy their search but definitely despise how they spread their proprietary tentacles everywhere. Sad.
Which ironically is the reason I most often hear from Gmail advocates/apologists for why they advocate Gmail over running your own e-mail server.
I don't disagree. E-mail is a mess. Still, on the whole, for any person or organization with modest technical chops, I feel the freedom and decentralization (which is a positive for the entire ecosystem) is reason alone for retaining control by self-hosting.
It's all about cost/benefit analysis. Maintaining an e-mail server and not getting overwhelmed by spam is hard; making sure you your e-mails will be received is even harder. Most of small businesses or organizations simply don't have resources to do that, if you count in the value lost when business e-mails get flagged as spam.
Decentralization is cool and all, but until we have some kind of self-contained, easy to deploy and maintain solution for e-mail servers, it's hard to me to recommend anything other than GMail to people - especially that typical small company or organization has a lifespan that's shorter than Google, so they don't get any important benefit from decentralization.
I haven't checked the new Outlook recently; maybe it's good. But all others that I know offer subpar service and honestly, look like crap. UX is important. For example, I cringe whenever someone makes me to use Roundcube-based webmail.
Yeah Roundcube's look definitely needs some help. Last year I did choose Roundcube for our webmail, but I spent several days altering its appearance so that it would look semi-professional.
I started with their new theme called "Larry" and then corrected the fonts, colors, graphics, and certain element sizing decisions.
Google has attempted to kill open protocols like xmpp and RSS in order to lock people into their services. Why is it so crazy to think they could be trying with GMail?
Most of the spam I get through to my Gmail inbox is from SEO spammers using Gmail. They keep sending the same material with the same username formats and same names, I mark hundreds of them as spam, and they still make it through.
Another large portion of my spam is from Chinese product spammers using Picasa albums and send to a friend feature. There are limited ways you can report this and despite marking almost every Picasa email I've received as spam, every single Picasa email still makes it through.
Already downvoted below zero (about 2 seconds). I'll stand by what I said/implied: it is hard to co-exist with Google systems (and apparently also with pro-Google shills/apologists). I know this post violates HN standards, but I didn't think the parent did.
I think its just that everyone who doesn't run a mail server hasn't had to deal with this shit so finds it hard to believe that their precious email providers are being a pain in the butt.
That is until you get a court summons after an invoice was sent several times and never got there because some fuckwit mail policy just blackholed it. This stuff does happen and when it does its costly.
It's easy to see how transactional email providers have made money these days. You pay them for 70% of the brain damage, then take on the other 30% for your own.
I think there needs to be a more substantial karma gate to downvoting (10 000+), I'd be fine with falling out of that category myself. I've only been interacting on HN for the past year or so, but lurked for several years prior: the community has been becoming a lot more "redditish."
I nearly always agree or can be convinced by the higher karma users, but the fresh users seem to be defaulting to reddit-level arguments (uneducated, immutable beliefs, hyperbole, taking things personally, no objectivity). I.e.
> just that everyone who doesn't run a mail server hasn't had to deal with this shit
"Everyone" in that sentence should not have the right to be downvoting in the first place.
For the record, I downvoted you for bringing up the "HN is becoming Reddit" meme, which is something I've seen popping up here all too often for the last 4.6 years I'm participating in HN community (and from what I heard, it happened long before then too) - and yet for all those years of "becoming Reddit" HN still did not become it.
Also, Reddit is cool, and there's ton of quality discussions - usually under particular subreddits. For example, I love /r/KerbalSpaceProgram for a perfect combination of being large, friendly, fun, and full of people willing to teach each other some more advanced science.
To address your point: I think current downvote gate is fine and adjusting it up won't really help much. What would help (though I don't know how to achieve that) is people realizing that when saying anything controversial, you might end up getting a bunch of downvotes immediately, but the score usually settles to a reasonable value within an hour or two. It takes time, but one just has to get used to it.
TL;DR: karma comes and goes; look at it through a low-pass filter, to filter small variations.
I've found that this community is beyond unwelcoming, bordering on toxic. I no longer see the merit in contributing, and furthermore I find that reading these comment threads has become less and less valuable.
I find hn a really to be a really good combination of intelligent, grown up discussion with little name-calling etc.
That said, I think on hn one must be prepared to be corrected, -that's just part of being scientifically minded. As the old saying goes: "Iron sharpens iron, and one man sharpens another."
Yeah that's the consensus, and that's what it was meant to be, that's what it once was. However these days, especially in a community where only certain individuals can downvote, they become the only ones to control the discussion. What was supposed to encourage a higher level of discussion has instead created a community that is the exact opposite.
Who cares about my stupid comments, sure, I'm an idiot- I'll agree to that. Unfortunately, I see it happen all the time. The parent of this, for example, was downvoted immediately.
Everyone in this community is prepared to be corrected- the problem is when there's no correction offered. It's a community of suppression, and it's not hard to see.
I agree that there is too many incorrect downvotes given.
I don't think it is a big problem though only that it has been slightly increasing.
(one comment you'd see back in the days when comment scores where displayed was: "sorry for the downvote, -reading hn from a mobile device ". You don't see this comment anymore but I'm not sure if that is because it was easier to detect accidental downvotes back then or because people where more polite.)
There is plenty of disagreement among people who can downvote. And everyone can upvote so bad downvotes should be corrected rapidly by the rest of the community.
Perhaps HN users need to be reminded to upvote things that have been unfairly downvoted? (Although I've seen plenty of examples of corrective upvotes supplied pretty quickly).
Personally speaking I downvoted your comment because I often downvote things that smack of conspiracy thinking - never ascribe to malice that which can be explained by stupidity.
Another issue I have with your argument - I don't believe Gmail has the market share to make such a fiendish strategy viable.
You did say that, but it felt like a disclaimer rather than something you genuinely considered to be a possibility.
I think, right or wrong, you'd probably get a better result if the rage in your text was clearly directed at the consequences rather than the motives - if nothing else because, as you say, the motives don't actually change the consequences and it's the consequences you, I, and everybody else who likes having their own email setup has to suffer.
(to be clear, I'm not commenting on whether or not you should have been downvoted, only on my best guess as to how to get across the same information without it happening)
So andybak, congratulations on clinging to a sweet sounding one-liner, but let's think a little more about it. So anytime something bad happens to a person (their wallet or phone is stolen, or their family dog is killed), and the victim believes the perpetrator acted maliciously, would you tell them that "smacks of conspiracy thinking" (whatever that means to you)?
Another educational moment for you would be to look up the word "conspiracy" to learn what it means. I understand that you were hoping to use it as a dismissive term, but you would have been better off using a universally dismissive term like "crazy", as there are a lot of rational people that understand the factual meaning of "conspiracy" and don't interpret it in the same way you do.
You and I had similar difficulty yesterday vis-a-vis arbitrary downvoting. I was referring to a comment made at me regarding how HN comments are for correcting people who are wrong, and accompanying that is downvotes. Except for when "correction" has nothing to do with the downvote, or none is offered.
It doesn't matter. The conversation at HN is toxic. I've bitched enough. I was just reaching out for commiseration.
This is not a general issue with delivering mail to Gmail users. This is not even a mailing list. He is using Gmail as a backend to his own mail setup.
His employees want to receive mail sent to @dnalounge.com in their Gmail inboxes. It's a non-standard configuration and brittle. You can't really expect them to support it for free.
Google offers a couple of supported ways to do it. Most prominently: pay for Google Apps. But you can also use Gmail like a regular mail client and retrieve mail using POP3.
"He is using Gmail as a backend to his own mail setup"
No, he is using a normal forward for those who want to receive their mail at gmail. He is not using gmail as his backend. You should read the original article.
I did read the article and that's exactly what he's doing. He forwards everything to Gmail, and then allows users to send from Gmail via his SMTP server (or maybe doing some spoofing but it wouldn't look so clean). Essentially outsourcing hosting of inboxes (but just inboxes) to Gmail.
I run my own mail server, but most of my employees use Gmail. So I have forwarding set up: employee@dnalounge.com simply forwards to employee@gmail.com. When sending mail using their Gmail account, they set their From line to employee@dnalounge.com. (Google lets you do this if you jump through some hoops to verify that you can actually receive mail at that account.)
And that is the description for a fairly normal "forward all my mail to my personal inbox" scenario. Its not rocket science and it is not using gmail as his backend. We do it with part-time faculty.
Those are Gmail accounts belonging to his employees but he is using them in the back of his mail setup instead of hosting @dnalounge.com inboxes himself (or paying Google to).
He doesn't use them. His employees chose to forward their email because they want to use the same interface for both their work and personal email. He does host @dnalounge.com inboxes.
I've long supported a proof-of-work concept augmented with whitelisted keys for email spam filtering.
There are basically two cases here:
1. The sender is sending one email to one receiver. The sender computes the proof of work and sends it along with the message. This takes some time, but typically it can be done in the background. Waiting few seconds between sending and receiving an email is typically not an issue, and in fact many email clients wait for some time before sending anyway to allow "undo send" functionality. Rather that doing this computation on the mail server, it should be done on the sender's client, so that the server doesn't get overloaded with proof-of-work computations.
2. The sender is sending many emails to many receivers (a mailing list). In this case when the receiver signs up for the mailing list, the sender sends a request for a whitelist token to the receiver's mail server. IF the user accepts the request, the receiver's mail server returns the token and then the sender sends all further communication with the token instead of with a proof-of-work. This solves a few problems: a) Receivers opt in rather than opting out of receiving communications. The ubiquitous pre-checked "send me spam" checkbox loses its effectiveness. b) Receivers can revoke tokens at any time. c) Senders who are sending large amounts of legitimate mail don't have to compute a proof-of-work for every email they send.
Receivers simply drop email which doesn't come with either a proof-of-work or a whitelist token. This drives up the cost of sending large numbers of spam emails because each spam email requires a large amount of computation. And even in cases where a spammer has a large amount of computation at their disposal (botnets are a common case) it makes it easier for servers to distinguish between mailing lists and spam: a large number of identical emails could be either, but mailing list mailings should come with a whitelist token.
Doing things this way would mean we can drop these terrible DKIM and SPF systems that both fail to prevent spam and make it difficult to send legitimate mail.
> Doing things this way would mean we can drop these terrible DKIM and SPF systems that both fail to prevent spam and make it difficult to send legitimate mail.
I can understand your feelings about SPF, but why is DKIM a terrible system in your opinion?
Because we still have spam and it's difficult to send legitimate email. It doesn't achieve its goals and gets in the way of achieving other core goals.
I have been running my own SMTP server for about 25 years. I finally decided, in the last month or so, that it just wasn't worth the effort involved in handling spam and anti-spam stuff -- not only from Google, but from other providers who are so worried about spam that they falsely tag all sorts of stuff as spam. I found that accurate configuration and maintenance, including of SPF and DKIM, wasn't worth my time and effort.
And so, I have now moved my e-mail to Rackspace. I had to use their chat-based support several times while migrating my e-mail to the new server, and it was truly fantastic. On that point alone, I feel fortunate.
The $20/month I'm paying Rackspace is a pittance compared to the time and effort I was spending trying to keep my old SMTP/IMAP servers secure, as well as the false-spam tagging that happened all-too-often.
It sounds like jwz wants to have his cake and eat it too, and I sympathize. But I'm not sure that it's possible any more to spend a non-trivial amount of time configuring e-mail servers, tinkering with them such that they'll work with big companies (and especially Google). The Internet is no longer the simple, fun playground that we old-timers remember, and that effectively means giving control over some services to people who are paid full-time salaries to take care of these inter-connectivity issues.
For me it luckily works – I only host a dozen of users on my server, but it works nicely, and – after having spent a few months setting it up once – also runs nicely with Google and the other large providers, even Yahoo and Hotmail (for now).
Gmail accounts created on or about June 2014 (exact date unknown, Google only mentions "second half of 2014" in their new authentication blog post) won't work with Thunderbird until the Thunderbird team implements Google's non-email-standard authentication. This is currently scheduled for Thunderbird 38 which will be released on April 7, 2015. See bug 849540 for the full technical details
I honestly can't figure out what he's attempting to do here.
It sounds like he doesn't want to force a work email account on his employees. I don't think many people would have a problem with a work email, it's the norm after all, but fine.
But he also doesn't want to add their personal email addresses to his address book (is this why he wants the @dnalounge.com addresses?)
I'm not sure what this setup is attempting to accomplish.
He wants to enforce a @dnalounge.com for all internal emails rule, but have the underlying tech be more flexible.
Actually, makes sense, because he can then save those emails on the server, back them up, and they'll be in his control if an employee ever leaves. It's good corporate policy. I believe the White House is having a similar conversation lately.
He does create a work email account for his employees. He doesn't want to force them to use a separate email client so he allows his employees to set up forwarding from his server to their personal gmail account. They can use his smtp server from gmail for sending stuff out so they get to use both their personal and work email from the same interface.
I've never used an email client that didn't support multiple accounts.
So again, I'm not sure what problem he's trying to solve here. To be clear I'm not saying he shouldn't be able to set up his email this way and have it work, just that I don't get why he WANTS to do it this way. What is it accomplishing?
gmail.com supports fetching email from external servers via POP (it's under settings, accounts). His users could use that and skip most of gmail's spam filtering completely. It doesn't (reliably) support forwarding mail to it without using some list software in the middle. I believe if jwz set up mailman or something similar, it wouldn't have a problem with that, because mailman would properly re-write the envelope sender as the list address.
With office 365 and hotmail email just goes into the abyss. We have SPF and DKIM yet some mail just never arrives when sent to hotmail. The server responds with "250 queued for delivery" but the mail never arrives and doesn't go into spam either.
Very annoying when order confirmations don't arrive.
Google has always had issues with legitimate e-mail the spambox, but then again, I have yet to see a spam filter with no false positives. Google just errors a lot more on the side of classifying email as spam compared to most email services (more false positives, but less spam in the inbox). What makes this even more annoying is that they hide the spam folder so that users don't even know that it exists anymore. I always have to give directions where to find it and invariably get reactions like "Oh, I didn't know GMail had a spam box!" or "Ah thanks, I would never have found that."
I did consider setting up a route in my mailserver to GMail using my Google account credentials for their SMTP server, but then decided they should better get their own shit straight. It wouldn't scale to do this for everyone.
I wouldn't have such a problem with its false positive rate if they at least learned from them better.
It doesn't seem to matter how many times I mark an email as "Not Spam". The next day an email from the same sender from the same server with the same Received: path will land right back in Spam again.
I realize that gmail can't immediately globally unblock based on "Not Spam" reports (otherwise spammers would ruin things by marking their own bulk mail) but could they at least apply them to my mailbox?
I guess adding the senders to my address book would help, but it seems like a silly thing to have to do. I already told it that a message is "Not spam", can't it take it from there?
The problem now is that GMail is so big, it becomes "well your emails are being filtered in GMail so you need to fix it". With smaller providers it would be that they were being too aggressive with email filtering. Fixing the problem is sometimes very hard with large providers as you really have nobody to contact for help or whitelisting. You can do all the best practices for sending emails (SPF, DKIM etc) but that will not help if you get filtered or blocked for some arbitrary and unknown reason.
So for this reason people give up and either send their emails through well known mail senders or switch their users to services like GMail. This is sad as perhaps the last communications medium with open standards is being forced proprietary due to the difficulty in dealing with the large email providers. My company switched to GMail for our email, and Mandrill for sending out our mailing list although we have competent sysadmins.
>so more false positives, but less spam in the inbox
This is a pretty strong understatement. The spam filter in gmail is incredibly aggressive. My secondary email is a gmail account and anytime I have a registration email from a service or anything with a couple URLs in it, it goes straight to spam. Meanwhile, the product we use at work is tons smarter, but gives the occasional spam into the inbox.
I guess someone at google decided that people seem happier when there's no spam in their inbox. People seem to drop 20 IQ points when they see spam. They either instantly rage about it or send the email to the helpdesk demanding an explanation on how someone dared email them something they weren't interested in. Its a bizarre behavior. Yet, when they're forced to fish through a spam folder or an anti-spam web interface, they seem to mind that less. Which is bizarre as its tons more work.
I think the guys at gmail realized this and just erred on the side of caution by giving people as much as a clean inbox as they can handle. Even if they don't like it, who are they going to complain to? Google? Who at google gives two shits and even if someone did, how could your reach them, and if you did who has the power to change policy? Meanwhile if you run mail servers, you get complaints about how your email goes into spam, even when you have all your DNS ducks in a row and have a good reputation. When you tell gmail users to talk to google about it, they scoff and tell you to piss off.
Gmail is awful regarding false positives. People discount it because: a) it's a free service; b) false negatives are more frequent and thus more annoying.
What is difficult in spam classification is having no false negatives and no false positives. Any decent CS graduate can create a spam classification system with no false negatives given the leeway to introduce false positives. That's what Gmail did.
> b) false negatives are more frequent and thus more annoying.
I think this is the case for the user. For the sender it's a major issue and a huge annoyance, but you don't measure user satisfaction by asking senders.
I don't know that I've ever had a false negative (spam in my inbox) in my gmail, and I've had it for just over 7 years now. I also don't know that I've ever had a false positive, but I tend to not check my spam folder very often so it's possible I've missed things.
Maybe I'm just an outlier, but I'm perfectly content with my gmail.
Gmail will dump emails from google.com into my spam folder if I have them forwarded from another gmail account. It is infuriating that they can't even solve this problem within their own domain.
Example of a message from Google, to a custom domain gmail account, forwarded to another custom domain gmail account — found in my spam folder:
The first thing to check is whether the DKIM and SPF checks are actually passing. Email forwarding tends to break them, and if that's what's happening then there are specific things you can do depending on what exactly is breaking.
Gmail is horrible in so many ways. Complete disregard for the MIME structure of the emails. All parts that aren't text/plain or text/html gets bumped to the bottom, even if they are clearly declared as inline. And then there's the infernal top posting they force on you. To get to one attachment I had to page through a wall of unsnipped irrelevant garbage, if they want to force an obnoxious posting style, at least put the attachments right below and together with the top-level message, not all the way at the bottom.
These guys.. is it just me, that think that Google miss some serious traction? Fundamental things like Email Forwarding not working now? Is everything just hit and miss there at Google HQ?
One guy out of millions having problems forwarding email is indicative of everything being hit and miss across all Google projects? You and the OP might be jumping to some unjustified conclusions.
This was happening to me as well. I was trying to get an email from a known individual, but after being sent 4-7 emails, none came through.
I checked the email service which the emails were being sent to and they were no longer there (forwarded already), and also not in my gmail inbox yet.
They're still no where to be found, might be in limbo, but email forwarding definitely took a hit yesterday
I run my own mail server, and I've found it to be pretty much pain free. Of course I'm the only one who uses it, and I don't do fancy stuff, but after a few bumps when starting out, it works great. I've only had to fix things occasionally, like heartbleed. On the whole, it's been really nice. Is my experience abnormal?
I've had the same experience running my own pair of iRedMail servers in FreeBSD jails. It was easy to set up, and not hard to secure. I haven't had trouble with spam so far, and it has allowed me to migrate off of gmail. Setting up failover took a little work, though.
I was looking into using email for rich content delivery yesterday as part of this "Watch me program" thing I'm doing. How difficult could it be to send rich html to somebody on email?
Turns out, freaking difficult. For one of the simplest protocols in the world, email is now a byzantine disaster. Different clients render things differently, different providers block or don't block things -- and that's not even getting into trusted domains and spam issues, which you end up depending on third parties to assist.
Add into this mix Google's notorious black-box/impenetrability/lost-in-space-customer-support? I don't see an optimistic future for you.
I'd rather go back to hand-coding COM in C++ that become a network email engineer. That's got to be a brutal job. You're dependent on so many moving pieces and you control so very little.
Do you think the strict restrictions reduce spam, and phishing attacks, and email client hacks? For example, many clients dont let you change the default colorbof a hyperlink.
I think you'd be better off with an open standard, rich, dynamic html that runs wherever the user is. Then put all the sophistication in whitelisting senders. We also need to bolt-in secured messaging so that nobody except the people corresponding have any idea what's in the message.
The way it's set up now, with hundreds of vendors implementing different control protocols and tens of thousands of spammers trying to break in? I don't see anybody being happy -- some spammers get in anyway, and email users have different experiences for the same dang net service. Admins want to pull their hair out. It really is a miracle things work as well as they do.
If we can't get email working right? It doesn't portend for a bright future for the rest of internet traffic.
I've been blaming my hosting company, getting everyone in their support chain to take a crack at the issue. I always just assumed that gmail was the working side of the problem, not the problem itself.
I wonder if it is related to a change I recently noticed in how email address suggestion and autocomplete, i.e., suggestions, work that behaves differently than before. It's almost like they deemphasized frequent user or maybe domain accounts or something. The effect is that the first suggestion is no the address you are most likely looking to use, and thus risking erroneously addressed emails. So take care.
I've got email forwarding set up for a domain on 1&1.co.uk and don't have any problems on gmail. Others who have mail forwarded from the domain don't have any problems on hotmail or yahoo either.
Maybe Google gives extra trust to 1&1; or maybe there is some other thing that they check for.
And to make it worst: Google don't run filters on messages that goes to the spam folders, so you can't even use that.
I always wanted to do the opposite: Run a filter on the spam folder to delete emails that I know for sure that really are spam, so I wouldn't need to delete them manually.
I'm not sure what you mean; do you mean that their search bar doesn't include spam filters? (I assume that's what you mean since filter creation is sort of built into the search bar.)
And, well, I am sympathetic to the argument that Google shouldn't do things the wrong way. But you really have no leverage to make them. Google offers the ability to use Gmail for e-mail sent to your domain, that's Google Apps For Business (or I guess Google Apps For Work). If that's unacceptable, you either have to deal with Google not caring about your needs (unless you're paying them, you're not the customer anyway) or do without Google.
Indeed. Email forwarding has been broken for years. I don't allow it on any MX I run. Even with the best defenses, it's too risky to allow users who are spam magnets to harm your server's reputation.
Anyone else wondering why JWZ feels the need to complain when he is clearly using Gmail in a way that Google advises you not to?
If he wants to use Gmail for his business he needs to get Google Apps. Having your employees use their personal e-mail accounts for business is completely unprofessional.
Why even bother running an in-house mail server if all it does is forward e-mails? Cut the costs and pay for the services Google offers for business and get your employees more focused by keeping them off their personal e-mail on the job.
I'm late to this thread, but hopefully I can add some value.
I've recently been through this for my own personal email. I have a domain with some interesting rewrites to the address that I forward to GMail. I put together my set up into a Docker container:
It is worth breaking this down into each direction -- sending and receiving.
Sending
This is the easy side. You simply run an SMTP server and have GMail use this to send. If you want others to not mark you as spam you'll want to look at setting up SPF (publishing which IPs are allowed to send from your domain) and perhaps DKIM (digitally sign your email).
GMail used to allow sending from aliases directly without an external SMTP server but that is disabled. Old accounts are grandfathered in.
Receiving
This is where things get complicated. Essentially, you are having folks send mail to your server and you are turning around and relaying that email to GMail. GMail has a hard job here. It doesn't know if it should trust you and that you are acting on behalf of the user or if you are an open relay sending spam. I've found after I change my forwarding set up I have to police my spam folder for a couple of days to retrain GMail.
The real complication here is SPF. Sending domains will publish their own SPF records. When doing this they can either specify a soft or hard fail. If they specify a soft fail (`~all`) then there is a chance that GMail won't mark it as spam. But if they use soft fail (`-all`) then it will go to spam all the time. The problem is that the sending domain (say evite.com) doesn't list your relay IP as having permission to send that mail and so GMail respects that.
Having GMail pull the mail via POP is one solution here but that introduces latency.
Or you can rewrite the "envelope sender" so that you are honest about the mail coming through your server. The accepted scheme to do this is SRS. This is not a silver bullet though. If you are forwarding a lot of spam, GMail may decide that your SRS domain is spammy and penalize all incoming mail.
Also, if you are forwarding a lot of spam, GMail will throttle you. It'll have you back off and wait to forward more mail. Your best bet is to find ways to eliminate obvious spam before you forward to GMail.
My solution seems to be working okay for now, but it is a pain the ass and I'm honestly not sure it is worth it.
1: email is hosted offsite, yet reliant on Google/Gmail to do in house work/intermail.
2: when you are reliant on an external source to solve a problem within your own house you are not prepared.
3: solution: reduce reliance or point of failure by either bringing it all in house and forwarding the remains or bring root to the source and source from root and use mask and forwarding.
I'm confused -- are you saying doing your own email hosting is the way to avoid getting stung by Gmail spam filters? Seems like an "out of the pot and into the fire" kind of situation.
confusion is the root of the problem, either remove the offending host and do it all in house or move yourself within the root of the problem and continue conducting business.
he could bounce off another provider/server so he can still maintain what he is doing now - he just has not explored any other solution except blaming. the arguement "it should just work" is not vaild if you are reliant on another service.
I get why it's hard to trust mail from some random IP from Amazon Web Services, but I'm very wary of taking big chunks of the internet and just saying, "well, people aren't allowed to do that anymore, just the huge corporations." One of the best parts of the web is that it's decentralized.
Google is forwarding mail for dnalounge.com but the SPF rule doesn't allow Google's SMTP servers to do that: "v=spf1 a mx ptr ~all". That could explain why the email gets in the spam box; failing SFP should increase the "spam score".
Besides I think SOFTFAIL shouldn't be used in production; and I also doubt that Google should be taking seriously a SOFTFAIL anyway; so please take this comment with a pinch of salt.
SPF is easy to get wrong, and it always backfires at you ;)
EDIT: seems that I may be right, according to this comment http://www.jwz.org/blog/2015/03/google-seems-to-have-broken-...