Then you realize the true pain of having a near-zero reputation when trying to email anything to people on Gmail, Yahoo, Live, etc, etc. Expect to go to spam even if you have DKIM, SPF, and no relaying. If anyone actually knows the secret to not having your own mail-server's mail go to spam on these bigger systems, please tell, I've Googled it for years with no success.
There is also no need to perpetuate the fear of running your own email. Decentralization of core internet services helps us all, not just the person doing it. So to anyone seriously interested and willing to learn, I highly recommend running your own email servers.
If you expect 100% delivery rate, you won't necessarily get that, no matter who is running your email infrastructure. At multiple companies, I've seen email end up in spam even though it is sent from gmail-hosted company account to gmail-hosted company account. So that's the bar you want to meet or beat and beating it is not hard.
From ~7 years experience now, I don't observe any problems with email delivery. I don't do anything special. I use postfix, it is well configured and everything works fine. I host several personal domains and a couple small business domains.
And by doing so I avoid giving google the power to cut me off from email because some arbitrary ML gone bad.
Heck I have seen emails from Google itself, not any gmail username, but information from @google addresses landed in my Spam emails, making me double check the headers that if it is spoofed or real.
I think that a good thing, since it mean they're committed to build a good algorithm, instead of just whitelist some big guy and throw everyone else to the trash.
Agreed! I've been running my own mail server for 20+ years, and I would not recommend that anybody do this on a lark. Deliverability to GMail is a huge problem for me and has been for years. I even asked friends at Google to find out what the story was. All I got back is that the mail group was so careful/paranoid it wouldn't talk to them either.
I would long ago have switched over to a vendor, but I use qmail-style tagging for sub-addresses (that is, instead of user+sub@domain, I use user-sub@domain). Almost nobody supports that. Especially not GMail (where my friends could tell me that was tagged as WONTFIX).
If anybody has solved this, please do let me know (on here, via Twitter, or the email address in my bio). It's maddening.
Funny. I'm somewhat new to the game but my email server has had very few problems delivering anywhere (hosting a small business' email, and personal email for the employees.) I've got spf, dkim, and dmarc. I use tls on everything. Got a static ip (linode) with rdns. I've never had a problem delivering to gmail. For a while i had problems delivering to Hotmail/outlook, but participating in some sort of junk mail reporting program (JMRP i think it was) fixed that issue. Now, if i have deliverability problems, it's cause something broke, or my tls cert expired, or i added a new ip that the email server decided to start using. (Sorry for formatting, on mobile)
Having run my own server for 20 years, my experience is also that it is relatively hassle free. I also run my own DNS and have a static IP, which might help with keeping email related DNS records up-to-date.
Running the server is generally "set and forget". Every few years an issue might pop up that requires attention. These issues are generally due to a tightening of other servers' requirements rather than an actual technical issue. When such things do occur, symptoms are an occasional email rejection, and a bit of digging reveals the cause and fixing the cause returns things to normal. I've never had a wholesale rejection of mail from my server
For example, when SPF and DKIM came in, I had to add those records to my DNS. When Let's Encrypt came on-line I proactively added TLS to the server. A number of years ago lack of a Reverse DNS record got me on a blacklist for a short time. That was fixed by contacting the IP address issuer (my ISP) and getting them to add a reverse DNS record. Just make sure the reverse DNS hostname matches the hostname that your email server uses in its HELO messages. A week or so later I was automatically off the blacklist and the few rejections went away. I've never bothered with DMARC. This is the sum total of my experience with running the server.
At times there have been physical problems with the server or network outages with my ISP, but I discount these on the basis that it is my decision to run a server on a desktop PC in my house rather than in a data centre. Easily fixed if I wanted to throw money at it.
Setting everything up in the first place was a massive pain in the arse, definitely not for the feint hearted (I imagine there are tools/scripts to make it much easier nowadays), but it's been almost plain sailing since.
It's very rare that I have delivery problems, maybe once every year or two, which is roughly the same as through my O365 mailboxes!
Only issue is when there is a delivery problem, there is usually nothing you can do. Some of the block lists have a procedure for removal, many don't.
But after running my own mail server for something like 15 years, I've decided I just don't want the hassle any more (even if it isn't much work), and plan to move everything to O365.
That's been my experience. I've been running a server since 2012. I've had more problem with incoming mail - specifically being unable to whitelist certain Yahoo servers which get killed by SORB and other spam trackers - than with sending email.
I would advice to try with mailinabox.email . It is very opinionated, which, for a beginner, is a good thing with mail.
I'd encourage everyone to try this on a tiny droplet or linode with an unimportant domain (don't just migrate your entire companies' exchange into it on a Friday afternoon). It takes you an hour tops, after which you can poke around and see all the moving parts that make a good mailserver tick.
You'll also be contributing to a stronger, more resilient internet, by making it a tiny bit more decentralized.
I run my own mail server for receiving, but send through a commercial provider (Fastmail). I also use user-sub@domain style email addresses. If I have occasionally have to send email from those addresses, I edit the From line in the Fastmail web interface or my email client.
I don't have a problem setting up a full email server, but I don't want to babysit it for just one person using it. I have too many real life obligations these days to try and fix the mess if a problem would arise.
Interesting! Maybe I should try doing the outbound via Fastmail or some other provider. I couldn't shut down my mail server entirely, but it might solve my deliverability issue.
I use postmark to do my delivery. They support outbound smtp that my internal postfix relays to and you pay per email even tho they gave me so many free credits I havent paid anything.
I've given out hundreds and hundreds of them. Almost any time I fill in an email address on a form, I'll tag it so I know where spammers and other miscreants get their addresses from. As have a number of friends I host mail for. Swapping to a mail provider that doesn't support them would be a giant pain for all concerned.
Probably because s/he has those addresses out there in the wild starting from a time that it was still reasonable to run an email server, so like 10+ years and probably a bit more.
That's exactly the case for me, but I've been handing out these addresses for nearly 20 years. I also suffer from the problems with gmail that the GP referred to. I can't even forward all my email to myself without having it dumped into gmail's trashbin!
I just add . in arbitrary places to distinguish sources. Since there's no telling what dots a gmail user thinks are a significant part of their address.. (plus I have seen fail and late enough to be a pain, but due to http parameter escaping, etc, rather than intentional fitering.)
So if I bought my “name.rocks” I’ll have trouble using that as an email? I was looking forward to “firstname@lastname.rocks” and I never even considered it would be a problem. I’m disappointed
I have something similar - I haven't had many problems with the not being accepted online. The few times I've had issues are with old point-of-sales or appointment booking systems.
I use Migadu.com for my email and they allow you to create regex based catchalls [0]. I have one that accepts every email address that begins with "cat". it makes it dead simple to create a fresh email for new services.
I don't have any problems with Gmail, but there is no way to know why :)
As far as I know the most important factor for deliverability is your IP address: my /24 have been clean for many years because the network operator actually respond to abuse@ reports.
If you have problems my first advice (aside from checking that SPF+DKIM+DMARC+PTR+banner+etc... are OK) would be to find a better ISP.
This is a physical server (at Datacate [1] in Santa Clara), so switching ISPs is not a small undertaking, especially given that I'm only using 1U of space. Last I looked the netblock reputation seemed fine, so I'd only be inclined to switch if I had clear evidence that was the real problem.
Yeah, the IP address of your MX looks clean indeed, and your config looks good too... I wonder, have you ever tried sending from @scissor.com instead of @williampietri.com?
+1 about the opaqueness of gmails spam filters. I once had an email from a google recruiter (@google address) go to spam in gmail so at least it's somewhat egalitarian (that was 5+ years ago).
That makes sense though. An aggressive recruiter overusing their work email to contact cold leads will get flagged a lot, even if the domain is clean I imagine they also train based off of individual addresses.
Yahoo's disposable addresses are constructed in that form (basename-keyword@yahoo.com), but I'm not sure if they support them on business/custom domain accounts.
(The base name also has to be different from your real email address, but that's a fairly desirable feature for disposable addresses since it doesn't reveal the real address like the '+' suffix on Gmail does.)
I don't see why sub-addresses would be a problem with regards to sending mail. Isn't it wholly up to the receiver to interpret the username part of the address (or not)?
1. Send mail via a static IP (that I pay extra for)
2. Send all mail through a reputable 'smarthost' (mine is provided by my ISP as part of the 'business' package).
This fixes all sending mail problems. For extra fairy-dust, I added SPF records to my mail server IP.
The BIG problem is Spam. Unless you pay for an intermediary spam filtering service, you will NEVER be on top of it. After self-hosting my own mail system for decades, I am now seriously planning a migration to Office 365.
I'm using SA as well; it's fine for me. My big trick is to route a bunch of stuff right into the spam training system. E.g., when a tagged address I've given out is compromised, that goes right to training. The same goes for spam to random names: also right to SA spam training.
My theory here is that statistically, by the time a spammer tries to send one to an active address, they're likely to have already fed things into the trainer, often multiple times.
I switched over to Gsuite and I use qmail style addresses, it’s not as friendly as using + addressing (as in it just works) but you can use regexp rewrites on the paid versions of Gsuite to fake it.
(Disclosure: xoogler, but still happy with Gsuite...)
I have a physical server colocated with Datacate in Santa Clara. As far as I can tell the address space is good, and I've had the server there for a number of years.
This is true! But when I last talked with them, I couldn't figure out a disaster recovery plan. They had no way for me to back up my users' mail aside from IMAP syncing. Which means I'd have to know my users' passwords. Which seemed slightly more ridiculous than just continuing to host my own mail.
1. share everyone's mailboxes to an admin user who does the backup via IMAP.
2. create an app-password for each user which can be used to backup that user. This can be done as an admin user on your account.
Neither of them require knowing the user's password - an admin can override into each account unless it's specifically locked down to deny that. It does require a separate app password per account, we don't have a way to create a single password which can view each user's account without them explicitly sharing the folders, but I'm not sure how you reasonably do anything else without it being a backdoor behind all the privacy settings on each account.
Ah, interesting! Yes, I'd be fine with either of those. Right now I'm just backing up via rsync of the maildir tree, so it's not like these are any worse. I just didn't want to a) know my user's passwords, and b) have to have them give it to me any time they changed it.
Thanks for the tip. Maybe my Christmas present to myself will be to murder my mailserver!
Sorry - busy at IETF this week, didn't get to reply before. I would recommend the per-user app passwords over sharing all the folders to an admin account - because that way you see the \Seen flags as the user when backing up.
You mean you can send emails to <user>-<tag>@<domain> and it'll be delivered to <user>@<domain> account (assuming MX for <domain> is pointing at fastmail)?
Is there anything you need to enable to allow this? I just tried, and the email got bounced back.
Or are you talking about email aliases? As in, you register <user>-<tag>@<domain> first before sending emails to that address?
For what it's worth, I don't even need to use qmail. I've been using Postfix for years, which allows this as a configuration option. But I'll check them out for sure.
This sort of snotty reply is one of the reasons I discourage people from using HN.
I chose it 20+ years ago because that's what qmail supported, back when subaddresing was a new idea. Over the ensuing years, most people ended up converging on + as the more common option, but that's only convention, not a standard. And what standard exists wasn't written until 2008: https://tools.https://tools.ietf.org/html/rfc5233ietf.org/ht...
If you took that as snotty then that's on you. Maybe people who can't take every simple question not being a delicately delivered paragraph full of qualifiers should be discouraged from using HN, yeah.
I have never had this issue. Generally the issue is either IP reputation of your server (common with VPS providers if you get a recycled IP of a previous spammer) or your domain name.
Otherwise you are probably just unlucky enough to tickle the spam-prevention mechanisms in the almighty "algorithm" run by $BIGMAILER.
I keep one "normie" email address at a $BIGMAILER for situations like this, but at this point in my life I mostly just shrug if some big advertising/surveillance company's email system won't deliver my mail, I just won't email that person.
My IP reputation is absolutely >>pure<< and my domain name is ok.
Still, usually when sending emails to Gmail & Hotmail my emails are "lost" (they don't even appear in the spam-inbox of the receivers).
My usual workaround used to be to 1) login into my throwaway-account on their Gmail/Hotmail systems 2) send an email to my domain 3) reply to that email. After that usually my emails got accepted by those service providers, at least for a while.
I admit that nowadays I don't even do that anymore - when I see a Gmail/Hotmail recipient I just ask for another email address at a different provider or I send the email using my provider's system.
Is your domain name a .com? I've found that domains that are not .com domain names generally get flagged by the evil algorithim even if everything else is perfect.
That's not even remotely true. I've worked in cloud/web hosting for years and 99% of users are using .com domains. Yes more new users may be registering ccTLDs or gTLDs but that's generally because they're cheap. And because they're so cheap it creates a nuisance as spammers use them. Look a the .xyz tld. It gives you a hugely negative score on any sort of mailbox spam scoring algorithm.
I used to keep statistics on this, although I don't anymore. I remember seeing one day that not a single e-mail I received was a .info domain was legit -- that is, 100% of e-mails received from any .info domain was spam.
However if you can get a correspondent to also set up their own mail server, then you may realise the pleasure of emailing directly from sender's server to recipient's, without using third party email "services" and third party email servers. First party to second party. No third party.
Every time these posts about setting up a mail server reach the HN front page, there are usually complaints and they always center around third party email services. Perhaps it is the use of those services that is the problem, not the process of setting up a mail server.
Spam is probably made easier by the fact that so many people use the same third party email providers. Spammers have less servers to target. (Not the same situation if every user had their own server.) As HN commenters oft point out, email, the protocol and software, is "decentralised". But the widespread "centralised" use of the same third parties to send and deliver mail has negated the benefits of being (theoretically) decentralised.
Yes. After many years, I think the algorithm is this:
If the FROMs top level domain is in the top 500 websites OR is a .edu NOT SPAM, ELSE, send to "proprietary spam filter". The "proprietary spam filter" is some random crap code that is unique to each email server company. You could spend years trying to figure out how to game "proprietary spam filter" - but it is a fools errand.
I don't think so. I see @facebookmail.com in SPAM folders way more often than you'd expect on both Office 365 and GSuite. I also see mail from Microsoft and Google filtered into SPAM on their own platforms once in a while.
facebookmail.com is not in the alexa top 500, whereas facebook.com is.
spam filters really are this dumb at big companies. usually the way we'd get around it is we'd call up $big_company and say, "hey, could you please remove our e-mails from your spam list? we would like to send your users a lot of mail right now." a few hours later, we'd be good to go.
this was ~2007. it may be harder to get them on the phone now, and the value of your company may need to be higher (we only had a few hundred million, which seems like chump change these days).
Yea I ran into the same issue. I've got SPF, DKIM, DMAC, and reverse DNS for IPv4 and v6 and still get flagged as spam. I wrote about this a while back:
It's not as bad now, so I suspect someone at Google is finally listening, but I still don't trust the reliability of e-mail. If I sent someone an e-mail I haven't e-mailed before, I'll let them know on Hangouts/SMS/Facebook that I sent them an e-mail and to check their spam folder.
Yea I was running into problems with my gmail address to other gmail addresses, even before I split off and started running my own server. Their spam filter has a super high false-positive rate.
I had DKIM, SPF, rDNS, etc. at first and went to Gmail spam. I registered on gmail postmaster tools and got some friends and family to add me to their contact lists and send me some outbound mail. Google now sends me more mail than I send them, and now I don't go to the spam list, even for gmail accounts I haven't contacted before. This was for personal email. If you are trying to send commercial email this strategy probably won't work.
I've had a similar experience. Switching IP address of my mail server lost me some reputation, but mostly just using the directions mailinabox.email has worked for me
This criticism came up last time too, and I replied that I've been running my own mail server for 20 years off of a DSL line, and I have no trouble sending to anybody (that I've heard of from recipients nor MAILER-DAEMON).
I have DKIM, SPF and no relaying and the only issue in recent memory was when AT&T started bouncing me for some reason. I emailed postmaster@ saying "hey wats up," and the block was removed a day or so later.
Do you know that your messages arrive in all cases? Not merely fail to bounce, but arrive?
I help with IT for a small non-profit membership org, and the problem isn't the places that throw errors when you send, but the ones that silently accept the message without complaint and route it to /dev/null. We get through fine to Google, but the other big providers are random at best about it.
>Do you know that your messages arrive in all cases?
"I have no trouble sending to anybody (that I've heard of from recipients" might not be as artfully written as I would like, but that's what that refers to.
Furthermore, blackholing emails like you suggest is generally rare. RFC 5321 (SMTP) states:
"[D]ropping mail without notification of the sender is permitted in practice. However, it is extremely dangerous and violates a long tradition and community expectations that mail is either delivered or returned. If silent message-dropping is misused, it could easily undermine confidence in the reliability of the Internet's mail systems. So silent dropping of messages should be considered only in those cases where there is very high confidence that the messages are seriously fraudulent or otherwise inappropriate."
Some mail systems can tell you if a user is valid or not during the initial connection while some forward to another mail system or delivery process and therefore cannot tell your MX during their conversation that the recipient is invalid.
The downstream MX could be configured to then send an NDR email itself but this lands you on spam lists as "backscattering" will then send spam back to whoever owns the email address the spammer is spoofing so it's generally not done.
>I've been running my own mail server for 20 years
Yes, maybe rhizome is successfully running his own personal mail server but your anecdote isn't really relevant. We can't learn from your example because the hidden rules of reputation & spam may not affect you but will affect others.
For example, a commenter (lucb1e) argued[0][1] that I was exaggerating the difficulties of reliably sending email but a year later in 2019, he confirmed the same difficulties! [2]
Do you see why we can't reliably extrapolate from personal experience?
I'm at least an instance of one that demonstrates that it's possible. The person you link to sounds misconfigured to some degree, but in the main two things they don't mention are whether they are using static IPs (as I am), or if they tried emailing postmaster@ to try and smooth out their difficulties. There is still a collegial network of people who run this stuff all over the world, and I'm certainly not the only person running a closet mailserver.
The biggest problems for mail delivery across the board are probably lack of correct helo/reverse ip lookup and being on an IP blacklist such as being on an ISP dynamic address. I will bin those with my mail server as well. That is a very low bar but any naive mail server implementation on a home server is probably going to fail those instantly.
DKIM and SPF are really good signals to have. I have DKIM, SPF, DANE(nobody cares), DMARC, MTA-STS, TLS (lets encrypt) which are all potentially useful in preserving reputation and repudiating emails spoofing my domain sent from other sources.
I have never had issues getting throough to gmail. But for some mail systems, Microsoft in particular I think, they seem to penalise low volume mailers very heavily. You seem to need a certain volume for their systems to cache a reputation score and if you have a small vanity domain/mail server with very low mail volumes you may go straight to the spam folder.
For some reason I never had problems, even after a recent change of my IP address because I moved to a new VPS.
Hetzner VPS, proper reverse DNS, SPF, that's all, not even DKIM. The only big ISP that doesn't like my mails is AT&T and I couldn't care less (in Europe, rarely have to deal with people @att.net).
I was running my own mail server for nearly 10 years on Hetzner. Prior to that on other hosts and in the distant past at home. Running mail servers is something I have done professionally and successfully.
The last time I moved boxes as I had many times before. I was on clean IP range but I had no IP reputation at all. In the past this wasn’t such a problem, especially with SPF, DKIM, rDNS, DMARC, server-to-server SSL etc. Around the same time I started having to deal with organisations (legal, etc for a death in the family and later rent) rather my my own circle of friends. It became extremely apparent that my mails weren’t hitting the inbox. But they were being accepted. This was extremely problematic.
I was in the group stating that running your own mail server isn’t hard. I still say that. The hard problem is convincing the big players to let your low volume domains and IPs hit the inbox. I begrudgingly gave up my MX servers last year.
I wonder if there's some sort of relay setup that could be used to mitigate this: e.g. if you want to run your own mail server, sign up for some kind of "mail ring" that transparently proxies the (encrypted) traffic from all the member mailserver administrators. That way, your public IP has a higher volume and more leverage getting delivered.
Integration with SPF and would be unfun and probably even impractical (max 10 lookups iirc?). I’d also worry about the inevitable abuse which would appear to originate from your range(s). If you can mitigate those though, it sounds interesting
I just do reverse DNS on a cheap VPS. I believe I don't even have encryption. Almost no problems since I switched to that setup 9 years ago. Some mails to Gmail end up in spam, and just yesterday I had my first "refused due to poor reputation" problem due to an unknown problem with Cloudmark. Have been able to reset my score there by filling an HTML form.
Of course, there are some mails where I don't know if they landed in spam or I just didn't get a reply.
Gmail has never been a problem for me, even when I only had SPF, sending from my server hosted at Rackspace. But I've had my domain for 20+ years, and Gmail since 2004, so they have had plenty of time to figure out its not a spam source, so this might not have been a fair test.
I just gave Yahoo a try, and that did go to spam, even though it was happy with both SPF and DKIM. I marked it as not spam, and then went and sent another. That went through fine.
I also tried Live. It says my SPF and DKIM are fine, but sends it to the spam box. I marked it as not spam. Nope. Next one still went to spam. Marked that as not spam and tried again. Still spam. Marked that as not spam, but didn't try sending another one yet. I'm going to wait a while, to allow for the possibility that it takes them a while to apply feedback.
Not a library, because a mailserver consists of a lot of moving parts and external configuration (DNS etc).
mailinabox.email is probably close to what you want, though. It's opnionated and my tip is to follow those opinions to the letter (or use another such project).
I've been running my mail server off DO for the last ±5 years - Postfix with SPF no DKIM, DMARC - I get good delivery for Gmail and yahoo. Initially, I thought I would go full on spamassassin and clamav but thought I should hold off until I got spammed. Turns out in the last 5 years the only spammers I've got are 2 senders, one threatening to share video recorded of me on webcam to my contacts and another keeps telling me my domains' seo has expired blah blah. These are 1 or 2 times in 3 months, SpamAssassin would definitely be overkill. Apple Mail client junks them anyway. About 1 year ago I moved the setup into a docker container to better maximize the use of the $10 droplet.
I have the same problem. I've been running my own mail server since 2000, and last year my sister's account started being used to send spam (I assume she reused her email credentials in another site that got hacked, and spammers used the credentials to authenticate as her in my SMTP server to send spam).
Google shows that my domain has a bad reputation, but I can't fix it because it doesn't have enough traffic. https://www.mail-tester.com gives me a 10/10 (I have DKIM, DMARC, SPF, everything), but my email still gets flagged as spam by Google.
I've run our email server for the last 7 years. Don't have any trouble with gmail.
Setting up was a bit of hassle, but since it was up, I don't do anything with it.
Occasionally I will get the odd notification that a (usually German for some reason) email address doesn't like our domain. But the major providers, never had a problem.
No idea why we haven't had a problem, I didn't know what I was doing at the time (still don't) just followed an online guide. From memory, spf, dkim which I think is all pretty basic things to setup when setting up a mail server and all guides cover it.
> If anyone actually knows the secret to not having your own mail-server's mail go to spam on these bigger systems, please tell, I've Googled it for years with no success.
I've been running my own mail server without sending problems for almost a decade. Right from the beginning, I had a clue and knew that sending SMTP directly from a subscriber IP address was going to be a nonstarter. Such IP's have a negative reputation due to abuse. Instead, from the beginning, I used my ISP's (big ISP in Canada) SMTP forwarding host, configuring my SMTP server to log into that server with my credentials.
(I had a couple of hiccups along the way with the ISP's SMTP service, and they wouldn't help me unless I reproduced the issue with a supported e-mail client. Naturally, an Exim server running on Debian isn't a supported e-mail client, because it's name doesn't sound like "Microsoft Outlook" or "Mozilla Thunderbird").
Anyway, the main advantage of running your own mail server is not the ability to send SMTP directly to someone's MX server, but rather that you have your own MX domain for receiving mail, via a machine you control. Your setup isn't "lesser" in any way just because you're sending through a more reputable SMTP forwarding host.
So your recommendation to having your mail not go directly to spam is to run it for a decade? Got it! :-)
I run a mail server for a few thousand users that has been going for ~20 years, and have regularly had to spend significant time figuring out adjustments the big mail providers make along the way. Spending a day roughly every year figuring out and implementing countermeasures isn't that bad, when you get paid for it. But I really don't have the time to do it for my personal e-mail.
Your answer reminds me of something I heard a greenskeeper say about the big garden he maintains, I think in the UK: Tourists come to me all the time and ask "How can I get my lawn to look like this?" I reply: "The secret is to roll the dew off the lawn every morning." "That's it?" "Just do that for 300 years and your lawn will look like this."
If you have to do it for your personal mail compared to a few thousand users, the problems will obviously be fewer, because you don't send to as many different destinations as a crew of a few thousand users.
> "Just do that for 300 years and your lawn will look like this."
My point is that I've never had problems in that decade, not that it took a decade to fix. I think on two or three occasions, mail to a gmail user mysteriously bounced, but it went away on re-try.
I understood the reputation problem from the beginning instead of banging my head against it. Why I understood that is because I did some research into running mail servers before diving into it.
And my point, not to belabor it, is that reputation seems to be a pretty big deal these days, and your experience running a mail server with a decade of reputation may be different than the experience someone has setting up an e-mail server today. :-)
Be sure to have a PTR record in the DNS zone of whomever owns your IP address. I have DKIM, SPF and DMARC setup, but my mails were getting sent to spam before rDNS was configured correctly.
More to the point, I discovered this by reading Google's postmaster/bulk mail guidelines. All the other major email providers have similar guidelines, so definitely consult those as a first reference if you are having problems.
There is no mail conspiracy. As always, the problem is DNS.
This is not a problem I have or have had using a self hosted mail server on and off for nearly a decade now.
What I have had an issue with is the network (the whole ####ing network, those guys are less than helpful) I’m on getting added to spamhause(I think?) and people setting up their (smaller) mail severs to blindly reject mails from networks like that. That’s really dumb and I’m glad for things like dkim because of it.
I used to run a mail server in the past, and initially had problems with reverse DNS resolution not setup correctly, it's an easy oversight depending on where you host.
Gmail seems to have no trouble with my mailserver. AT&T/Yahoo used to, but there was some online form I filled out at one point to get myself whitelisted (it was years ago; don't remember the details) and that stopped being an issue. The main problem right now is Craigslist (which outright refuses to deliver my server's mail, so I have to use my Gmail account or stick to SMS / phone calls).
Stack: OpenBSD VM (on 1984.is) running OpenSMTPd + Dovecot + a DKIM proxy + ClamAV + spamd.
The more relevant pain point is that the VM hangs every couple months and I have to hard-reboot it from the VPS web console. Also, I've been pretty lazy about staying on top of new OpenBSD releases. One of these days I'd like to go redundant (whether with both mailhosts on 1984 or putting one on a different provider), which would make it less stressful to do OS upgrades (I could keep one host running while doing an offline upgrade of the other, as opposed to hoping the online upgrade process doesn't break anything).
I am writing this with sadness and frustration in my heart.
The only issue I have found seriously limiting my ability to use my own self hosted email server is the deliberate decision of Google, Yahoo and Microsoft to make it hard for me to do so.
Their decision is essentially un-appealable. I still get hotmail complaints about spam being sent from an IP I do not own anymore. For the life of me I could not find a way to solve this. Yes I did remove that address from their “responsible domain owner nonsense thingy” to no avail.
Often times I would send an email to a domain owned by one of the big three and that email would just disappear. No bounce back at all. The recipient would not even see it in their spam folder.
They tell you to implement various technical solution line DMARC and DKIM but in the end what really matters is the arbitrary and capricious decision of these three companies against which there is no recourse.
I agree that hosting your own email server is important but this is a battle that I gave up fighting.
Run your own MX servers but use an external service for your SMTP server. A good service will help you with your SPF, DKIM and DMARC setup as part of the service. As you are only using the SMTP service you can optimize for that and ignore mailbox sizes, etc. when shopping around. You should be able to get multiple App Passwords so no shared credentials.
I run a couple of mail servers and the beginnings are always a bit rough, but then everything works (mostly) fine. I start by asking Microsoft to unban my IP (for some reasons the IP range with my favorite hosting provider are banned by default by MS servers). They lift the ban and since I'm a good netizen and use my servers strictly for the communication with the customers (and never for marketing), I never have any problems with delivering to outlook.com, hotmail.com, etc.
Google is a different beast. Last time I remember I had to do strange things like disabling IPv6 support in my MTA (which is a good idea anyway as I'm not an expert in IPv6) in addition to the usual SPF setup etc. E-mail is delivered in general, but once in a while some letters go into spam for unknown reason. But this happens rarely, maybe a couple a times a year.
Whenever I set up a new server, I have to start from scratch though.
Not just the big players, I had a conversation with the administrator of the Australian ISP Bigpond about why my server was being spammed. Their response was verbatim "We blocked Digital Ocean, because we get spam from there" Even with good reputation servers, admins are happy to reject legitimate mail for any reason.
Sample size of one but I’ve run my own mail server for years and have almost never encountered problems of this nature.
Right when I started with the VPS, rejections from Comcast started showing up in my logs, and I traced it back to the IP I got being in some blacklist. Requested a new IP and it has been smooth sailing ever since.
I had problems with MS (Hotmail/Live/Outlook/whatever they're call it now) when I changed hosting provider which I traced back to the hosting provider owning an IP that had in the past been associated with spam (that for-real is what I was told) -- that seemed a bit over-zealous to me.
As it wasn't possible to pre-check IP addresses (it wasn't blacklisted, and in any case the email was being served from a different IP to the one that was apparently previously an issue) for any particular hosting company - and I could only afford cheap hosting - then I decided to relent and made an Outlook online account and sent the mail from that.
It seemed totally crazy to me to reject mail from a 15 year old domain that sent at most 30 emails to Outlook per month, _always_ in replay to an email; SPF/DMARC were set AFAIR and whitelisting the email address from within my own 20+yo hotmail account didn't make any difference, MS still wouldn't let _me_ receive the emails. Like if your user says the email address is fine, and sends an email to that address, .. then perhaps you should allow the user to receive that one email??
The trick to this is not hosting your mail server with a provider which has a bad reputation.
Mine goes through a DigitalOcean droplet and my mail usually gets through (based on the assumption that if I received a reply, the other person must have received it!)
By contrast a volunteer group I'm involved with was using OVH "because it's the cheapest". Suppliers and customers were routinely telling us the emails were going to spam. Moved to another provider (no idea who, the infrastructure team does that) and the complaints fell to a trickle.
Good IP block reputation (check RBLs) + valid SPF + valid DKIM is usually enough to get mail through. Except occasionally to Hotmail... but that's Hotmail...
Easy. Dont run your own SMTP server and instead use Mailgun or other reputable SMTP servers to send emails. they are used by all large newsletter/marketing systems. Free tier services are largely enough for personal users.
I have run my own server for the past 20 years. I don't spend a lot of time maintaining it but managed to enable SPF, DKIM, etc as they rolled in. The most disruption I have faced has been changing connectivity providers and IP ranges. But even then I have been able to insist on being allowed to relay and removed from blacklists consistently. I think it is important some of us do it even if most people live off big provider webmail.
I must have gotten lucky because mine just works. My exact setup is mailinabox installed on a vps at vultr. Have been sending to gmail addresses mostly fine. Had 1 email go to spam in 2018 but everything else has gone through correctly.
Since mailinabox configures everything for you maybe the issue is that when you set it up yourself you don't quite get it exactly right to what gmail and microsoft demand.
I added my domain and have never had it marked as spam at any major email provider, only a few small, self-hosted servers and that was probably due to misconfiguration or high spam sensitivity. See if this might help you out.
I have been running my own mail server for over a decade, and still do. However, last year I had to throw in the towel and configure my server to send via an SMTP relay that my hosting provider (transip) provides. That solved all problems I had with sending to the big mail providers.
You have to configure a super complicated system of SEC-DNS and connect it to Postifix.
Thanks to the Spaghetti Fly Monster, this guy made all of this automatically
the reputation is almost like a protection scheme when it comes to emails. Its prob to make it hard for people to run their own email server. Its better for the security services if most people are doing email from a few providers.
My email server (@chrisdone.com) has been running for a year and a half and is written in Haskell, saves all mail into a postgresql database, has a web interface and an Emacs interface. https://github.com/chrisdone/duta
I redirected my gmail account to forward all emails to this new domain, and changed all my account emails to theservicename@chrisdone.com e.g. for github it’s git@chrisdone.com. Once I’m done, Gmail won’t be receiving any new mail.
It has SPF which has captured most spam. I haven’t implemented sending yet. I thought this would be a problem initially, but after a year I’ve realized I rarely send personal email out. But I’ll do it one day just for completeness.
I’ve found two bugs of my server, which were easy to debug because mail sending is very robust. I fixed the bug, and the sender server tried again the next day automatically.
I did it for exactly the same reasons as the author, to control my own data. I wrote my own because existing open source offferings are silly complicated, but receiving mail is not.
Duta is easy to deploy with docker machine and DigitalOcean. My current setup has been deployed like this. It connects to a managed DO Postgres database, which runs separately, has monitoring and backups, so it’s easy to redeploy the mail server without worrying about losing my data.
I feel like if people are gonna take back email this process needs to be easier; like a single application with a web front-end.
Hosting email is a pain! This is why everyone seems to outsource their email hosting.
Read the comments in the 2017 discussion. Favorite quote: "I run my own mail infrastructure. To say the least I wouldn't recommend it even to my worst enemies. It's horrible."
https://news.ycombinator.com/item?id=16238937
I have been running a mailinabox email server for over a year now. You basically just run the install script, tell it your domain name and set the server as the name server for your domain name and its all done. The thing does everything for you including renewing certificates. It then sends you a monthly status email and an email whenever something needs your attention like the certbot renewal failing.
So far in the past year the only manual intervention after setting it up was updating it from ubuntu 14 to 18
It's sort of a managed server that lives in your home. I'm not sure whether that's actually better than having your data out in the cloud or whether it's more on the security theater end of the spectrum, but it is interesting.
Then EVERYONE stopped because it's awful. Modern ISPs don't like SMTP floating around, (DKIM, DMARC, SPF) are a pain to maintain over time, exploits will happen, and your free off-the-shelf virus and spam filters suck compared to the big companies seeing billions of messages per day.
I had my own mail server running for close to a year after which i gave up because i started seeing issues delivering mail to gmail and apple mail accounts. I had to keep tinkering with the DKIM/DMARC/SPF entries to make sure i'm compliant with gmail/outlook/etc. At somepoint i gave up because it wasn't worth the investment of time.
FWIW, i used mail cow [0], which was a delight to setup and use. And it comes bundled with a decent-ish material design webclient.
I would love to go back to hosting my own mail server though. As of now, i shut it down, and use gsuite.
Still running my mailcow. I don't care if people don't get my mail. I can show them that i handed the over to their server - everything else is their concern. At least legally. If you can't read my mail cause you misconfigured your spam filter I can't help.
When someone in a company agrees on a meeting in their office with someone from outside the company, and the visitor arrives at time well-dressed and is rejected by the companies security without any explanation, then who is at fault? It is not the visitor, it is the awful cooperation between the person in the company and their companies security.
It is entirely reasonable to show how you've handed your mail to gmail and they refused it without actionable explanation, so it is a problem between the recipient and gmail.
GMail being at fault isn't going to make up for a missed invitation (or RSVP) to a special event, or not getting a note of congratulations on an accomplishment, or not getting a message of condolences.
Assigning blame is done after the fact to make up for a mistake.
If one completely disregards political consequences, and generally anything but personal short-term benefit, I agree with your reasoning. Otherwise, I see not how that would be possible. And I think arguing just for short-term practicality is deeply flawed; by that you'd also cooperate with a murdering dictator if it helped you personally.
> your free off-the-shelf virus and spam filters suck compared to the big companies seeing billions of messages per day
gmail spam filtering is quite bad, in both directions. Legit email goes to spam (even when sent from gmail itself) and spam does get through.
The basic open source tools are actually very, very good at it. I get about 0-2 spams a month, everything else is caught by either my postfix configuration or by spamprobe. And I never get legit email in my spam spool.
This is a much better success rate than what I observe at my gmail account.
Indeed, I'd argue that a selfhosted learning spamfilter will be far more attuned to your personal mail-regime than one that has to do 'well enough' for millions of users.
The latter will catch new spam tricks easier. But the former will be trained by you. For you. For your niche, using your lingo and so on.
To take an extreme example: let's imagine you are the ITperson for a doctors' practice aimed at people with sexual disfunctionality. One thing is for certain: using Google or Outlook for your mail is a disaster waiting to happen.
Most ISP don't care about it. But there are certain enterprises that know almost nothing about email, they employ "specialized tools" that scan and classify incoming email (read Symantec whatever) and make evereyone's lives harder.
It's not that they dont know or don't care - it's that not being super aggressive with filtering caused people to almost abandon email a decade ago because spam became unbearable for most people.
Those "specialized tools" make email viable. Without it, everyone would be spoofing everything and you wouldn't use email anymore.
At my last place of work one could send as ceo@company. Still, they had crazy (proprietary) filtering. People make things better by accident, in the mean time they tick check boxes.
my thoughts exactly. it used to be fun to provide hosting and email for yourself and friends, but then constantly upgrading and tweaking spam filters was just a huge waste of time.
from what i understand, the gmails, yahoos, outlooks, whatevers of the world basically make it hard on purpose to make their own jobs easier.
The key to get high delivery rates to GMail and Office 365 is to setup DMARC. When you have a proper DMARC configuration (and at least SPF) your delivery problems will suddenly go away.
Hosting your own mail server is not rocket science, but you need to have solid sysadmin skills and a good understanding of email as a whole.
If anyone is interested in doing this: Start simple with only Postfix and Dovecot, don't use a database for username/mailbox configuration as most tutorials suggest (start with text files instead). You can also start with OpenSMTPD and Dovecot if you think that Postfix is too complicated.
And if your setup is finally running, make sure to setup proper monitoring (e.g. make sure your mail server is running and answering SMTP connections). You can use free tools like uptimerobot.com for that and get notified before you loose mail.
> Hosting your own mail server is not rocket science, but you need to have solid sysadmin skills and a good understanding of email as a whole.
Not really. I had neither and got it up and running. Sure I had some issues, but as long as you have some competence, most things can be sorted out / figured out.
I have been using Mail-in-a-Box https://mailinabox.email/ successfully for many years now, for both my personal account and my company. The only problem is that occasionally Gmail and Outlook mark my sent email as SPAM to the receiver.
I'd like to point to another smtp server, which I am using and is much easier to set up than postfix in my opinion, especially for small servers like mine: https://www.opensmtpd.org/
I'm running a personal mailserver (opensmptd, dovecot, dkim, spf, dmarc, spamassassin) for some years now and although I initially had problems with deliverability to google there aren't any (apparent) issues so far.
Also I don't understand why people keep emphasizing google's spam filter being so much better than anything else. For my personal server SpamAssassin has proved itself to be more than sufficient, its spam filtering performance is on par with Gmail's (I have a Gmail account aswell), at least for me.
Of course, Gmails spam filter works better for the billions of accounts they manage, but in order to handle the spam of a tiny mail server I'm probably not the only person who is satisfied with SpamAssassin.
A ( somewhat ) happy medium between hosting your own email ( and dealing with getting IP blocked ) and fully entrusting youtself to Google is to use webmail as provided by your domain name registrar.
For example, gandi.net provides web mail on your own domain. Even, better, it has support for wildcard email just like google does.
Earlier this year, I had to move a number of email accounts (my own, family, friends, clients).
Based on a fairly extensive comparison of email hosts - with brief stints trying self-hosting as well as Zoho - I migrated to Migadu and have been very happy with their service.
Their pricing is clear, structured well to suit my needs, and the service has been reliable.
Mailbox.org is fairly cheap (a few euros a month) and it supports custom domains. It also has some neat features like auto-encrypting all incoming emails to a given PGP key (while not perfect it makes all of your emails more private in storage and makes it harder for a compromise to result in past private information from being stolen -- though of course Mailbox.org could in theory just keep the original messages around). They're also based in Germany.
Great write up. I will almost certainly learn something from it in review.
I have run my own company email for around 20 years and other people's for over 30. I'm in the UK.
No matter how technically competent your setup, if your IP is blacklisted by one of the big lists eg Spamhaus then you are screwed. So you need to avoid that. Avoidance tactics involve properly configured MX, A and PTR DNS records. Then you'll need a proper SPF TXT record. Can you do DKIM? DMARC?
Oh and please avoid sending spam to people - it ticks them off and get's you blacklisted.
Kudos to the author for this, this is an endeavor of madness IMHO...
The only thing I've ever failed to setup was a functioning mail server using Postfix and Dovecot (it was like ~2007/8). It was a circle of configuration hell I would wish on no one and since then I've never understood why anyone would want to run their own mail server. Pure, unadulterated, hell... I honestly don't even know why we have email at all it's so shit (obviously I do, but hopefully the sentiment I'm expressing is clear).
Granted, if you know how to do it, once it's "running" things mostly work (I've known plenty of people who didn't fail as miserably as I did)... but as others have mentioned actually sending emails and not being labeled as SPAM is almost impossible. Plus, making sure your service is literally ALWAYS available, is really quite tricky. It's pretty damn easy to miss some emails.
I keep hoping some other protocol will take off so we can free ourselves from email, but it looks like the chance of that ever happening is probably zero.
>It was a circle of configuration hell I would wish on no one
>actually sending emails and not being labeled as SPAM is almost impossible
Anecdotal, but I really didn't have any issues with either of the above. Just don't use a tainted IP (protip: basically every cheap VPS provider's IP space is tainted) and set up SPF/DKIM properly. Following a tutorial is probably the way to go if you're not familiar with Postfix/Dovecot. And as you mentioned, once you have it up and running it's basically zero maintenance.
>making sure your service is literally ALWAYS available, is really quite tricky. It's pretty damn easy to miss some emails.
Mail standards actually make it really hard to miss emails. Every mail service will, by default, retry delivering mail for 4 days (!). This is the default in MTAs and the big boys (Gmail, etc) do the same. If you want to be more HA you can set up backup MXes but that's overkill for a personal server.
I don't understand people who try and cook 3 star michelin food at home. I would rather buy a kebab. But for any competent pro setting up a mail server is ridiculously easy. After the first few you have a recipe. Changes are incremental and relatively infrequent. It is much easier than keeping up with any programming ecosystem or framework.
I don't know that I like some of the deskilling I see. It seems like many people would rather learn a proprietary cloud api or use a prebuilt container than learn the fundamentals of setting up their own services.
A personal mail server, tested and refined over decades serves as a template that can be applied to other projects. If I have a web server backend that needs to send emails I have a pre-built and tested low maintenance solution I can deploy very quickly.
The main issue people will encounter is that you are at the mercy of a much bigger ecosystem where high volume senders have all the power and your good work implementing best practices is rarely rewarded. In the end your mail delivery is at the mercy of some shody unmaintained commercial abandonware spam solution maintained by someone who failed up and doesn't give a shit and has never read an RFC.
I've been running my own mail servers since 1994. I started with smail, then sendmail, then qmail, various pop3 and imap servers, now postfix and dovecot If you set it up and do it right, which means SPF, DKIM, DMARC these days, plus obviously basics like reverse DNS, it is almost trouble free.
Making sure your service is always available isn't even necessary. Queuing and retries are built into the SMTP protocol. I run my servers off a cable modem and I easily have 99.9% availability.
This topic (though not always this particular article) seems to come up on HN every 6-9 months, and the discussion is ALWAYS the same: "Yeah, you can, but you'll have to deal with X, Y, and Z. Probably not worth it for most people but interesting to know that you can."
These tutorials are nice but you can just hack https://hub.docker.com/r/analogic/poste.io (shameless ad) or any other containerized solution. You will get fully working solution in couple minutes and it will be somewhat easy to keep mailserver updated...
This. I have been running my own mail server since UUCP bang paths and sendmail. I am slowly preparing to migrate to a containerized solution for exactly this reason. Managing the updates of all the parts going forward is just too heavy a load, while swapping in a new container is easy. The initial configuration is painful, just because there are so many options to decide on, but those decisions are all there anyway, whether you realize them on day one or not. I am planning to use this image[0].
Offlineimap + mu + mu4e is my current setup. I use gmail, but pull it all down with offlineimap. So it is all backed up even if I lose access to google. MU indexes your mail and gives you very fast search - much better than gmail's search. And mu4e lets you read your email in emacs.
In the diagram, the program that connects via IMAP to the MDA can also be a standalone MRA[1]. With a standalone MRA, the MUA doesn't need to know the protocol used by the MDA. It just reads the mail from a local mailbox (using formats like Maildir or mbox) where the MRA wrote it.
Does anyone know if there are MTAs that can pickup mail from a local directory to send out and MUAs that don't need to use SMTP to send out mail, but have the support to rather just write it to a directory?
I was wondering if the sending of email could also be handled by a standalone program separate from the MUA like it can be done with the receiving of mail.
One thing that would still be missing is SRS [1], in case you want to forward mails to an email address handled by a different mail server.
Your mail server must not use the original from address in the envelope from, as that would very likely trigger SPF record checks at the receiving end. When forwarding mails, it is therefore strictly necessary to rewrite the envelope from address with SRS.
For this postfix setup, postsrsd [2] can be used to implement SRS with only a few more configuration options.
I would like this to be true (really) and it can be true only if only you are the only mail system user.
As soon as you give an email account to a non techie and non email politeness aware user you are doomed. Like your windowsxp user mum with a compromised outlook express sending crap to full addressbook. Your friend is sending last penis graphic joke to their (gmail/msn) friends. Dad has used email to register to many dark sites and now all *@yourdomain.com receive viagra offers and they ask you to stop receiving that. Girlfriend is trying to send a 1GB video to her workmates.
But then, one user mail system does worth the effort?
I exited gmail a few days before the "hey lets read your mail and generate ads" took effect. It would great if someone someone else decided to use it and fix it up for their use case:
Does anyone use Hashcash? I haven't run my own personal mail server in 5+ years, but I used to feel like Hashcash really helped keep my messages out of spam folders. It seems like SpamAssassin is really the only filter that implements it, and I don't know how prevalent that is today. But I used to spend 10-60 seconds computing a hashcash on every outgoing message that left my server, and has basically no problem with getting caught as spam. But I also had a domain and IP address with like 20 years reputation behind it...
I've had a relatively trouble-free experience running my own email server for years by just setting up outgoing email to relay through through the gandi SMTP I got with my domain registration. Email providers don't verify the the reputation of what they want to send to very rigorously, so I don't have any problem receiving my mail.
It depends on why you want to run an email server, I suppose. I just didn't want google collecting, analyzing, and indefinitely storing my incoming mail (left gmail).
Hosting email is definitely a pain. You could do all the validations with DKIM , SPF , ARC etc. Its not helpful if the reputation score of the server is low.
I have been thinking about this for a while , there should be a way to decouple email server and storage into 2 different services, where I could authorize email server to grab the data from storage. The storage service itself should have zero knowledge encryption. This way, I could just switch between email services.
Before you get too far down this rabbit hole (like I did) for a home mail server, make sure that your ISP supports inbound/outbound mail traffic. Mine blocks it permanently, and I've been unsuccessful at getting it changed. In fact, I can't even get the tech support people to understand the traffic is blocked before it reaches my house and it's not a "firewall" thing.
Should you host your own email server? Yes! It’s a really fun learning project. There are tons of resources out there to make it do-able, so go for it!
I've been doing this for inbound email on my domains, and outbound mail from daemons and system services. I don't think relying on your personal email server for all email is practical (deliverability), but there are good use cases.
I've been running my own email since about 8 months ago. I have Google's MX (that I used before) on backup (with a lower priority). Will not help me with every sort of fuckup possible, but it will at least deliver the mail if my server is down.
Judging from some of the experiences upthread, even that isn't enough. Billing messages from google themselves and gsuite-to-gsuite messages getting spammed.
Any source can have some fraction their messages treated as spam; I was just saying that if you take a random IP address and set it up as SMTP, you will have trouble getting any messages through, even if DKIM &c. are setup correctly.
Roundcube is _acceptable_ and easy to set up (it's just a PHP web app that acts as a full-fledged MUA and connects via IMAP to your MDA, no different than any other mail client).
Fun is a very curious way of putting it, considering how batshit crazy IMAP is. The rest of the involved email standards are also a huge dumpster fire.
You know the term "opinionated software"? IMAP is an opinionated protocol, designed for clients and servers that work exactly the same way that Mark Crispin's PINE and UW-IMAPd did. Disagree with Crispin about anything, and the protocol will fight you every step of the way.
Mind you, I think PINE may have been the greatest email program ever written, but Crispin was not the easiest person to get along with.
Why would you need to deal with IMAP at all for a webmail client? If the mail server is the one hosting the webmail frontend, you could read the Maildir or mbox (or whatever you're using to store emails) directly. IMAP's only really relevant for locally-installed clients (i.e. the exact thing webmail's supposed to replace).
If I were to write a webmail client for either myself or as a non-toy-project, I would definitely want to be able to host the webmail backend somewhere else than the MTA/MDA, and especially not give the webmail full access to every user's email.
That leaves you with either using some protocol that already exists (IMAP or now JMAP as another persion mentioned in the thread) or coming up with my own.
There are still a couple options here besides IMAP:
- Use server-local users, and have the webmail prompt for those credentials and use them to browse that user's Maildir via SFTP. "Good enough" for small-scale operations. Probably not the best choice for large-scale operations, but likely "good enough" for small-scale.
- Store the emails in a SQL database (e.g. Postgres), with row-level permissions to SQL users for each address, and have the webmail prompt for credentials for those SQL users and use those to connect to the DB and query messages. Probably the ideal choice for large-scale operations.
Both of these options seem more reasonable to me than trying to do anything with IMAP.
The first option would likely be terribly slow. Scanning emails to check for things that have been moved around and client-side-grepping emails would be extremely slow. For this to even work you'd probably want some local cache, which is not trivial to do. Granted, my benchmark here is people like me who have over 100k emails on their account.
The second is likely the better thing to do (But likely not in postgres for large-scale, if anything due to the SPOF. Unless someone really enjoys maintaining postgres clusters). However, you still need some IMAP/JMAP somewhere for normal mail client access, and in this case you'd likely have to implement your own translation/access layer. I would probably come up with something that wouldn't have users directly access a database, and instead provide some sort of identity-aware access layer. Basically reimplementing IMAP/JMAP, again.
I wrote a webmail client (I've apparently long since lost most of the code) to go along with a personal email server I was running about 12 years ago, so it's entirely possible I was just speaking for myself.
If you are going to write your own webmail, use dovecot for your mail-server and then target the maildir store rather than IMAP. You will be much happier.
It really is not. At most it can be tedious, because it has large surface area of protocols and formats. Most of them other than SMTP and message format are optional.
Minimal mail system involves just postfix with config in a bunch of files, delivering mail to a folder on the server, and pretty much everything on top, like SPF,DKIM,DMARC,... is optional.
You can ssh to your server from anywhere and run mutt there, or install a webmail if you're into web stuff. That's pretty much all that's needed for a basic thing. Security wise, there's not much to do wrong in this setup. Mail server is open for reception by default as it should be and submission is only allowed from localhost.
Webmail can be protected via HTTP auth and ssl. As long as your password is secure, noone will get there that way.
Rolling your own crypto is a much bigger can of worms.
It's not the hard part, it's simply impossible to ensure from your side. You can have everytrhing righy, and they'll still block you based on IP address.
You can configure your dmarc so the big mail systems all provide you with reports on delivery failures. I get lots of them when I reply to a mailing list that doesn't deal with spf or dkim sensibly.
Not too much, unless you are completely new to server environments and networking.
You'll likely experience security breaches rolling your own crypto unless you are an expert in cryptology.
You'll also experience security breaches running your own email server if you also don't know what you're doing, but it's far easier to learn what it takes to configure an email server correctly from a security perspective than rolling your own crypto.
The bigger problems with running your own email server is NAT/CGNAT, ISP restrictions, and the fact that the big players will still ignore you due to only caring about the other big players.
Alternatively pay for a GSuite account for like £3 a month (Or Office 365) and all of this is handled for you with superior spam filtering, search, webmail, no manual administration and no issues with email delivery due to other filters thinking your server is a bit dodgy.
It's too much effort to run your own email server well.
I'd recommend Rackspace. $3 per user per month, and allowed me to bring my own domain. Haven't had any issues after setting everything up.
I tried registering this domain through G Suite and Zoho first, but they both said my domain had a TLD that's commonly used by spammers, and refused to bind it. Zoho eventually made an exception but took over a week for them to get back to me.
The obvious rebuttal here is that not everyone is comfortable having a Google or Microsoft account, or having Google and Microsoft scanning their emails.
In that case, Fastmail is a good alternative. It’s not quite as good as Gmail on security and anti-spam, but it’s much better on privacy.
Can you clarify what you mean by "security"? I've found Fastmail's spam filtering to be quite excellent: very few false positives (usually just non-spam marketing messages) and almost no false negatives.
Security - I think Fastmail is pretty good at security, but I don't think there's another firm on the planet that does as well at security as Google. Specifically, I found Fastmail's implementation of 2FA to be a bit lacking. I imagine that a determined attacker would find it easier to abuse their support processes, since Fastmail's support is quite helpful whereas Google barely even responds.
Spam filtering - I have both Gmail and Fastmail. I'd score Gmail 10/10 on spam preventing, and Fastmail at 8/10.
By the way - I told our support team that you told them they were quite helpful. They are pleased to hear that feedback :)
Seriously though, we have been making as much effort as possible to push the account recovery away from being an "agent decides" to following an automated process. We famously got this wrong once many years ago, and have learned from that!
So the fact that our lovely support people are very helpful once they have identified a person does not mean they are pushovers for an attacker.
They genuinely are very helpful. I always get the feeling I'm speaking to a human being that knows enough technology to answer my questions, and has the empathy to try to understand my problem.
I remember on one occasion needing to extract billing info from a number of different accounts that were under one master account. The support explained there was no native solution, but helped me come up with a workaround. That kind of support is sadly missing from a lot of vendors.
I'm very proud of our support team! That's exactly the kind of help I'd expect them to give anybody :)
We are expecting people to pay money for a service which others provide "for free", so we need to differentiate ourselves, and having real humans who not only provide great service, but are located close to the technical teams and can feed quickly back to improving our service is a key differentiator for us.
It’s not hard at all, and in fact is Typically easier. Google has degraded search to the point where it rarely gives me a useful result; instead the primary results are ads and “boxes”, so switching to ddg was a pleasure. tools like mail and photo aren’t as fast or convenient as local apps; with local data I can switch from app to app. Yes Apple Maps isn’t as good as google maps, but in some ways is better. I watch the odd YouTube video in private mode but if I watch three videos a month it’s a lot so if it went away I doubt I’d notice. and docs...blech, where to start. I still get new documents from companies I left long ago. The editing tools are primitive.
I know people talk about Apple having a “walled garden” but really for me it’s google that has one, now with a shitty web based interface.
Basically at this point my options are so much better that the privacy is a bonus.
