On the perceived foolishness of self-hosting e-mail

TL;DR: There has been some activity within the Fediverse about how running your own mailserver is or isn’t foolish, and how “we” lost the fight against big corporate. This is my take on it.

Carlos Fenollosa recently published an article titled “After self-hosting my email for twenty-three years I have thrown in the towel. The oligopoly has won.”.

It gained quite a bit of attention in my online social circles, and while I do agree with some of the points they made (specifically that e-mail isn’t as simple as it used to be a decade ago), I disagree with fair bits of their opinion. I know that I’m late to the party, but I want to use this post to elaborate why nonetheless.

Before I throw my opinion at you, some specific context on that topic about myself: I haven’t been hosting my own mail for as long as the author does, but it’s been a while. If I remember correctly I had my first full-fledged mailserver in 2013, then based on the excellent “ISPMail” tutorial-series.

Since then I ran my own mailserver, mailservers for friends, helped to build and maintain mailservers for ISPs and large corporations and was involved in incidents, both security and operational, involving e-mail quite regularly. I’m not an expert, but I’d be confident enough to say that I’m not entirely clueless on the topic.

I understand why the author decided to stop self-hosting their e-mail, letting Apple do the dirty work for them (at least according to the MX-record for their domain). And he is, obviously from the title of their post, recommending other people to do the same, to “stop self-hosting their e-mail”.

What’s the usual answer from experienced sysadmins? “Stop self-hosting your email and pay [provider].”

Yes .. and no. I am aware that this has a gatekeeping tone to it, but I am of the opinion that a lot of these posts the author refers to are made by people who should not be running their own Internet-facing mailserver yet.

And while, yes, there are situations where ’experienced sysadmins’ will tell people to stop self-hosting their email straight away, more often than not from resignation, but because the use-case would be better handled by a commercial provider or because the requester has made the impression that their systems are having significantly more, and worse, problems than their mailserver-issues they were asking about in the first place.

But, if what the author claims happens to be true, it doesn’t really matter if the issue people are having gets fixed, because they already made a mistake and thus they wasted their chance at participating in the distributed exchange of messages:

One strike and you’re out. For the rest of your life.

If that were the case, I would not be able to send or receive e-mail on my own.

I was once a confused beginner asking odd questions too. I luckily never managed to create an open relay, but I have had issues with my DNS, I have had issues with authentication, I have had issues with encrypted connections, I have had issues with mail loops .. the point I’m trying to make is that I fucked up a lot of times while learning how to do e-mail somewhat properly.

Yet despite all my errors and mistakes, my e-mails tend to be delivered. At least the ones from the IP-space I control. Despite my best efforts it doesn’t look that good for my domains where the MX is located with one of the many “Cloud”-providers out there.

And this leads me to an issue that’s, in my opinion and from my experience, a more significant issue with regards to deliverability. An issue that tends to be ignored a lot: Reputation. Specifically the reputation of an IP-address or whole blocks of IP-addresses.

It’s something the author even mentions:

At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned, due to arbitrary reasons, by mistake, it doesn’t matter. It’s not if, it’s when. Say goodbye to your email. Game over. No recourse.

Once again, yes .. but no. Most people self-hosting their e-mail do so from a cheap virtual server they rented from one of the literal hundreds of service providers out there.

And I understand the appeal, I use these offerings myself. Services like Digitalocean, Linode, Vultr, Bluehost and others are cheap and convenient.

These two major advantages are huge disadvantages at the same time. Single customers are absolutely irrelevant to these hosts, they need the numbers to be economically viable.

They really don’t care who you are and don’t really care if you have abused their service before - just get yourself another mail address and you’re good to register with them again! Which means there will be abuse, a lot of it. Repeatedly, of various different types.

On top of that, at least some providers - even well-known ones - have issues with how they handle abuse. Time for a (admittedly not necessarily representative) story!

When I worked with a national CERT, DigitalOcean started to grow significantly and become vastly popular - thus also increasing the amount of malicious traffic originating from their networks. And the increase was immense.

And I will never forget their response to one of the first abuse messages we sent their way. I genuinely wish I was joking, but their Abuse-team replied with “What do you want us to do with this useless information?”.

I’m sure they receive plenty of bogus abuse reports every day, and even more useless ones lacking key information. But I’d give ourselves the benefit of the doubt, that we were able to send reasonable reports. Especially given the fact that this was the only negative response we got with regards to this specific malicious activity. Plenty of other recipients happily cleaned up their networks.

They have gotten better, but it took them a while to get their processes and procedures right. In the meantime their networks’ reputation suffered greatly. Other popular hosters face similar issues. OVH, several years ago, was close to having almost all of their networks blackholed by upstream service providers because of how much shit came out of their networks.

With experiences like this - and, I’d figure, significant data and statistics to back the experiences up - it makes sense for the ‘big players’ to be wary of messages coming from networks like the ones I just talked about.I think people severely underestimate how relevant IP-reputation (still) is.

You can adhere to all the de-facto standards that exist today, to things like SPF, DKIM, DMARC and even MTA-STS if you are feeling like hurting yourself.

And still, if you are with one of those hosters (aka “with a hoster whose networks are considered harmful”) you will most likely have a bad time, facing delivery issues that don’t really make sense to you.

I tried several times to write about how to deal with this issue, but @stevelord on Mastodon put it better and more concise than I ever could:

@modulux you pick a hosting provider that doesn’t offer free credits or isn’t bargain basement prices. If you want cheap email, don’t self-host (or at least relay through a provider). If you want to self-host email, don’t do it cheaply if you want reliability.

Nowadays, if you want to build services on top of email, you have to pay an email sending API which has been blessed by others in the industry. One of them.

From my, once again personal and limited, experience, the reason why people and companies pick Sendgrid, Mailgun and other services like that is rarely for ensuring deliverability.

It’s for the interface that allows the marketing department to create newsletters or spam advertising campaigns without the need to involve operations. It’s for the analytics and statistics they provide. It’s for the integration into other services, like a CMS they are already using.

The industry must self-establish clear rules which are harsh on spammers but give everybody a fair chance.

I don’t think that this is the issue - we already have rules, regulations, laws on spam. But what good are they if we aren’t enforcing them?

This isn’t the place for a rant about law enforcement, legal issues and the geopolitics of cybercrime, nor am I an expert for those topics. But a lack of rules truly isn’t the problem here - and I don’t think it’s an area that us technies are the deciding factor either.

To avoid a false impression: The author definitely has some points. There are several areas of e-mail that could do with improvements. I’d argue, however, that most of the improvements are not necessarily radical changes, but instead some modifications to how we use existing tools:

• We should stop looking at blacklists in absolutes. This way we avoid, or at least lessen, the problem of suffering from entire address-blocks. Instead blacklist should be another scoring metric, representing general reputation. This would, at the same time, reduce the power blocklist operators have - and which they happily abuse. Yes, I’m looking at you Spamhaus.
• Improve delisting procedures. Right now, whenever I have to delist an address from a blacklist, it’s a 50/50 chance of either having to simply open a website or having to sacrifice my firstborn at the first light of the day when Gandalf arrives from the East. I like the suggestion of the original author, making blocklists “cooldown-based”.
• We should be more clear about why we are rejecting messages. I understand that that might be a difficult thing, especially for the operators of large systems whose filtering process is complex and based on several factors / criteria / sources. But if you’re telling me why exactly you are rejecting my mail, then it’s easier to me to go and fix it, thus improving the Internet for everyone.

I wholeheartedly concur that there have been large mail providers that have engaged in scummy behaviour before. But: We’re not in the horrible communication dystopia (yet?) that the author claims that we are in. I mentioned before that while I’m not as experienced as they are, I’ve been around the block a few times. And we’ve been here before.

Every couple of months, maybe every other year, there’s a blog post or social media comment stating how self-hosting your e-mail is nigh impossible, how e-mail is doomed because of $reason_mostly_involving_google and so on.

Every single time the comments from within the community are similar. I don’t want to call it backlash, because it’s never toxic. There are contradicting comments, plenty of reports from people running / dealing with all sizes of mail systems clearly reporting otherwise.

And while people, myself included, absolutely agree that there are issues with e-mail, I’m pretty convinced that everything is as usual - e-mail is here to stay, and we haven’t lost “the fight” (yet?). And it’s important that we continue to provide alternatives to the “big players”. Giving them up is exactly what helps them.

(Even though I do understand the pain and frustration that the technical side of e-mail causes sometimes. Or causes often .. on second thought, it’s probably pretty close to DNS in that regard.)