Mail blacklists are a sad fact of life. They exist because of spam and work like this:
- I send an email from my computer
- my computer connects to the server that hosts my email
- my server connects to the server that hosts the recipient’s mail
- the recipient server looks up my server’s details in some blacklists
- if my server is found in a blacklist then my mail is rejected
Disreputable servers that send a lot of spam end up on blacklists and find that they can’t send mail any more. Problem solved!
What about false positives?
A false positive occurs when a blacklist contains a server that it shouldn’t: a server is labelled as a bad actor when it’s actually polite and friendly. I’ve been lucky so far: I only send a small amount of personal email from my server so haven’t run into trouble.
I have seen this problem under different circumstances:
- I used to send mail from a shared server used by many other people. This worked well for several years until the server started to be blacklisted every few days (almost certainly because another person using the server was sending spam). I eventually had to change my hosting arrangements to resolve the problem. This is a false positive because I wasn’t sending spam – a shared server was blacklisted because of one bad actor.
- My server used to use a blacklist (NJABL) that was shut down. When that occurred my server started rejecting all mail sent to it: every message was marked as spam. I reconfigured the server to stop using the defunct blacklist.
- By way of counterexample: an email account on a server owned by my then employer was compromised and used to send spam. The system worked as it should in this case and the server was blacklisted (taking down our corporate email with it). This was a true positive, but after resetting the password on the compromised account it still took a few days to get the server removed from all the blacklists that listed it.
The second issue described above happened last week to a mail provider used by many of my university friends.
What happened?
I tried to send an email to a contact and received the following bounce message:
Reporting-MTA: dns; aws.simpleigh.com
X-Postfix-Queue-ID: <redacted>
X-Postfix-Sender: rfc822; <redacted>@simpleigh.com
Arrival-Date: Sun, 31 Jan 2021 15:35:49 +0000 (UTC)
Final-Recipient: rfc822; <redacted>@cantab.net
Original-Recipient: rfc822;<redacted>@cantab.net
Action: failed
Status: 5.7.1
Remote-MTA: dns; mta02.prd.rdg.aluminati.org
Diagnostic-Code: smtp; 554 5.7.1 Service unavailable; Client host
[46.137.167.228] blocked using bl.spamcop.netMy server (at the time of writing this is aws.simpleigh.com, with the address 46.137.167.228) is trying to send a message to my friend at cantab.net. Their server (mta02.prd.rdg.aluminati.org) rejected the message because it thinks my server is listed on a blacklist (hosted at bl.spamcop.net).
This was a serious worry: if my server is on a blacklist then I can’t send mail to anyone! I navigated to spamcop.net to see what was going on and found this:
Oops. Their domain name expired and the blacklist went offline. According to the domain’s WHOIS record this happened at 05:00 on 30th January at which point their site was replaced with their domain registrar’s holding page.
Why does an expiring domain break email?
Domain names expire all the time. Microsoft famously allowed hotmail.co.uk to expire in 2003 and Foursquare went completely offline in 2010 for the same reason. Why is this so catastrophic for email? To understand this we need to know more how blacklists work.
Mail blacklists are also known as Domain-Name System-based Blackhole Lists (DNSBL) because they use DNS – the system that we use every day to find sites on the web. DNS converts friendly domain names like google.com into server IP addresses like 216.58.204.46 (which happens to be the server providing google.com for me today).
Mail servers check blacklists by converting server IP addresses into domain names. Here’s what happened while I was trying to email my friend:
- My server (aws.simpleigh.com,
46.137.167.228) connected to mta02.prd.rdg.aluminati.org to send my message - The recipient server reverses the sender address and appends the blacklist domain, in this case obtaining 228.167.137.46.bl.spamcop.net.
- The recipient server made a DNS request for the above address and received a response:
91.195.240.87. - The positive response indicated my server was in the blacklist and my message was rejected.
What is this magic 91.195.240.87? It turns out this is the registrar’s domain holding page! When the domain name expired the registrar started to return that page for any subdomains of spamcop.net. This meant that every server on the internet was temporarily included in the blacklist!
How big was this issue?
Mail servers using the SpamCop blacklist rejected all messages during the outage. This lasted from the expiration of the spamcop.net domain name until its renewal later that day (potentially slightly longer because DNS responses are commonly cached). If their domain expired at 05:00 and was renewed at around 18:00 then the incident might have lasted for 13 hours.
It’s not really possible to determine how many servers are configured to use the SpamCop blacklist. Large mail providers such as Google and Microsoft maintain their own blacklists but many smaller providers will be using public lists provided for free. SpamCop is frequently recommended so is likely to be popular.
What can we do about it?
The uptime of a service can not exceed that of its critical dependencies.
If the above estimate of 13 hours is accurate then SpamCop’s uptime dropped to 98% for the month of February. The cantab.net email service’s uptime cannot exceed this limit: their service was down for the entirety of the SpamCop outage and may have been down at other times for their own maintenance work.
How can we manage this risk?
Contracts
Companies manage risks posed by their suppliers by agreeing formal contracts with documented terms. If cantab.net lost money because a supplier was negligent then perhaps they can recover their losses. In practice this wouldn’t have helped: SpamCop doesn’t work like this.
NO WARRANTY OR LIABILITY: BY USING THE SCBL, OR ANY INFORMATION CONTAINED ON THE SPAMCOP WEBSITE, YOU ACKNOWLEDGE AND AGREE THAT THE SCBL IS PROVIDED “AS IS”, SPAMCOP DOES NOT GUARANTEE THE EFFECTIVENESS OR RESULTS OF THE SCBL OR ANY OTHER SERVICE OR PRODUCT PROVIDED BY SPAMCOP, AND ANY AND ALL WARRANTIES, IMPLIED OR OTHERWISE, ARE EXPRESSLY EXCLUDED. IN NO EVENT SHALL SPAMCOP, OR ITS PARENT, SUBSIDIARIES OR LICENSORS, BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND ARISING OUT OF OR IN CONNECTION WITH YOUR USE OF THE SCBL OR THE SPAMCOP WEBSITE, HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY.
SpamCop.net – SpamCop FAQ: What is the SpamCop Blocking List (SCBL)?
If the risk is significant enough then no doubt commercial blacklists will exist that do provide such contractual guarantees.
Monitoring and alerting
If a server is rejecting 100% of the email it receives then there’s almost certainly something wrong! The rate at which mail is being rejected is an important metric that should be measured. Dispatching an alert to an administrator could allow any problem to be resolved quickly.
Regardless of the technical aspect, it’s prudent for businesses to measure the performance of their suppliers (even free ones)!
Eliminate
If a blacklist is unreliable then we should stop using it. Unfortunately it’s difficult to determine the reliability (or otherwise) of a particular blacklist. MxToolBox lists 94 separate blacklists. How can we know which of these are safe to use? In practice administrators rely on recommendations from others (as described above). SpamCop isn’t a tiny independent company or even a free service but is owned by Cisco Systems: a multinational tech giant. As we’ve observed already this doesn’t guarantee the ability to renew domain registrations on time!
Companies can eliminate the risks posed by third-party providers by bringing work back in-house. Large email providers do exactly this, but the cost of establishing an independent email blacklist is likely to be significant.
Perhaps we should stop using all blacklists? Sadly this isn’t a great option: blacklists are an important weapon in the fight against spam.
Accept
The cost of SpamCop (free) is greatly outweighed by its benefit. Despite last week’s incident the risk of failure remains low. Alternatives are expensive or present the same risks.
Until last week I’d not heard of SpamCop, but based on their reputation I will soon be configuring my email server to use their blacklist. At least they’re unlikely to let their domain name expire again.