Scanning 1.7 million Australian domains and finding 1.62 million SPF & DMARC email security issues
Including 542 domains that have authorised any IP address to send emails from their domain.
TL;DR: 58% of Australian domains have some form of security issue with their SPF and DMARC configuration, with 542 domains mistakingly allowing any IP address on the planet to send SPF authenticated emails masquerading as their domain.
About 6 months ago I ran an experiment... I tried to find Australian domains which were vulnerable to IP takeover attacks through dangling SPF IP addresses. This experiment ended up being a huge success and I found 264 Australian organisations were vulnerable to this type of attack.
At the outcome of that experiment, I found myself wondering... what other issues could I find by conducting at-scale scans of Australian SPF & DMARC records? I ended up sitting on this research, but last week I decided to pick up where I left off!
What is SPF? The Senders Policy Framework (SPF) is a public record that organisations publish on their Domain Name Server (DNS) and it typically contains a list of IP addresses that tell receiving organisations where they can expect legitimate and authorised emails to originate from.
What is DMARC? Domain-based Message Authentication, Reporting and Conformance (DMARC) is an email authentication protocol that complements SPF and DKIM. Importantly for the purpose of this research, a hardened DMARC record also mitigates an email spoofing attack known as spf-bypass.
Types of SPF & DMARC Security Issues
There are various types SPF & DMARC related issues that can be abused and each has a varied impact on how they expose a domain to spoofing. For the purpose of this research, I analysed domains against 10 different types of security misconfigurations.
| # | Issue Title | Issue Detail | Severity |
|---|---|---|---|
| 1 | No SPF record exists | There is no SPF DNS record. Mail receivers have no mechanism to determine what the authorised mail servers are. Mail receivers will pass authentication checks with a "None" result, indicating no check could be performed. Spoofed emails are likely to be accepted. | High |
| 2 | No SPF "all" mechanism set or set to "?all" | The "all" mechanism at the end of the end of an SPF record tells receivers how to treat unauthorised (i.e. spoofed) emails - if the mechanism is missing or set to "?all", failed authentication checks will always return a "Neutral" result which many receivers interpret to accept all mail from a domain (including spoofed emails). | High |
| 3 | SPF "+all" (Pass) mechanism set | The "all" mechanism at the end of the end of an SPF record tells receivers how to treat unauthorised (i.e. spoofed) emails - the "+all" setting tells receivers to pass/accept all mail from a domain (including spoofed emails). | Very High |
| 4 | SPF "~all" (SoftFail) mechanism set | The "all" mechanism at the end of the end of an SPF record tells receivers how to treat unauthorised (i.e. spoofed) emails - the "~all" setting tells receivers to 'SoftFail' (i.e. quarantine) emails that fail SPF authentication. In practice however, many email filters only slightly raise the total spam score and accept 'SoftFailed' (i.e. spoofed) emails. | Medium |
| 5 | SPF has too many lookups for receiver validation | The SPF record requires more than 10 DNS lookups for the validation process. The SPF RFC states that maximum 10 lookups are allowed. As a result, recipients may throw a PermError instead of proceeding with SPF validation. Recipients treat these errors differently than a hard or soft SPF fail , but some will continue processing the mail (i.e. accept spoofed emails). | Medium |
| 6 | No SPF sub-domain record exists | The SPF sub-domain policy is a catch-all mechanism used to prevent threat actors from maliciously spoofing sub-domains from which an explicit SPF record hasn't been set. This is typically represented through a DNS entry similar to "* IN TXT v=spf1 -all", effectively telling recipients to block mail if an explicit SPF entry for the sub-domain hasn't been set. | Medium |
| 7 | No DMARC record exists | There is no DMARC DNS Record set for the domain. Spoofed emails utilising an attack technique known as SPF-bypass are likely to be accepted. See FAQs for more information. | High |
| 8 | Insecure DMARC policy 'p' qualifier | The DMARC policy 'p' qualifier is "none". If the DMARC policy is neither "reject" nor "quarantine", spoofed emails utilising an attack technique known as SPF-bypass are likely to be accepted. See FAQs for more information. | High |
| 9 | Insecure DMARC sub-domain 'sp' qualifier | The DMARC policy 'sp' qualifier for sub-domains is set to "none". If the DMARC policy is neither "reject" nor "quarantine", spoofed emails from any sub-domain utilising an attack technique known as SPF-bypass are likely to be accepted. See FAQs for more information. | High |
| 10 | Partial DMARC coverage | The DMARC "pct" value is set to less than '100' (i.e. 100%), meaning the DMARC policy will only be applied to a percentage of incoming mail. A threat actor can continously deliver spoofed emails (via SPF-bypass) until the DMARC policy isn't applied and mail is accepted. See FAQs for more information. | Medium |
Scanning 1.7 Million Australian Domains
Fortunately, from the research I conducted 6 months prior, I already had a domain list ready to go... To understand how I collected this list read the full write-up. After sanitising, deduping and removing domains that didn't have an active DNS server I landed on my target list. 1.7 million Australian domains.
Scaling from one to many for DNS scanning
After some initial testing, I realised that extracting each domain's full email sender supply chain (i.e SPF & DMARC record) one-by-one just wouldn't be feasible. Even with multi-threading it just wouldn't be enough... The issue is, I'm not just going one layer deep into a domains SPF, I'm recursively querying all sub-lookups and on average I'm querying 6 SPF records per domain. That's 10.2 Million DNS requests!!!
I then remembered, AWS Lambda functions are designed for exactly this use-case. They provide the ability to have the same piece of code running 100s of times concurrently. Best of all, not all functions would appear to come from the same IP, meaning I wouldn't be sinkholed by DNS servers for sending excessive requests.
What is a Lambda function? It's a type of AWS cloud compute that runs code in a highly efficient manner - commonly referred to as serverless computing.
Running 400 concurrent lambda functions for 24 hours
With this in mind, I refactored the SPF & DMARC extraction script and built a lambda function around it. After some testing and reading up on AWS rate limits I decided that running 400 concurrent lambda functions in 5 minute intervals was the safest option. Each lambda function would scan 15 domains and save the results into a DynamoDB (NoSQL) database. Using some basic math I realised I'd need to keep these Lambda functions running for 24 hours!
(15 Domains x 400 Concurrent Lambdas) / 5 Minute Intervals = 1,200 Domains a Minute x 60 Minutes = 72,000 Domains an Hour x 24 hours = 1,700,000 Domains
Sifting through the results
After wrapping up the scan, I exported the data and began analysing it. I immediately noticed that the scan had discovered 1,621,112 SPF and DMARC security issues across 979,268 unique domains - representing 58% of Australian domains.
I then noticed that the scan had detected 542 domains had implemented the "+all" mechanism at the end of their SPF record. This essentially tells mail receivers that any IP address can send SPF authenticated emails on behalf of the affected domain.
I have to admit, I wasn't entirely surprised. When all it takes is a "+" symbol instead of a "-","~" or "?" to expose a business to this type of issue, I figured that surely there'd be a few.
What did surprise me is that when digging into the issue, I noticed that 87 of the affected domains (and possibly more) are all customers of a single Melbourne-based IT Managed Service Provider (MSP) who specialises in the sporting industry. In this case, the MSP has mistakingly implemented the "+" symbol instead of a "-". So instead of locking the domains down, they've actually exposed them to the greatest possible SPF issue.
At this point I realised I needed to validate that these issues were real. So I sent myself a single test email, appearing to come from from one of the affected organisations that had used the "+" SPF mechanism.
As suspected, the email passed all SPF checks - evading any spam filtering with the email going directly to my inbox, with the only issue being the inclusion of a single over-permissive character at the end of an SPF record.
What's the impact?
Each of the 542 domains that have implemented the "+" SPF mechanism are significantly more susceptible to business email compromise and phishing-related attacks. Anyone can spin up an email server and begin sending SPF authenticated emails as though they are any of these domains.
As for the remaining 1,620,570 SPF and DMARC security issues, while not as severe, still showcase the poor adoption of email authentication across the country and are likely representative of global adoption. In the lack of technical authentication checks, email receivers and secure email gateways have a decreased ability to spot the spoof.
Wrapping up
Controlling your email supply chain is absolutely critical to ensuring your business and downstream customers aren't introduced to unnecessary risks relating to email-based threats. By utilising the free domain scan tooling created here at CanIPhish, you can visualise the full extent of your email sender and receiver supply chains.
Sebastian Salla
A Security Professional who loves all things related to Cloud and Email Security.