Scanning millions of domains and compromising the email supply chain of Australia's most respected institutions

TL;DR: I've taken over IP addresses that can deliver SPF authenticated emails on behalf of Australian Parliament House, University of Sydney, REDACTED, Mirvac, REDACTED and 259 other Australian organisations.

A month prior to writing this article I ran a small experiment... I did a one-time scan on a few hundred Australian domains to see if any of the IP addresses listed in their SPF records overlapped with the public IP ranges offered by Amazon Web Services (AWS). I then checked to see if I could takeover any unused IPs. This experiment ended up being a huge success and I found an Australian IT Managed Service Provider had exposed 190 of it's customers to IP-takeover attacks - read the full write-up.

At the outcome of that experiment, I found myself wondering... If I was able to find the above issue by scanning a few hundred domains, what could I find if I scanned every domain in Australia? So that's exactly what I did.

What is SPF? The Senders Policy Framework (SPF) is a public record that organisations publish on their Domain Name Server (DNS) and it typically contains a list of IP addresses that tell receiving organisations where they can expect legitimate and authorised emails to originate from.

Scanning 1.8 Million Australian Domains

The first hurdle was figuring out how to collect an up-to-date listing of Australian domains. I ultimately aggregated the results from three seperate methods.

Method 1: GitHub! I managed to stumble across an amazing GitHub project called domains that's maintained by a Researcher out of Ukraine. This mechanism ultimately gave me 99% of the domains I ended up scanning.

Method 2: ASX200! Some ASX200 domains were missed in Method 1 as some businesses use a .com top-level domain (TLD) structure instead of .com.au.

Method 3: Sublist3r! As the catch-all, I ran a tool called Sublist3r which discovers domains through publicly available information. In this case I queried information on all domains that use .com.au, .org.au, .net.au, .edu.au and .gov.au as their TLD structure.

After collating, deduping and sanitising the domains, I had my target list... 1.8 million Australian domains.

Scaling from one to many

After some initial testing, I realised that extracting each domain's full email sender supply chain (i.e SPF record) one-by-one just wouldn't be feasible. Even with multi-threading it just wouldn't be enough... The issue is, I'm not just going one layer deep into a domains SPF, I'm recursively querying all sub-lookups and on average I'm querying 6 SPF records per domain. That's 10.8 Million DNS requests!!!

I then remembered, AWS Lambda functions are designed for exactly this use-case. They provide the ability to have the same piece of code running 100s of times concurrently. Best of all, not all functions would appear to come from the same IP, meaning I wouldn't be sinkholed by DNS servers for sending excessive requests.

What is a Lambda function? It's a type of AWS cloud compute that runs code in a highly efficient manner - commonly referred to as serverless computing.

Running 400 concurrent lambda functions for 25 hours

With this in mind, I refactored the supply chain extraction script and built a lambda function around it. After some testing and reading up on AWS rate limits I realised that running 400 concurrent lambda functions in 5 minute intervals was the safest option. Each lambda function would scan 15 domains and save the results into a DynamoDB (NoSQL) database. Using some basic math I realised I'd need to keep these Lambda functions running for 25 hours!

(15 Domains x 400 Concurrent Lambdas) / 5 Minute Intervals = 1,200 Domains a Minute x 60 Minutes = 72,000 Domains an Hour x 25 hours = 1,800,000 Domains

Trawling available AWS IPs

After waiting 25 hours, I exported the supply chain data and filtered down to only look at IP addresses associated with AWS' EC2 IP Address Pools. Doing this gave me an immediate idea of where I should focus my efforts: AWS' ap-southeast-2, eu-central-1, us-east-1, us-west-1 and us-west-2 regions.

What is EC2? It's a type of cloud compute and AWS terminology for a Virtual Machine - commonly referred to as Infrastructure as a Service (IaaS).

Looping through 60,000 AWS IPs in 20 hours

The next hurdle was figuring out how to trawl through AWS IPs. The biggest limitation here was how much I was willing to spend!

To keep prices down, I decided to run 50 t3a.nano EC2 instances across 5 regions and restart them every minute (hypothetically you could scale to 120 EC2 instances in each region and stay within AWS rate limits). With each restart the EC2 instances would get a new public IP and I'd then cross-reference the IP to all the IPs found during the email supply chain extraction process.

After 20 hours of restarting EC2 instances I felt I had a large enough sample-set to begin sifting through the results. Keep in mind, AWS reserve 56,080,253 IPs for EC2 instances. Meaning I've only scanned 0.1% of the address space (approx. 1 in a 1000 IPs), so I've barely cracked the surface!

Sifting through the results

I ultimately found I had compromised the email sender supply chain of 264 Australian organisations. While sifting through the results, there were a few that stuck out.

At this point I was shocked... These are some of the most respected institutions in Australia. I ultimately decided to validate that these issues were real so I sent myself a single test email, appearing to come from Australian Parliament House (aph.gov.au).

As suspected, the email passed all SPF and DMARC checks - evading any spam filtering with the email going directly to my inbox. This is in stark contrast to an otherwise flawlessly configured SPF & DMARC record for aph.gov.au, where the ultimate downfall is the inclusion of a single over-permissive IP address block (highlighted in red below):

What's the impact?

Each of the affected 264 organisations and their downstream customers are significantly more susceptible to business email compromise and phishing-related attacks. Anyone with a credit card can sign-up for an AWS account, cycle through EC2 instances until they get a desirable IP, request AWS to remove any SMTP restrictions and begin sending SPF authenticated emails as though they are any of these organisations.

When we consider the position that some of these organisations are in, we can better understand the impact. Imagine a parliamentary staffer receiving an email that appears to come from a Minister, or a student receiving an email posing as someone from university admissions and so on... The recipients in these cases have no technical mechanism to determine the real from fake.

Wrapping up

Controlling your email supply chain is absolutely critical to ensuring your organisation and downstream customers aren't introduced to unnecessary risks relating to email-based threats. By utilising the free domain scan tooling created here at CanIPhish, you can visualise the full extent of your email sender and receiver supply chains.

If you're in need of active monitoring for domains under your control - please don't hesitate to contact the team at CanIPhish as we provide this as a service.

Note: The organisations identified in this blog post have had the vulnerability responsibly disclosed in coordination with the Australian Cyber Security Centre (ACSC). A 30 day remediation period was provided prior to the blog going live.