- Discord Performance Community Server and some new server rules - November 27, 2024
- Calling for early access test files for Super Speedy Imports - November 27, 2024
- Cyber Monday Deals Live – grab a bargain before the Scalability Pro licensing model changes! - November 21, 2024
For the past 2 weeks, Super Speedy Plugins has been under a sustained attack from a hacker.
Someone, somewhere, has purchased a massive database of emails + passwords and has been trying to hack into our website using these emails. The scale of this is fairly enormous – in the past week alone, over 2 million hack-attempts were made. Like I said, the biggest WordPress hack attack I have ever seen.
Table of Contents
Why the normal security measures don’t help
Normally, fail2ban would help by allowing us to ban these IP addresses using various traffic filters but this botnet is using hundreds of thousands of IP addresses, so there is very little benefit to banning IP addresses as the botnet just moves onto another IP address.
The amount of traffic was swamping our servers and hurting our regular user traffic, so something really needed to be done, either at the nginx or the fail2ban level.
Using something like Sucuri or WordFence wouldn’t help here either, since they involve a full load of PHP and WordPress which for this level of traffic is very heavy. I did try them, but neither helped and actually slowed things down further.
I couldn’t use the UserAgent to detect them either, as the useragent they are spoofing is the most common type for the most common browser (the lastest version of Chrome).
How I finally solved this and stopped the botnet dead
The attack pattern they were using was to attack our /my-account/ page with GET and POST requests. Both of these were passing email+password to try and discover a usable login. Cookies are required on our site in order to log in, so the pattern I decided to use was to check for specific cookies.
At first I added rules to nginx so that this traffic could be killed, by returning a 444 error, whilst using minimal server resources. But the attacker just moved onto a different IP address. The ultimate solution was to send them a huge file, if I suspected they were attackers:
- Check for cookie
- If cookie doesn’t exist, redirect to other URL
- That URL sets a cookie then redirects back to /my-account/
- If it’s their 2nd time on /my-account/ and still no cookie set, redirect them to download the biggest file I could find
Why it’s important to send them a massively large file
Running a hack like this uses server resources from the hackers computer or botnet. They clearly have resources, and I needed to slow them down. If I just ban the IP address, they’d move on to another IP address, so – instead – I send them a 120GB file.
That file will take at least a few hours for them to download, and given that they were running multiple threads, it holds up their attack for at least a day between requests.
The cost of downloading these massive files will cost them a lot of money through their ISP or hosting and potentially I may even fill up their hard disk (on their PC or server) and crash their script.
There are a few alternatives to sending them massive files, like limiting the number of login attempts per minute or banning their IP address but both of these are circumvented by using a rotating set of IP addresses.
This hacker had upwards of 300,000 IP addresses, so presumably is paying a service to use proxies around the world.
Another alternative would be to try to hack the hacker, but there are many problems with this. First, if you get a false-positive, you could end up hacking one of your own customers. Secondly, it’s illegal.
Redirecting them to a massive file on the other hand is harmless for false-positive real-users – they’ll just cancel the file download – and still has the biggest impact on slowing down the hackers. By my estimate, they had 10 threads running attacking my website – all of these threads quickly starting downloading a 120GB file which uses up all their bandwidth, hopefully gets them kicked from their hosting, and potentially even fills up the hard disk of their PC or server and crashes the script completely. Thankfully, none of this is illegal for me to do. I’m doing the equivalent of a URL shortener and redirecting them to another URL. It just so happens, if robots are visiting this, they’re going to flood their bandwidth and hard disk.
Pretty much immediately upon implementing this solution on my server, the attacks stopped.
Summary of the biggest WordPress hack attack I’ve ever seen
Here is a summary of the attack that we underwent. The primary page they were attacking was our /my-account/ page. They were attempting both GET and POST requests against this page.
Once I had installed the WP fail2ban plugin, I enabled the option to track ‘missing user’ login attempts and found that the hackers have hold of a massive database of email addresses + passwords.
The WP fail2ban plugin logs unauthorised logins to /var/log/auth.log and then the same plugin provides you with a fail2ban script to ban these IP addresses if you wish (you can configure how many retries you’ll allow). That was useful in order to uncover the issue, but it didn’t help with actually combating the attacker in this case, because of the huge number of IP addresses they were using.
In the redacted log listed above, you can see the volume of attempts. For logins, they were attempting 4 or 5 times per second. This was outnumbered by the number of actual requests we were receiving (about 12 per second from the would-be hackers), but these login attempts use the full PHP, Nginx and MySQL stack to serve up the answer. You can’t cache authentication requests.
On the day of the solution above being implemented, my traffic recorded in Google Analytics changed radically.
- Page views – Up 148%
- Unique page views – Up 125%
- Bounce rate – Down 21%
On the bigger plus-side, sales for our plugins have resumed. These guys were using so much resource that, even when I doubled the RAM to the server, they would still consume everything I had available. That affects people viewing uncached pages the most. Your users using uncached pages are normally your most valuable, since they’re typically adding products to their cart.
Summary
If you ever find yourself under attack from a high-resource hacker, consider writing some conditional logic to send this type of traffic a massively huge file. It’s the only legal way I could find to slow and ultimately stop them. WordFence, Sucuri, fail2ban, they all rely on IP address banning. The user-agent of this hacker was of no use in identifying the traffic, so ultimately sending potentially dodgy traffic a huge file was the only way I could stop this attack.
For all users out there, be aware that this hacker has purchased a *huge* file of emails + passwords, so I highly recommend you install something like LastPass and use it to eliminate duplicate passwords from all the websites you use. By my estimates, their email + passwords list numbers in the 10s of millions, so get LastPass and get your online security sorted out.
I have a ‘friend invite’ link for LastPass that gets you 1 month Premium and gets me 1 month premium, although to be honest you don’t need premium, unless you have a team of people in your business.
Here’s the affiliate link:
https://lastpass.com/f?42895282
And here’s the direct link:
If the hacker is out there listening, or if anyone else is interested in replicating this new behaviour to see how it works, just visit our site in incognito mode and then visit https://www.superspeedyplugins.com/my-account/. Just be sure to click cancel to avoid downloading the 120GB file.
And if the hacker is listening – try again mate, I’ll beat you again, no matter what you do. Feel free to comment if you’re brave enough.
why didn’t u use a captcha ?
Because they would still use up nginx, php and sql resource to run the page prior to failing the captcha. If I were to do this again, I would just block the traffic using Cloudflare. Back then I was using my own custom-made stack with fail2ban etc. I no longer rely on fail2ban, although I still have it as a backup plan on my servers.
Great to hear you fixed the issue and stopped that hacker! Awesome work.
Every time a go to the link you specified it came to this:
http://prntscr.com/l4i7pi
Each time I refresh it goes between these two links with same error page.
https://www.superspeedyplugins.com/my-account/
https://www.superspeedyplugins.com/bot-check/
Only plug-ins that I have in Chrome are uBlock Origin and DuckDuckGo privacy essentials.
https://www.superspeedyplugins.com/ –> works ok.
Is this what should have happened? No download were offered to me.
Without cookies enabled you won’t be able to login. Probably your security essentials are stopping the download link opening too.
Thanks for the idea. Would you like to show your config / script?
I’ve listed something close to my script in another reply. Basically, check if cookie set. If not, redirect to /bot-check/ where a cookie gets set, then user gets redirected back to /my-account/?botcheck=1. In nginx, if cookie not set AND botcheck = 1, then redirect them to a large file.
Hi Dave, your protection method seems super cool. I want to do something like this for one of my sites. Can you share more technical information about the realization? I suspect that the download file comes from an external server, not from yours. What do you use to force download the File – PHP with header(“Location: url-to-file”), readfile ($file), or something else? Share if possible and more details about Cookie Verification.
It’s something similar to this:
location /my-account/ { set $botrisk "0"; if ($arg_botchecked) { set $botrisk "${botrisk}1"; } if ($cookie__checkforbots = "-") { set $botrisk "${botrisk}2"; } if ($botrisk = "02") { rewrite ^ https://www.superspeedyplugins.com/bot-check/ redirect; #temporary redirect to page which will set cookie then send user back here with URL param set too break; } if ($botrisk = "012") { rewrite ^ https://s3-eu-west-1.amazonaws.com/planet/full-history/history-latest.osm.bz2 redirect; break; } try_files $uri $uri/ /index.php?$args; }Thanks for the story. Actually, I am doubtful with lastpass, but after your honest review, I believe its safe.
Yeah, I didn’t mean for it to be a review of LastPass, but honestly LastPass is awesome. They encrypt your passwords, so even although your passwords are stored on their servers, they are stored encrypted, and only you can access them. But usefully, when the same password from a different site gets encrypted, they can tell the encrypted version has been used at other sites of yours and they can warn you.
This is probably the number one security issue for most people – re-using the same password at multiple sites. So LastPass helps firstly by spotting these duplicate uses of the same password, but secondly by making it easy to just use a 20 to 30 character key-mash for a password. Most sites I don’t even know what my password is – when I’m creating a new password I just hit CTRL+R to bring up the RUN window, I mash the keyboard until I have a long enough password, I copy it, then paste it into the two password boxes. Then LastPass asks me if it should remember the password and I’m done.
Thanks for sharing 🙂
Love the idea of a massive download file. Good thinking. I think I’ll utilise that next time I have the same problem.
Dude this is genious… hahaha #HackThePlanet