Our Apache web servers got flooded with proxy requests from all over the world, the typical botnet attack that every system administrator loves on a Monday morning. Running on EC2, we could quickly scale to more instances so that our customers would still be able to access our services while we dealt with the attack. AWS’s Security Groups works great for general firewalling, but we needed something lower level for this situation. Because we had multiple EC2 instances running Apache, we needed something that would work the same across all of them.
Since our instances run Ubuntu, we could leverage ufw (Uncomplicated Firewall), an iptables manager, for this task. All requests coming from bots started with “http://”. We could just tail Apache logs, pick all those requests and block that IP. Nice and simple:
The awk script keeps an internal blocked_ips list which ensures that we don’t try adding the same IP address over and over again. Even if ufw won’t re-add an already blocked IP, it’s a costly operation which slows our utility down. The other subtlety would be ufw insert 1 which ensures that all deny rules come at the top, before all allow rules. This is important because if a rule passes, iptables won’t continue evaluating any of the subsequent rules.
Remember to allow your SSH port (usually 22) before starting your firewall, or you will lock yourself out. You will also need to allow all other ports which you’re using, e.g. 80, 443 etc. To see the status of your firewall, run ufw status. I personally prefer the ufw status numbered version.
In just a few hours, we had no less than 1,500 IPs blocked from Russia, the Ivory Coast, France and even US. The attack is far from over, but this was the immediate effect of our shell utility on Apache web traffic:
As a last tip, when you configure Apache proxying, triple-check that ProxyRequests is disabled. In hindsight, we could have avoided this situation altogether had we been more careful with this rule.
