We are very sorry that our users have to suffer these problems one by one. Right after architecture migration, which took longer than we expected (almost 18 hours), we have been attacked by a botnet. What is a botnet exactly? A botnet is a large number of computer, IoT and servers that have been hacked. Once the devices have been taken over, it will be known as bot or zombie. Thus, a botnet is a network of zombies or bots that can be controlled remotely by the attacker to do their dirty works.
The Attack Vector
What happened at RunCloud last night (7PM – 2AM GMT+8) was an attack to brute-force email and password. We were down from 7PM until 1:30AM. And 1:30AM – 2AM was a restoration process. We were attacked by 7000+ devices at the same time trying to test email and password combinations. We know this because when the incident happened, we piped all those authentication attempts to a text file in order to see all those tested emails and password. Those passwords were unique to the email. When we checked the tested email, the emails were not found inside our database. So whoever they are, they were testing from an unknown leaked database (dictionary-based attack) and not brute-forcing password one by one.
We built our panel using Laravel and Laravel has very good security to prevent cross-site requests by providing a CSRF token. When we analyze the log, we found that those bots load the page to get the CSRF token and then submit those tokens to the authentication page. Whoever using Laravel, you are not able to mitigate from this attack inside Laravel itself. But you need special equipment in front of that Laravel to prevent this. This is not Laravel’s fault, but that person who creates the attack script can attack any range of framework. It doesn’t matter if you are using NodeJS, Python, Java, Go or even C itself, you can’t run from this.
How We Were Holding
During those attack, our servers were holding just fine, however, we have maxed out the PHP process and thus creating 504 gateway timeout and 502 gateway error by the load balancer. We were thinking about upgrading the server PHP process, but that will create more of a window of opportunity for the attacker to attack.
By reading the webserver log, we were able to distinguish the difference between a true user and a zombie, but we will not be revealing this information. Using the information, we were trying to create middleware to prevent those bots from creating another attack by sending them 404 response. However, those bots do not stop until they found 302, 301 or 200 code to send those valid data back to the attacker. Besides the middleware, we already placed throttle limit when doing authentication, but those bots didn’t honor 429 HTTP code so they keep trying although our servers have sent them 429 code.
If anyone ever runs a DNS query to our domain (runcloud.io), they will find that we were using Digital Ocean DNS. Because of the attack, we moved our DNS to Cloudflare since we are no longer able to withstand the heat coming from those attack. Inside Cloudflare, we use Page Rules to set our authentication page is under attack. Cloudflare did a good job at preventing it. But to prevent those additional attacks, we are using other tools from Cloudflare. Cloudflare successfully blocked those attacks and the last attack happened at 01:25:54AM. We were not using Cloudflare before because we were hoping that by horizontally scaling our server, so we can withstand those attacks. But we had not done it because would create a window of opportunity for the attacker to attack.
The IP Addresses Lists
We have successfully collected the IP Addresses of the bots and please check whether your server, computer, network or IoT devices have been compromised by the attacker to create a botnet. This list consists of 7379 unique IP Addresses that we have piped to
sort | uniq to sort and get a unique IP Address. This is the IP Address that we have collected ourself and not including the address blocked by Cloudflare. So we don’t know the exact infected devices. http://pasted.co/a925f9ba