Blog

Secure your API from DDoS attacks with NGINX and fail2ban

Hello everyone!

Having a server attacked by a botnet farm its always a very stressful situation, so we always need to have some kind of protection for this kind of issues.
For this solution we are going to use the rate-limiting functionality from NGINX and fail2ban, a program that bans external APIs when they break a certain set of rules. Let’s start!

NGINX configuration


In NGINX is simple, we just need to configure the rate-limiting on our website level.

We went with a configuration of 5 requests per second (5r/s) with extra bursts of 5 requests. This works fine for most APIs, but you might need to tweak it for your configuration.

This allows a total of 10 requests (5 processing and 5 in queue) before our API returns a 503 server error

Let me explain what is going on here:

  1. First, we are setting up the limit_req_zone.
  2. The $binary_remote_addr specifies that we are registering the requests by IP
  3. zone is the name of the limit_req_zone, and 20m is its total size
  4. rate is the permitted rate per IP address. Here we are allowing 5r/s, which translates to 1 request every 200ms.
  5. Then, inside the server directive we use the limit_req_zone referencing it by name.
  6. zone specifies the limit_req_zone to be used, in this case, we named it one
  7. burst is the number of requests that can be queued by the same IP per second, giving us a grand total of 10 requests per IP (5 in process and 5 in queue)
  8. We want our queued requests to be processed as soon as possible, by giving it the nodelay directive when a slot is freed, an item in the queue is going to be processed

If a client goes over the 10 requests limit, NGINX is going to return a 503 (Service Unavailable) error and will record the attempt in the error log. And here is where it becomes interesting :)

If you want to read more about NGINX rate limiting, you can check this link https://www.nginx.com/blog/rate-limiting-nginx/

fail2ban

According to their documentation, fail2ban is:


Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs – too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).

We are going to use fail2ban to scan our NGINX error logs, and if it finds too many occurrences of the same IP, it will ban it for an x amount of time.

First, we need to install fail2ban.

In Debian based distros:

After installing fail2ban, we need to configure our local configuration file. In fail2ban they are called “jails”. We can make a local copy with the following command:

Once we have our local configuration file, we can create our own directive. At the bottom of the /etc/fail2ban/jail.local file, add this configuration:

Here is what’s happening:

  1. First, we name our filter. In this case, our filter is called [nginx-req-limit]
  2. Then we enable our filter with enabled = true
  3. We set our filter to be nginx-limit-req. This is a default filter from fail2ban
  4. We define the action that fail2ban is going to do when it finds a suspicious IP. In this case, it will process it with iptables-multiport and block the http and https ports for that IP address
  5. We tell fail2ban which logfile it is going to scan. In our case, since we are using NGINX our log file is /var/log/nginx/error.log
  6. findtime is the time in which fail2ban is going to limit the searches, and maxretry is the max amount of times an IP can appear on the log before it is banned. In our case, if an IP address appears 2 times in less than 5 seconds on our error log, fail2ban is going to ban it.
  7. And finally, we set our bantime to 300 seconds (5 minutes).

And that’s it! We need to restart fail2ban to see if everything is working correctly:

To check if the service is running, you can run:

It should return something like:

To know more, you can run:

And it will return something like:

And that’s it! It is fully working, you are now protected from DDoS attacks dynamically.

Bonus


What if I want to ban an IP address forever?

We can create another jail in fail2ban to achieve this. On our /etc/fail2ban/jail.local file, add this:

This jail wont pick up anything by itself because we are expecting 99999 errors in less than a second, but it will ban anyone for 1 month. Once we restart fail2ban again, you can manually ban IP addresses with that jail:

If you check the status, you will see something like:

The IP 1.2.3.4 will be banned for 1 month. You can unban it with

Home