Blocking User Agents on a NGINX and Apache Web Server

There are instances where you might want to block specific user agents. In those instances you have a few options available to you depending on the web server (i.e., Apache, NGINX) being used. You also have an IPTables option, but in this article we’ll focus on using the web server as IPTables can be limited with the growth of HTTPS.

NGINX

NGINX allows you to control what user agents and referrers are allowed via the nginx.conf file. You will find the nginx.conf file in the default install location (e.g., /etc/nginx/nginx.conf, /usr/local/nginx/conf/nginx.conf).

User Agents

In this file, in the server block. , you will add something like the following for user agents:

## Block http [user-agent]
if ($http_user_agent ~* (user-agent) ) {
   return 403;
}

The ~* makes it case sensitive. And you can also append multiple user agents using |. So if you wanted to block: java and python, you could write it as such:

if ($http_user_agent ~* "java|python") {
    return 403;
} 

You can also redirect the request to a specific page:

if ($http_user_agent ~* "java|curl|python") {
    return 301 https://yoursite.com/naughtyyou/;
} 

Be sure to restart after making the change.

systemctl restart nginx

Referrers

In this file, in the location block. , you will add something like the following for referrers:

if ($http_referer ~ "example\.com")  {
  return 403;
}  

The same rules apply with redirecting using a 301, using | to string multiple referrers, and case sensitivity using ~*. Lastly, be sure to restart after making the change.systemctl restart nginx

Apache

Apache is a little different, and will require the mod_rewrite module to be enabled. You also have the option to configure the changes in either the configruation (.conf) file for Apache, the virtual hosts file for the domain, or directly in .htaccess for each site.

User Agents

It will look like this regardless of the file you choose using the same example as above with java, and python for user agents.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} java [NC,OR]
RewriteCond %{HTTP_USER_AGENT} python [NC]
RewriteRule . - [R=403,L]

Referrers

It will look like this regardless of the file you choose using the same example as above with example.com as the referrer:

RewriteEngine on
RewriteCond %{HTTP_REFERER} example [NC]
RewriteRule . - [R=403,L]

Be sure to restart after making the change.

systemctl restart apache2