Otdfbt

Context

I have a Rails app with a nginx server and Passenger.

The app is dynamically generating pages based from the request url: if the url exists in the database the app renders the corresponding page. Or if the url does not exist in the database the app renders a 404 page.

Problem

Many crawlers are trying to find vulnerabilities and request lots of urls (.git, admin/config.php, wp-login.php etc...)

Each of those requests are reaching the Rails app, which is generating hits in the database.

Solution

I am looking for a way to do this:

1. first time a "non existent" url if requested it goes through the Rails app, which responds with a 404


2. nginx caches and remember this url


3. next time the same url is requested, nginx directly respond with 404 status without going through the Rails app

Also when the Rails app is restarted (through Passenger) this cache should be purged.

Tries

• I tried to add fastcgi_cache_valid 404 10m; in the server block, it's not working.


• Also tried proxy_cache_valid 404 10m;
  

As you may guess I'm new to nginx.
Thanks for your help.

Nginx config

server 
 listen ...;

 server_name ...;

 root /path/to/rails/app;

 error_page 404 /404;
 error_page 500 502 503 504 /500;

 # First I tried this, no success so I removed it
 fastcgi_cache_valid 404 10m;

 # Then I tried this, no success so I removed it also
 proxy_cache_valid 404 10m;

 location / 
 gzip_static on;
 etag off;
 charset utf-8;
 add_header Cache-Control "max-age=0, private, must-revalidate";
 add_header Referrer-Policy strict-origin-when-cross-origin;
 add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
 add_header X-Content-Type-Options nosniff;
 add_header X-Frame-Options deny;
 add_header X-XSS-Protection "1; mode=block";

 location = / 
 try_files /cached/index.html @rails;
 

 location / 
 try_files /cached$uri.html /cached$uri $uri @rails;
 
 

 location @rails 
 passenger_enabled on;
 passenger_ruby /path/to/ruby;
 passenger_app_env production;
 

I'm most familiar with caching in a reverse proxy environment, so that is the approach I'd take. Thankfully, Nginx is able to proxy for itself fairly easily:

# define your cache
proxy_cache_path /path/to/cache levels=1:2 keys_zone=cacheName:[metaDataSize] max_size=[maxCacheSize] inactive=60m use_temp_path=off;

http 
 server 
 # Any TLS, caching, or gzipping on this virtual server
 listen ...;
 server_name [Actual Domain];

 location / 
 proxy_pass http://127.0.0.1:80;
 proxy_set_header Host [domain.passenger];

 # Activate and configure caching here
 proxy_cache cacheName;
 proxy_cache_valid 404 10m;
 ...any other proxy settings you want.

 # Forward original request info
 proxy_set_header X-Original-Host $http_host;
 proxy_set_header X-Original-Scheme $scheme;
 proxy_set_header X-Forwarded-For $remote_addr;

 # Gzip if you want
 gzip on;
 gzip_proxied any;

 ...etc
 
 

 server 
 # Any Rails/Passenger configuration on this virtual server
 listen 80;
 server_name [domain.passenger]; 

 # Don't log requests twice
 access_log off; 

 # Only allow local requests
 allow 127.0.0.1;
 deny all;

 location / 
 passenger_enabled on;
 passenger_ruby /path/to/ruby;
 passenger_app_env production;
 
 

Purging the cache just requires running rm -rf /path/to/cache/*, so you could script that into your Rails restart procedures in whatever way pleases you best.