Otdfbt

I am running a docker swarm on "swarm.example.com". On the server, there's a container running that can be accessed on "swarm.example.com:3000".

On server "example.com" I'm running an nginx reverse proxy with following rules

server 
 listen 80;
 server_name app.example.com;

 location / 
 proxy_pass http://swarm.example.com:3000;
 

When I try to access app.example.com im getting 502 Bad Gatway error. Am I missing something ?

All servers are running CentOS 7.6

Thanks!

Trying to access the backend via the reverse proxy results in a 502 Bad Gateway error:

$ wget -S --spider http://nginxtest.example.com/
Spider mode enabled. Check if remote file exists.
--2019-05-18 10:12:11-- http://nginxtest.example.com/
Resolving nginxtest.example.com (nginxtest.example.com)... 192.168.15.20
Connecting to nginxtest.example.com (nginxtest.example.com)|192.168.15.20|:80... connected.
HTTP request sent, awaiting response...
 HTTP/1.1 502 Bad Gateway
 Server: nginx/1.12.2
 Date: Sat, 18 May 2019 08:12:11 GMT
 Content-Type: text/html
 Content-Length: 3693
 Connection: keep-alive
 ETag: "5a9e5ebd-e6d"
Remote file does not exist -- broken link!!!

This is most probably because selinux by default doesn't allow outgoing connections for webservers, because that's usually something they don't do.

You will find entries like this in /var/log/nginx/error.log:

2019/05/18 10:12:11 [crit] 1041#0: *5 connect() to 192.168.15.52:3000 failed (13: Permission denied) while connecting to upstream, client: 146.140.37.47, server: _, request: "HEAD / HTTP/1.1", upstream: "http://192.168.15.52:3000/", host: "nginxtest.example.com"

Additionally, you will find entries like this in /var/log/audit/audit.log:

type=AVC msg=audit(1558167131.910:463): avc: denied name_connect for pid=1041 comm="nginx" dest=3000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket permissive=0
type=SYSCALL msg=audit(1558167131.910:463): arch=c000003e syscall=42 success=no exit=-13 a0=8 a1=562671c4eef0 a2=10 a3=7ffcfbc72530 items=0 ppid=1006 pid=1041 auid=4294967295 uid=996 gid=994 euid=996 suid=996 fsuid=996 egid=994 sgid=994 fsgid=994 tty=(none) ses=4294967295 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)

Run the following command to allow nginx to connect to other hosts:

setsebool -P httpd_can_network_connect true

(The parameter -p makes the setting persistent. Otherwise it would be reset after the next reboot.)

And now the proxy works:

$ wget -S --spider http://nginxtest.example.com/
Spider mode enabled. Check if remote file exists.
--2019-05-18 10:15:14-- http://nginxtest.example.com/
Resolving nginxtest.example.com (nginxtest.example.com)... 192.168.15.20
Connecting to nginxtest.example.com (nginxtest.example.com)|192.168.15.20|:80... connected.
HTTP request sent, awaiting response...
 HTTP/1.1 200 OK
 Server: nginx/1.12.2
 Date: Sat, 18 May 2019 08:15:15 GMT
 Content-Type: text/html
 Content-Length: 40
 Connection: keep-alive
 Last-Modified: Sat, 18 May 2019 08:08:16 GMT
 ETag: "5cdfbd70-28"
 Accept-Ranges: bytes
Length: 40 [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

If you want to know more, there is a very detailed article about nginx and selinux on the nginx website.