Otdfbt

I am operating a server hosting a set of services, each run in a separate Docker container. In addition, there is a KVM running pfSense acting as firewall. The firewall has a physical interface that is connected to the external network and a virtual network card that is connected to the internal container network, using MACVLAN Docker-side, so each container has its own IP address, but all of them are in the same subnet.

For security reasons, the containers need to be isolated and shall not be able to communicate with each other principally (just with the external network). For this, MACVLAN is configured in VEPA mode, which allows traffic from and to the parent device, but not to other addresses on the same parent device.

Now, I would like to allow specific traffic between specific containers, so pfSense must route the traffic to the same interface it received the traffic, considering the configured firewall rules (read, if incoming traffic on the internal interface matches a PASS rule it shall be forwarded to a host on the same interface / same subnet).

I can't seem to get that scenario working (no traffic between the hosts on the internal interface, traffic from and to the external network works as expected). Any ideas on how to proceed from here?

Is there any configuration item in FreeBSD in general or pfSense specifically the prevents such scenarios, like "filter traffic on own interface" or "in practice this should not happend because traffic is forwarded on the switch in front of the router because it is the same subnet so do nothing with it"?

Interestingly enough, pfSense does not even reply to the ARP request (which might have the same reason):

[root@server ~]# ip r
default via 10.0.20.1 dev server proto static metric 410 
10.0.20.0/24 dev server proto kernel scope link src 10.0.20.2 metric 410

21:52:49.651286 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28
21:52:50.673895 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28
21:52:51.697860 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28
21:52:52.721992 ARP, Request who-has 10.0.20.4 tell 10.0.20.2, length 28

I would assume a response with the MAC of the 10.0.20.0/24 interface. The trace was made on the firewall on that interface (PING from the firewall to 10.0.20.4 works as expected).

When adding the entry manually I can see the ICMP echo request, but no reply:

[root@server ~]# arp -s 10.0.20.4 02:42:0a:00:14:04
10.0.20.4 ether 02:42:0a:00:14:04 CM server

22:00:21.403515 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 1, length 64
22:00:22.450162 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 2, length 64
22:00:23.473790 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 3, length 64
22:00:24.497803 IP 10.0.20.2 > 10.0.20.4: ICMP echo request, id 5622, seq 4, length 64

It depends in part on how you intend for one container to reference another. It would be worth considering whether this network topology is suitable for your use case and security policy.

If you wish for container A to address container B directly by hostname or IP address, this needs to follow Layer 2 switching and Layer 3 routing:

• If they are in the same subnet then they will only be able to address each other directly through the switch (which will be blocked by VEPA as you suggested). The pfSense host will only see the traffic if:
  
  
  • the traffic destination IP is outside of the container subnet, in which case the container will send the traffic to the default gateway to be routed; or
  
  
  • the traffic is directly addressed to the pfSense hostname/IP address.
  
  


• If you placed each container on a different subnet (or even separate virtual networks) then traffic between them would be routed via the gateway (pfSense). This would place pfSense in control of firewall policy.


• If you can determine a policy of which groups of containers are permitted to access each other directly, it would make more sense to group these into virtual/docker networks; then the problem goes away. In this case, you're aligning the network topology with the security policy. This is generally easier to get right, maintain, and for others to reason with.

You could look at Docker Macvlan's 802.1q trunk bridge mode which might make it easier to attach multiple container networks to the same libvirt/pfSense interface.

Alternatively, it may happen that there are only a very narrow and consistent set of ways in which containers access each other. In this case, you could consider port forwarding. You would:

• identify a specific container service which you wish to be accessible to other containers


• on pfSense, forward incoming traffic on a particular port of the MACVLAN interface to the IP/port of the desired container


• when other containers need to access the service, they use the IP address (and nominated port) of the pfSense host (not the target container)
  

In this case, the other containers don't know anything about where the target service is - it may as well be hosted directly on the pfSense host. Note however that this doesn't scale well, and would only work for simple TCP/UDP traffic (FTP could be painful to set up).

There may well also be some other Docker or libvirt networking features which would allow you to define more detailed firewall policy between containers on a virtual network, though I haven't looked deeply into this myself.