Otdfbt

I have a Raspberry Pi which establishes an SSH proxy to a remote server (VPS) and opens a port on the Pi so I can use it as a SOCKS proxy. This is the command I use to establish the tunnel:

ssh -D 1080 -f -C -q -N user@hostname

This shows you how it is supposed to work:

 ______________________________
 | |
 | Client |
 |______________________________|
 |
 | L2TP over IPSEC
 ________________|_______________ __ 
 | | |
 | VPN (192.168.1.XXX) | |
 |________________________________| |
 ________________|_______________ |-RaspberryPi
 | | | 
 | SOCKS5 (127.0.0.1:1080) | |
 |________________________________|__|
 |
 | SSH tunnel
 ________________|________________
 | |
 | VPS (Amazon EC2) |
 |_________________________________|
 |
 / 
 / 
 the internet 

These are my IP tables:

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 192.168.42.0/24 -o eth+ -j MASQUERADE
-A POSTROUTING -s 192.168.43.0/24 -o eth+ -m policy --dir out --pol none -j MASQUERADE
COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -p udp -m udp --dport 1701 -m policy --dir in --pol none -j DROP
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p udp -m multiport --dports 500,4500 -j ACCEPT
-A INPUT -p udp -m udp --dport 1701 -m policy --dir in --pol ipsec -j ACCEPT
-A INPUT -p udp -m udp --dport 1701 -j DROP
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -i eth+ -o ppp+ -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i ppp+ -o eth+ -j ACCEPT
-A FORWARD -s 192.168.42.0/24 -d 192.168.42.0/24 -i ppp+ -o ppp+ -j ACCEPT
-A FORWARD -d 192.168.43.0/24 -i eth+ -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.43.0/24 -o eth+ -j ACCEPT
-A FORWARD -j DROP
COMMIT

The script "setup-ipsec-vpn" was used to make the VPN.

So my question is: How can I change these IP tables to drop all of the VPN clients' outbound packets unless it is using the SOCKS5 proxy on the Raspberry Pi (:1080)

The answer depends in part on whether you will configure your clients.

In other words, do you intend for the SOCKS proxy to be explicit, or transparent?

If you can configure your client apps to use an explicit proxy, then it should be simple enough (although you may want to have your SOCKS listen on the VPN interface - or create a DNAT rule).

-A INPUT -s 192.168.1.0/24 -d 192.168.1.1 -m tcp -p tcp --dport 1080 -j ACCEPT
-A INPUT -s 192.168.1.0/24 -j DROP
-A FORWARD -s 192.168.1.0/24 -j DROP

If you wanted to have a transparent proxy, I think it may be worth considering introducing haproxy.

This haproxy blog post provides an overview of how to setup transparent binding, which sounds like it might work for your use case.

You may in that case find it easier to have SSH in tunnel mode rather than SOCKS (or add another VPN connection between Pi and gateway).

The specific rule causing your 'issue' right now is, I think, this one:

-A FORWARD -i ppp+ -o eth+ -j ACCEPT

More generally, it seems as though you might want to rewrite your firewall rules to achieve what you want.