Proxmox - Netfilter POSTROUTING SNAT rules not hit

Proxmox comes with extensive firewall rules, which can be configured and managed through the web interface. The setup creates a number of chains, and the rules that are automatically configured can get extensive very quickly.

Some parts of the firewall configuration must be done manually. Proxmox for example does not support the configuration of NAT tables. If some of the virtual machines or containers are running on a private network and need to communicate to the outside world via the host external IP, then some special setup might be configured.

The bridge interface that is used by the virtual machines in the private network can be created in the /etc/network/interfaces with the following SNAT rule:

auto vmbr0
iface vmbr0 inet static
    bridge_ports none
    address <host address in the private network>
    netmask <network mask in the private network>
    post-up iptables -t nat -A POSTROUTING -o <external_host_interface> -s <private_network -j SNAT --to <external_ip>

If such an interface with the SNAT is created, the initial assumption is that the SNAT rule would make sure that all packets sent out of the <external_host_interface> should be natted to the external IP of the host.

However, this assumption might not be correct. Proxmox creates a large number of rules that are traversed. We can enable tracing for outgoing packets to debug the problem:

iptables -t raw -A PREROUTING -p icmp -j TRACE

If they are traced by enabling the trace rule for ICMP packets, then a trace similar to the following can appear:

0 6 - 19/May/2019:13:20:15 +0200 TRACE: raw:PREROUTING:policy:2 IN=fwbr101i0 PHYSIN=tap101i0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: nat:PREROUTING:policy:2 IN=fwbr101i0 PHYSIN=tap101i0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:FORWARD:rule:1 IN=fwbr101i0 OUT=fwbr101i0 PHYSIN=tap101i0 PHYSOUT=fwln101i0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:PVEFW-FORWARD:rule:4 IN=fwbr101i0 OUT=fwbr101i0 PHYSIN=tap101i0 PHYSOUT=fwln101i0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:PVEFW-FWBR-OUT:rule:4 IN=fwbr101i0 OUT=fwbr101i0 PHYSIN=tap101i0 PHYSOUT=fwln101i0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:PVEFW-FWBR-OUT:return:5 IN=fwbr101i0 OUT=fwbr101i0 PHYSIN=tap101i0 PHYSOUT=fwln101i0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:PVEFW-FORWARD:rule:5 IN=fwbr101i0 OUT=fwbr101i0 PHYSIN=tap101i0 PHYSOUT=fwln101i0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:PVEFW-FORWARD:return:6 IN=fwbr101i0 OUT=fwbr101i0 PHYSIN=tap101i0 PHYSOUT=fwln101i0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:FORWARD:policy:2 IN=fwbr101i0 OUT=fwbr101i0 PHYSIN=tap101i0 PHYSOUT=fwln101i0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: nat:POSTROUTING:policy:2 OUT=fwbr101i0 PHYSOUT=fwln101i0 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: raw:PREROUTING:policy:2 IN=vmbr0 PHYSIN=fwpr101p0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:FORWARD:rule:1 IN=vmbr0 OUT=eth0 PHYSIN=fwpr101p0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:PVEFW-FORWARD:rule:5 IN=vmbr0 OUT=eth0 PHYSIN=fwpr101p0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:PVEFW-FORWARD:return:6 IN=vmbr0 OUT=eth0 PHYSIN=fwpr101p0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0
0 6 - 19/May/2019:13:20:15 +0200 TRACE: filter:FORWARD:policy:2 IN=vmbr0 OUT=eth0 PHYSIN=fwpr101p0 MAC=fa:c2:10:c8:4d:71:92:0f:4c:6d:21:e2:08:00 SRC=10.1.1.1 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=3293 DF PROTO=ICMP TYPE=8 CODE=0 ID=2135 SEQ=0

As can be seen in the last line of the trace, the last rule hit by the packet is in the FORWARD chain. The POSTROUTING rule is not hit, even if the subnet for the internal network is correct for the outgoing ICMP packet. The solution is to assign the fwbr-interfaces that are traversed to conntrack zone 1:

iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1