Packet loss in IpSec/GRE tunnel

Elijah: 02 February 2022

This is my setup:

  +--------------------+     +-------------------+  +--------------------+
  | Router B         a +-----+ Router+Firewall C |  | b         Router D |
  | ----------------------------------        |
  |                    +-----+                   |  |                    +--- more nets
  | |     |       +--+        |
  +----|---------------+     +-------------------+  +--------------------+
       |                                                 |                
  +----|---------------+                            +----|---------------+
  |       |                            |        |
  |Server A            |                            |Client E            |
  |                    |                            |                    |
  |                    |                            |                    |
  +--------------------+                            +--------------------+
  • Routers B and C have public IPs on the internet and a IpSec tunnel (Racoon) for
  • All machines are running Ubuntu Linux.
  • There is an embedded GRE tunnel between with the tunnel IPs and
  • The tunnel is required to be able route data from the net to other networks behind router D (eg.
  • Every IP can ping every other IP.
  • If client E opens a web page on server A, TCP/IP handshake and the "GET /" arrive at server A, but the (large) response of server A does not arrive at client E, but is lost "in" the GRE tunnel.
  • I thought we louse large packets due to fragmentation and reduced the MTU on interfaces a and b, in the end down to 1000 bytes, but this did not help.
  • tcpdump on interface a shows the correct HTTP traffic
  • tcpdump on interface b shows the handshake and HTTP traffic from E->A, but does not show the large response packets from A->E.
  • tcpdump on Router+Firewall C shows the GRE packets dropping out of the IpSec tunnel, no large packets arrive
  • HTTP request from B->E works fine.
  • Large packets between D<->B not via the GRE but directly in the IpSec tunnel (i.e. ssh on the interface) work
  • Large packets between B and "more nets" work (via GRE tunnel!).
  • Swapping the roles of A and E does not help. (E as the server, A as the client or large packets in the other direction don't work.)

Now I am stuck. Any advice what to check? Which config would help to diagnose? Thanks a lot!

Rowan: 02 February 2022

The default table is filter, so when you provide a rule like iptables -A FORWARD ... you just put the rule in filter table. There are also several another tables and table mangle is one of them. As of man iptables: "This table is used for specialized packet alteration."

So, for editing a packages (which is done by -j TCPMSS --set-mss) one should use a mangle table, that is why the rule should be like:

iptables -t mangle -A FORWARD -i tun+ -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400