fix(incus): clamp TCP MSS on traffic forwarded through incusbr0 #1

Merged
yuuhikaze merged 1 commit from :fix/incus-mss-clamp into master 2026-10-24 00:22:59 -05:00

Why

Forwarded traffic from Incus guests (Kubernetes nodes and pods) stalls on internet downloads. The rgxr CI runner's clones of code.forgejo.org die at <100 KB/s, and rustup hangs on static.rust-lang.org. Every failed log stops right after the first chunk, at round sizes ("32752 / 49068 / 65504 bytes of body are still expected"). That is a path-MTU black hole: the WAN path MTU is below 1500 and full-size segments are dropped without ICMP. Handshakes succeed and bulk data never arrives.

net.ipv4.tcp_mtu_probing = 1 (modules/features/sysctl.nix) only protects sockets on the server itself. Pod egress is forwarded (pod → Cilium SNAT → incusbr0 → masquerade, docs/kubernetes/networking.md), and nothing clamps MSS anywhere in this repo or the gitops repos. GitHub works because it advertises a lower MSS than most hosts.

What

  • modules/features/incus.nix: new incus-mss-clamp nftables table (inet, forward hook, mangle priority). It lowers the MSS of TCP SYNs entering or leaving incusbr0 to 1360, and only when above 1360.
    • Both directions: our SYN caps what remotes send us; the remote SYN-ACK caps what guests send out.
  • docs/kubernetes/networking.md: Path MTU paragraph under Pod Routing, with the verification command.

Verify

  • Before deploying, on the server: ping -M do -s 1464 -c2 1.1.1.1 should pass and -s 1472 should fail. The largest passing size + 28 = path MTU; MSS = MTU − 40. 1360 is conservative (fits MTU ≥ 1400); tighten it if you like.
  • After nixos-rebuild switch:
    • nft list table inet incus-mss-clamp shows both rules
    • from a pod: curl -o /dev/null -w '%{speed_download}\n' https://static.rust-lang.org/dist/rust-1.99.1-x86_64-unknown-linux-gnu.tar.xz runs at MB/s
  • The ruleset passes nft -c (NixOS also runs checkRuleset at build).

Notes

  • Root cause is inferred from logs plus config, not yet measured with the ping test above.
  • docs-html was not regenerated (tern needs a TTY; the sandbox has none). Run nix develop path:. --command tern once.
## Why Forwarded traffic from Incus guests (Kubernetes nodes and pods) stalls on internet downloads. The rgxr CI runner's clones of `code.forgejo.org` die at <100 KB/s, and rustup hangs on `static.rust-lang.org`. Every failed log stops right after the first chunk, at round sizes ("32752 / 49068 / 65504 bytes of body are still expected"). That is a path-MTU black hole: the WAN path MTU is below 1500 and full-size segments are dropped without ICMP. Handshakes succeed and bulk data never arrives. `net.ipv4.tcp_mtu_probing = 1` (`modules/features/sysctl.nix`) only protects sockets on the server itself. Pod egress is forwarded (pod → Cilium SNAT → `incusbr0` → masquerade, `docs/kubernetes/networking.md`), and nothing clamps MSS anywhere in this repo or the gitops repos. GitHub works because it advertises a lower MSS than most hosts. ## What - `modules/features/incus.nix`: new `incus-mss-clamp` nftables table (`inet`, forward hook, mangle priority). It lowers the MSS of TCP SYNs entering or leaving `incusbr0` to 1360, and only when above 1360. - Both directions: our SYN caps what remotes send us; the remote SYN-ACK caps what guests send out. - `docs/kubernetes/networking.md`: **Path MTU** paragraph under Pod Routing, with the verification command. ## Verify - Before deploying, on the server: `ping -M do -s 1464 -c2 1.1.1.1` should pass and `-s 1472` should fail. The largest passing size + 28 = path MTU; MSS = MTU − 40. 1360 is conservative (fits MTU ≥ 1400); tighten it if you like. - After `nixos-rebuild switch`: - `nft list table inet incus-mss-clamp` shows both rules - from a pod: `curl -o /dev/null -w '%{speed_download}\n' https://static.rust-lang.org/dist/rust-1.99.1-x86_64-unknown-linux-gnu.tar.xz` runs at MB/s - The ruleset passes `nft -c` (NixOS also runs `checkRuleset` at build). ## Notes - Root cause is inferred from logs plus config, not yet measured with the ping test above. - `docs-html` was not regenerated (tern needs a TTY; the sandbox has none). Run `nix develop path:. --command tern` once.
WAN path MTU sits below 1500 and drops full-size segments silently, so
pod downloads (CI runner fetching code.forgejo.org, static.rust-lang.org)
stall after handshake. tcp_mtu_probing only covers host sockets; clamp
forwarded SYNs in both directions to MSS 1360 and document it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tt5Kc71LcLjqfyZWoZosqh
yuuhikaze merged commit c5fe102f10 into master 2026-10-24 00:22:59 -05:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
yuuhikaze/nixos!1
No description provided.