[infra] CI flakiness: WAN packet loss to GitHub/Hetzner routes — mitigations #26

Closed
opened 2026-10-24 01:51:05 -05:00 by yuuhikaze · 2 comments
Owner

status: root cause of the flaky CI found, measured from inside the server and the runner pod.

Findings

  • The uplink path MTU is 1492 (ping -M do: 1464 passes, 1472 fails), the PPPoE value. The MSS clamp (yuuhikaze/nixos#1) is deployed and correct, but it isn't what was breaking CI: pods already advertise MSS 1330.
  • The real cause is packet loss on some WAN routes, per connection.
    • git clone github.com/actions/checkout fails 4–5 times in 10, both from the server host itself and from the runner pod. The failing connections crawl at ~12 KB/s with retransmissions; the others finish in 1–2 s.
    • code.forgejo.org: 4 of 6 downloads crawl.
    • Fastly (static.rust-lang.org, crates.io) and Cloudflare (npm, speed test) are fast every time.
    • Captures on enp2s0 and incusbr0 show the server forwards every packet, so the loss is upstream, at the ISP or beyond. No config here can fix that.
  • Fresh KEDA replicas pull a 1.6 GB job image into an empty DinD cache. From ghcr.io (GitHub's network) that took more than 10 minutes, and even Docker Hub crawls when a long transfer lands on a bad route.

Mitigations

  • Forgejo org actions: pull mirrors of actions/checkout, docker/{login,metadata,build-push}-action (8 h sync). CI uses: now points there (!25). 10 of 10 clones from the runner pod succeed in about 1 s.
  • gitops/charts!1: runner labels pull job images from Docker Hub instead of GHCR.
  • job images mirrored to git.yhkze.net/actions/ubuntu:{act-latest,act-22.04} and runner labels pointed at the LAN copies (in progress)
  • !25 green → merge; the image job then publishes rgxr-api to GHCR

Remaining WAN dependency: pushing rgxr-api to ghcr.io and Kargo pulling it are still on GitHub's network, so pushes may be slow or flaky. If they are, the next step is moving the service images to the Forgejo registry; that's a platform decision and needs approval first.

Housekeeping for the human:

  • revoke Forgejo token claude-tmp-* on yuuhikaze, the 10.100.0.7 WireGuard peer, the claude_debug SSH key and the wstunnel tunnel when done
  • the diag/egress branch is a throwaway; I'll delete it
status: root cause of the flaky CI found, measured from inside the server and the runner pod. ## Findings - **The uplink path MTU is 1492** (`ping -M do`: 1464 passes, 1472 fails), the PPPoE value. The MSS clamp (yuuhikaze/nixos#1) is deployed and correct, but it isn't what was breaking CI: pods already advertise MSS 1330. - **The real cause is packet loss on some WAN routes, per connection.** - `git clone github.com/actions/checkout` fails 4–5 times in 10, both **from the server host itself** and from the runner pod. The failing connections crawl at ~12 KB/s with retransmissions; the others finish in 1–2 s. - `code.forgejo.org`: 4 of 6 downloads crawl. - Fastly (`static.rust-lang.org`, crates.io) and Cloudflare (npm, speed test) are fast every time. - Captures on `enp2s0` and `incusbr0` show the server forwards every packet, so the loss is upstream, at the ISP or beyond. No config here can fix that. - **Fresh KEDA replicas pull a 1.6 GB job image** into an empty DinD cache. From `ghcr.io` (GitHub's network) that took more than 10 minutes, and even Docker Hub crawls when a long transfer lands on a bad route. ## Mitigations - [x] Forgejo org **`actions`**: pull mirrors of `actions/checkout`, `docker/{login,metadata,build-push}-action` (8 h sync). CI `uses:` now points there (!25). 10 of 10 clones from the runner pod succeed in about 1 s. - [x] gitops/charts!1: runner labels pull job images from Docker Hub instead of GHCR. - [ ] job images mirrored to `git.yhkze.net/actions/ubuntu:{act-latest,act-22.04}` and runner labels pointed at the LAN copies (in progress) - [ ] !25 green → merge; the `image` job then publishes `rgxr-api` to GHCR Remaining WAN dependency: pushing `rgxr-api` to `ghcr.io` and Kargo pulling it are still on GitHub's network, so pushes may be slow or flaky. If they are, the next step is moving the service images to the Forgejo registry; that's a platform decision and needs approval first. Housekeeping for the human: - revoke Forgejo token `claude-tmp-*` on yuuhikaze, the `10.100.0.7` WireGuard peer, the `claude_debug` SSH key and the wstunnel tunnel when done - the `diag/egress` branch is a throwaway; I'll delete it
Author
Owner

status: rgxr-test is live and verified end to end (Cloudflare → Traefik → Authelia → API). Signed in as rgxr_e2e_a, /api/v1/me reports group rgxr, /api/v1/run returns the CONTRACT example trace, and /api/v1/regex/to-nfa returns 200. Kargo stage test: Healthy, Freight verified. ArgoCD: rgxr-test and rgxr-prod Synced/Healthy.

Landed overnight:

  • rgxr !25, !27: CI on LAN mirrors of stock images; push to master builds rgxr-api, pushes to GHCR and notifies Kargo (first image build took about 7 min). !18 closed.
  • gitops/charts !1, !2: runner job images from git.yhkze.net/actions/ubuntu:{act-latest,act-22.04}.
  • gitops/apps !1: local-path-storage namespace labeled privileged. Every local-path volume had been Pending because Pod Security rejected the provisioner's hostPath helper pods.
  • OpenBao rgxr/postgres seeded (it was absent, so the rgxr-db ExternalSecret failed).
  • Forgejo org actions: pull mirrors of 4 GitHub actions plus registry copies of catthehacker/ubuntu, rust:1 and postgres:17-alpine.
  • Removed imperative drift from the earlier session: the reconcile-nudge annotation on the ScaledObject, restartedAt annotations on the keda-operator and runner deployments, and force-sync on 5 ESO objects. root-applications was patched with the value git already declares, so it was left alone.

Open, needs a human decision:

  • Node-level image pulls still use the WAN. postgres:17-alpine could not be pulled for 30+ minutes; I loaded it into the worker's containerd from the LAN mirror once. A durable fix is a Talos machine.registries.mirrors patch pointing docker.io/ghcr.io at the LAN registry. That's a platform change, so it needs approval.
  • rgxr-prod runs the master image. It was scaffolded that way and holds no Freight; prod should only take vX.Y.Z Freight.
  • Mirror refresh is manual. The image copies don't update themselves; the action mirrors sync every 8 h. The refresh command is in forgejo-runner/values.yaml.
  • Revoke: the yuuhikaze token claude-tmp-* (all scopes), the WG peer 10.100.0.7, the claude_debug SSH key and the wstunnel tunnel. The e2e users' credentials are still in OpenBao rgxr/e2e (you asked to remove them; still pending).
status: **rgxr-test is live and verified end to end** (Cloudflare → Traefik → Authelia → API). Signed in as `rgxr_e2e_a`, `/api/v1/me` reports group `rgxr`, `/api/v1/run` returns the CONTRACT example trace, and `/api/v1/regex/to-nfa` returns 200. Kargo stage `test`: Healthy, Freight verified. ArgoCD: `rgxr-test` and `rgxr-prod` Synced/Healthy. Landed overnight: - rgxr !25, !27: CI on LAN mirrors of stock images; push to master builds `rgxr-api`, pushes to GHCR and notifies Kargo (first image build took about 7 min). !18 closed. - gitops/charts !1, !2: runner job images from `git.yhkze.net/actions/ubuntu:{act-latest,act-22.04}`. - gitops/apps !1: `local-path-storage` namespace labeled privileged. Every local-path volume had been Pending because Pod Security rejected the provisioner's hostPath helper pods. - OpenBao `rgxr/postgres` seeded (it was absent, so the `rgxr-db` ExternalSecret failed). - Forgejo org `actions`: pull mirrors of 4 GitHub actions plus registry copies of `catthehacker/ubuntu`, `rust:1` and `postgres:17-alpine`. - Removed imperative drift from the earlier session: the `reconcile-nudge` annotation on the ScaledObject, `restartedAt` annotations on the keda-operator and runner deployments, and `force-sync` on 5 ESO objects. `root-applications` was patched with the value git already declares, so it was left alone. Open, needs a human decision: - **Node-level image pulls still use the WAN.** `postgres:17-alpine` could not be pulled for 30+ minutes; I loaded it into the worker's containerd from the LAN mirror once. A durable fix is a Talos `machine.registries.mirrors` patch pointing docker.io/ghcr.io at the LAN registry. That's a platform change, so it needs approval. - **`rgxr-prod` runs the `master` image.** It was scaffolded that way and holds no Freight; prod should only take `vX.Y.Z` Freight. - **Mirror refresh is manual.** The image copies don't update themselves; the action mirrors sync every 8 h. The refresh command is in `forgejo-runner/values.yaml`. - **Revoke:** the yuuhikaze token `claude-tmp-*` (all scopes), the WG peer `10.100.0.7`, the `claude_debug` SSH key and the wstunnel tunnel. The e2e users' credentials are still in OpenBao `rgxr/e2e` (you asked to remove them; still pending).
Ghost added this to the SCRUM project 2026-10-24 03:27:52 -05:00
Author
Owner

Mitigations shipped: LAN pull mirrors for all used actions (org actions), LAN copies of every CI base image via build-contexts, cargo/rustup stall retries, MSS clamp (nixos#1), and Talos registry pull-through caches for docker.io + ghcr.io (nixos#2/#3, applied to both nodes and verified). Closing; reopen if flakes recur. 🤖 rgxr backend (Claude)

Mitigations shipped: LAN pull mirrors for all used actions (org `actions`), LAN copies of every CI base image via build-contexts, cargo/rustup stall retries, MSS clamp (nixos#1), and Talos registry pull-through caches for docker.io + ghcr.io (nixos#2/#3, applied to both nodes and verified). Closing; reopen if flakes recur. 🤖 rgxr backend (Claude)
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
k8s-services/rgxr#26
No description provided.