[infra] CI flakiness: WAN packet loss to GitHub/Hetzner routes — mitigations #26
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
status: root cause of the flaky CI found, measured from inside the server and the runner pod.
Findings
ping -M do: 1464 passes, 1472 fails), the PPPoE value. The MSS clamp (yuuhikaze/nixos#1) is deployed and correct, but it isn't what was breaking CI: pods already advertise MSS 1330.git clone github.com/actions/checkoutfails 4–5 times in 10, both from the server host itself and from the runner pod. The failing connections crawl at ~12 KB/s with retransmissions; the others finish in 1–2 s.code.forgejo.org: 4 of 6 downloads crawl.static.rust-lang.org, crates.io) and Cloudflare (npm, speed test) are fast every time.enp2s0andincusbr0show the server forwards every packet, so the loss is upstream, at the ISP or beyond. No config here can fix that.ghcr.io(GitHub's network) that took more than 10 minutes, and even Docker Hub crawls when a long transfer lands on a bad route.Mitigations
actions: pull mirrors ofactions/checkout,docker/{login,metadata,build-push}-action(8 h sync). CIuses:now points there (!25). 10 of 10 clones from the runner pod succeed in about 1 s.git.yhkze.net/actions/ubuntu:{act-latest,act-22.04}and runner labels pointed at the LAN copies (in progress)imagejob then publishesrgxr-apito GHCRRemaining WAN dependency: pushing
rgxr-apitoghcr.ioand Kargo pulling it are still on GitHub's network, so pushes may be slow or flaky. If they are, the next step is moving the service images to the Forgejo registry; that's a platform decision and needs approval first.Housekeeping for the human:
claude-tmp-*on yuuhikaze, the10.100.0.7WireGuard peer, theclaude_debugSSH key and the wstunnel tunnel when donediag/egressbranch is a throwaway; I'll delete itstatus: rgxr-test is live and verified end to end (Cloudflare → Traefik → Authelia → API). Signed in as
rgxr_e2e_a,/api/v1/mereports grouprgxr,/api/v1/runreturns the CONTRACT example trace, and/api/v1/regex/to-nfareturns 200. Kargo stagetest: Healthy, Freight verified. ArgoCD:rgxr-testandrgxr-prodSynced/Healthy.Landed overnight:
rgxr-api, pushes to GHCR and notifies Kargo (first image build took about 7 min). !18 closed.git.yhkze.net/actions/ubuntu:{act-latest,act-22.04}.local-path-storagenamespace labeled privileged. Every local-path volume had been Pending because Pod Security rejected the provisioner's hostPath helper pods.rgxr/postgresseeded (it was absent, so thergxr-dbExternalSecret failed).actions: pull mirrors of 4 GitHub actions plus registry copies ofcatthehacker/ubuntu,rust:1andpostgres:17-alpine.reconcile-nudgeannotation on the ScaledObject,restartedAtannotations on the keda-operator and runner deployments, andforce-syncon 5 ESO objects.root-applicationswas patched with the value git already declares, so it was left alone.Open, needs a human decision:
postgres:17-alpinecould not be pulled for 30+ minutes; I loaded it into the worker's containerd from the LAN mirror once. A durable fix is a Talosmachine.registries.mirrorspatch pointing docker.io/ghcr.io at the LAN registry. That's a platform change, so it needs approval.rgxr-prodruns themasterimage. It was scaffolded that way and holds no Freight; prod should only takevX.Y.ZFreight.forgejo-runner/values.yaml.claude-tmp-*(all scopes), the WG peer10.100.0.7, theclaude_debugSSH key and the wstunnel tunnel. The e2e users' credentials are still in OpenBaorgxr/e2e(you asked to remove them; still pending).Mitigations shipped: LAN pull mirrors for all used actions (org
actions), LAN copies of every CI base image via build-contexts, cargo/rustup stall retries, MSS clamp (nixos#1), and Talos registry pull-through caches for docker.io + ghcr.io (nixos#2/#3, applied to both nodes and verified). Closing; reopen if flakes recur. 🤖 rgxr backend (Claude)