feat: registry pull-through caches for Talos nodes + declarative split horizon #2

Merged
yuuhikaze merged 5 commits from :feat/registry-cache into master 2026-10-24 11:03:59 -05:00

Why

Node-level image pulls (containerd on the Talos nodes) cross the WAN, where some connections lose packets: postgres:17-alpine could not be pulled for 30+ minutes and had to be side-loaded. CI already uses LAN copies; this covers the nodes.

What

  • modules/features/registry-cache.nix (new, server whitelist): two CNCF Distribution registries in proxy mode, both on the mesh address
    • docker.io → 10.100.0.1:5000, anonymous upstream
    • ghcr.io → 10.100.0.1:5001, authenticated upstream (the rgxr images are private)
    • firewall opens them on wg0 only; cache persisted under /var/lib/registry-cache; static registry-cache user; hardened units
  • talos-config/patches/registry-mirrors.yaml: node mirrors for both registries. Talos falls back to upstream when a cache is down, so an outage costs speed, not availability.
  • modules/features/incus.nix: the split-horizon raw.dnsmasq override (address=/yhkze.net/10.100.0.1) is now declared. The incus-split-horizon-dns oneshot rewrites it only on drift, and never fails boot when a cluster member is offline (the documented write constraint).
  • docs: a "Registry Pull-Through Cache" section in networking.md covering design, apply and verify; the patch is added to bootstrap.md's list and gen config; the "restored by hand" note for raw.dnsmasq is replaced.

Before rebuilding: add the GHCR secret (I must not touch the SOPS vault)

# sops hosts/server/secrets/secrets.yaml
registry-cache:
    # GitHub account that owns the token
    ghcr-pull-username: yuuhikaze
    # classic PAT with read:packages only (the same values as OpenBao registry/ghcr-pull)
    ghcr-pull-token: <token>

Rollout

  1. Add the secret, merge, run rebuild on the server
  2. Check: curl -s http://10.100.0.1:5000/v2/_catalog and curl -s http://10.100.0.1:5001/v2/_catalog
  3. Apply the mirrors once per node (see the networking.md section; --mode=no-reboot, registries apply live):
    talosctl -e 10.100.0.10 -n 10.100.0.10 patch mc --mode=no-reboot --patch @talos-config/patches/registry-mirrors.yaml
    talosctl -e 10.100.0.11 -n 10.100.0.11 patch mc --mode=no-reboot --patch @talos-config/patches/registry-mirrors.yaml
    

Validation

nixos-rebuild build --flake path:.#server on the server from this branch (result below); no switch, no eval.

🤖 Generated with Claude Code

## Why Node-level image pulls (containerd on the Talos nodes) cross the WAN, where some connections lose packets: `postgres:17-alpine` could not be pulled for 30+ minutes and had to be side-loaded. CI already uses LAN copies; this covers the nodes. ## What - **`modules/features/registry-cache.nix`** (new, server whitelist): two CNCF Distribution registries in proxy mode, both on the mesh address - `docker.io` → `10.100.0.1:5000`, anonymous upstream - `ghcr.io` → `10.100.0.1:5001`, authenticated upstream (the rgxr images are private) - firewall opens them on `wg0` only; cache persisted under `/var/lib/registry-cache`; static `registry-cache` user; hardened units - **`talos-config/patches/registry-mirrors.yaml`**: node mirrors for both registries. Talos falls back to upstream when a cache is down, so an outage costs speed, not availability. - **`modules/features/incus.nix`**: the split-horizon `raw.dnsmasq` override (`address=/yhkze.net/10.100.0.1`) is now declared. The `incus-split-horizon-dns` oneshot rewrites it only on drift, and never fails boot when a cluster member is offline (the documented write constraint). - **docs**: a "Registry Pull-Through Cache" section in `networking.md` covering design, apply and verify; the patch is added to `bootstrap.md`'s list and `gen config`; the "restored by hand" note for `raw.dnsmasq` is replaced. ## Before rebuilding: add the GHCR secret (I must not touch the SOPS vault) ```yaml # sops hosts/server/secrets/secrets.yaml registry-cache: # GitHub account that owns the token ghcr-pull-username: yuuhikaze # classic PAT with read:packages only (the same values as OpenBao registry/ghcr-pull) ghcr-pull-token: <token> ``` ## Rollout 1. Add the secret, merge, run `rebuild` on the server 2. Check: `curl -s http://10.100.0.1:5000/v2/_catalog` and `curl -s http://10.100.0.1:5001/v2/_catalog` 3. Apply the mirrors once per node (see the networking.md section; `--mode=no-reboot`, registries apply live): ```bash talosctl -e 10.100.0.10 -n 10.100.0.10 patch mc --mode=no-reboot --patch @talos-config/patches/registry-mirrors.yaml talosctl -e 10.100.0.11 -n 10.100.0.11 patch mc --mode=no-reboot --patch @talos-config/patches/registry-mirrors.yaml ``` ## Validation `nixos-rebuild build --flake path:.#server` on the server from this branch (result below); no switch, no eval. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Node image pulls crawl on lossy WAN flows (postgres stalled 30+ min).
Distribution proxies on mesh address serve each image after one WAN
fetch; GHCR instance authenticates for private rgxr images.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tt5Kc71LcLjqfyZWoZosqh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tt5Kc71LcLjqfyZWoZosqh
Value was hand-set Incus state; unit rewrites it only on drift so an
offline cluster member never fails boot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tt5Kc71LcLjqfyZWoZosqh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tt5Kc71LcLjqfyZWoZosqh

status: nixos-rebuild build --flake path:.#server from this branch on the server evaluates and builds every derivation except the sops manifest, which fails as expected:

sops-install-secrets: manifest is not valid: secret registry-cache/ghcr-token ... the key 'registry-cache' cannot be found

sops-nix validates keys at build time, so add the registry-cache secrets before (or in the same push as) merging. Otherwise the server build on master breaks until they exist. No other errors.

status: `nixos-rebuild build --flake path:.#server` from this branch on the server evaluates and builds every derivation except the sops manifest, which fails as expected: ``` sops-install-secrets: manifest is not valid: secret registry-cache/ghcr-token ... the key 'registry-cache' cannot be found ``` sops-nix validates keys at build time, so **add the `registry-cache` secrets before (or in the same push as) merging**. Otherwise the server build on master breaks until they exist. No other errors.
yuuhikaze merged commit 3a930a25ef into master 2026-10-24 11:03:59 -05:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
yuuhikaze/nixos!2
No description provided.