fix(dns): end the recurring DNS outages, and put pi3 in git #207

Merged

kblack0610 merged 5 commits from fix/dns-hardening into master

2026-09-01 19:04:24 +00:00

kblack0610 commented

2026-09-01 19:03:47 +00:00

Owner

Ends the recurring house-wide DNS outages, puts pi3 under Ansible, and adds the monitoring that is now the only mitigation for AdGuard being a single point of failure.

Full reasoning, including the two approaches that were tried and reverted, is in ansible/roles/adguard/README.md.

What was wrong

D1 AdGuard had one upstream and an empty fallback list. 14,716 logged upstream timeouts across 54 distinct days since 2026-03-05; the most recent was a 3-minute blackout on 09-01 that swallowed teams.microsoft.com.
D2 The router runs no-resolv with AdGuard as its only upstream, so pi3 down means the house has no DNS. docs/architecture.md claimed the opposite.
D3 .193 was a dynamic DHCP lease that the router hardcodes as that upstream.
D4 AdGuard saw the whole house as one client, so all of it shared a single 20 qps ratelimit bucket.
D5 6.2 GB of query log on an SD card with no wear telemetry, growing ~50 MB/day.
D6 None of the box was in git. AdGuardHome.yaml existed only on the card.

Verified on the live system

Check	Result
Both Quad9 addresses blackholed	names resolve in 57-122ms (previously 20s then failure), filtering intact
Ansible converge, second run	`changed=0` after AdGuard rewrites the file
Disk reclaimed	27% -> 13% root (4 GB)
Filtering, 18 blocked domains, cache flushed each	0/18 bypassed
Probe regexes	match healthy records, reject the degraded case
Alert expressions	parse against live Prometheus
Container	`(healthy)` on the pinned tag

Two things tried and REVERTED

A public resolver as a dnsmasq fallback cannot work. Measured over 18 queries for domains AdGuard blocks:

`strict-order`	Filtering	Failover
ON	0/18 bypassed	none: uncached names fail after 15s
OFF	18/18 bypassed	works

dnsmasq prefers the faster server and public anycast always beats a Pi 3. There is no setting in between, so pi3 stays a SPOF. Removing it needs a second filtering resolver, not a public one: two AdGuards with strict_order: false gives both properties, because there is nothing left to bypass. That is a follow-up, not this PR.

EDNS Client Subnet does not give per-device attribution. AdGuard uses ECS outbound only and keys clients off the transport source IP (AdGuardHome#1727 open, milestoned v0.108.0). It also forwards an inbound ECS option upstream unstripped, which would disclose LAN addressing to Quad9, and OpenWrt 23.05's dnsmasq has no addsubnet option so the uci key is inert anyway. Per-device visibility needs a topology change (DHCP option 6, or a NAT redirect of :53). The ratelimit side effect is fixed.

Monitoring

Three blackbox DNS probes against the router - the previous checks only hit the Pi, so they could not see the chain - plus four alerts. The canary resolves a domain AdGuard blocks and requires 0.0.0.0; it is the only signal that separates "DNS works" from "DNS works and is still filtering". DNSProbesAbsent covers the probes themselves vanishing.

Two latent bugs fixed on the way

openwrt.sh mangled multi-value uci lists (uci show renders them on one line; the parser stripped only the outer quotes). Invisible with one upstream, permanent false drift with two.
headscale MagicDNS pointed tailnet clients at 10.43.199.233, a ClusterIP from when AdGuard ran in-cluster. No Service has held it since.

Note on the commit range

Branched from origin/master, which was 4 commits ahead of forgejo/master (#46, #47, #49, #50 - merged on GitHub, never deployed, since Flux reconciles only from forgejo). This is deliberate rather than incidental: #49 adds the release label to prometheus-rules-node-health.yaml, and without it the rule this PR extends would apply cleanly and never be evaluated. Merging here also finally ships those four.

Risk

Applied and verified live. Revertable: infrastructure/openwrt/{dhcp,openwrt}.sh snapshot uci before each sync (dhcp-20260901-112811.uci), and the Ansible template task keeps backup: true.

Ends the recurring house-wide DNS outages, puts pi3 under Ansible, and adds the monitoring that is now the only mitigation for AdGuard being a single point of failure. Full reasoning, including the two approaches that were tried and reverted, is in `ansible/roles/adguard/README.md`. ### What was wrong - **D1** AdGuard had one upstream and an empty fallback list. 14,716 logged upstream timeouts across 54 distinct days since 2026-03-05; the most recent was a 3-minute blackout on 09-01 that swallowed `teams.microsoft.com`. - **D2** The router runs `no-resolv` with AdGuard as its only upstream, so pi3 down means the house has no DNS. `docs/architecture.md` claimed the opposite. - **D3** `.193` was a **dynamic** DHCP lease that the router hardcodes as that upstream. - **D4** AdGuard saw the whole house as one client, so all of it shared a single 20 qps ratelimit bucket. - **D5** 6.2 GB of query log on an SD card with no wear telemetry, growing ~50 MB/day. - **D6** None of the box was in git. `AdGuardHome.yaml` existed only on the card. ### Verified on the live system | Check | Result | |---|---| | Both Quad9 addresses blackholed | names resolve in 57-122ms (previously 20s then failure), filtering intact | | Ansible converge, second run | `changed=0` after AdGuard rewrites the file | | Disk reclaimed | 27% -> 13% root (4 GB) | | Filtering, 18 blocked domains, cache flushed each | 0/18 bypassed | | Probe regexes | match healthy records, reject the degraded case | | Alert expressions | parse against live Prometheus | | Container | `(healthy)` on the pinned tag | ### Two things tried and REVERTED **A public resolver as a dnsmasq fallback cannot work.** Measured over 18 queries for domains AdGuard blocks: | `strict-order` | Filtering | Failover | |---|---|---| | ON | 0/18 bypassed | none: uncached names fail after 15s | | OFF | **18/18 bypassed** | works | dnsmasq prefers the faster server and public anycast always beats a Pi 3. There is no setting in between, so pi3 stays a SPOF. Removing it needs a **second filtering resolver**, not a public one: two AdGuards with `strict_order: false` gives both properties, because there is nothing left to bypass. That is a follow-up, not this PR. **EDNS Client Subnet does not give per-device attribution.** AdGuard uses ECS outbound only and keys clients off the transport source IP (AdGuardHome#1727 open, milestoned v0.108.0). It also forwards an inbound ECS option upstream unstripped, which would disclose LAN addressing to Quad9, and OpenWrt 23.05's dnsmasq has no `addsubnet` option so the uci key is inert anyway. Per-device visibility needs a topology change (DHCP option 6, or a NAT redirect of :53). The ratelimit side effect **is** fixed. ### Monitoring Three blackbox DNS probes against the **router** - the previous checks only hit the Pi, so they could not see the chain - plus four alerts. The canary resolves a domain AdGuard blocks and requires `0.0.0.0`; it is the only signal that separates "DNS works" from "DNS works and is still filtering". `DNSProbesAbsent` covers the probes themselves vanishing. ### Two latent bugs fixed on the way - `openwrt.sh` mangled multi-value uci lists (`uci show` renders them on one line; the parser stripped only the outer quotes). Invisible with one upstream, permanent false drift with two. - headscale MagicDNS pointed tailnet clients at `10.43.199.233`, a ClusterIP from when AdGuard ran in-cluster. No Service has held it since. ### Note on the commit range Branched from `origin/master`, which was **4 commits ahead of `forgejo/master`** (#46, #47, #49, #50 - merged on GitHub, never deployed, since Flux reconciles only from forgejo). This is deliberate rather than incidental: #49 adds the `release` label to `prometheus-rules-node-health.yaml`, and without it the rule this PR extends would apply cleanly and never be evaluated. Merging here also finally ships those four. ### Risk Applied and verified live. Revertable: `infrastructure/openwrt/{dhcp,openwrt}.sh` snapshot uci before each sync (`dhcp-20260901-112811.uci`), and the Ansible template task keeps `backup: true`.

kblack0610 added 5 commits

2026-09-01 19:03:47 +00:00

fix(monitoring): alert on the blackbox site probes (#46 ) 21bd258e20

The probes in probe-sites.yaml recorded the 2026-08-17 seven-day outage
correctly and nothing alerted on them, so the data was there the whole time
and no one was told.

Adds homelab-site-availability, deliberately independent of the
homelab_app_health_state chain: it asserts what a visitor experiences end to
end, so it still fires when the fault is in a layer Kubernetes considers
healthy (DNS, cert, tunnel, ingress routing) and cannot be silenced by a bug
in a recording rule.

Carries release: kube-prometheus-stack - without that label the Prometheus CR
never selects the rule, and it is applied, listable, and never evaluated.

docs(iot): forever-document for the home smart-home buildout (#47 ) 72e01864db

Tracks the multi-phase Home Assistant work, the hardware list, the per-room
rollout, and - the part that earns the file - the decisions explicitly ruled
OUT of scope, so they are not re-litigated every time the topic comes up.

fix(monitoring): add release label so node-health rules actually load (#49 ) ebf1d0a13b

The homelab-node-health PrometheusRule (NodeDown, NodeExporterAbsent, and the
new DiskPressure/root-fs alerts) was never loaded by Prometheus: its
ruleSelector matches {release: kube-prometheus-stack} and this rule lacked
that label. It has been dead since creation ~22d ago, so no node-health alert
has ever fired. Adding the label makes the operator select it.

Verified: the label is the only difference from the working stock node-exporter
PrometheusRule.

feat(immich-kiosk): weather overlay (OpenWeatherMap, SOPS config.yaml) (#50 ) 686ccaaa93

Mounts a SOPS-encrypted config.yaml at /config/config.yaml with the weather
block (OWM key + home lat/lon 32.74,-117.16, imperial, forecast, default) — the
one immich-kiosk option with no env form. Weather shows automatically on the
photo screensaver alongside the clock/metadata overlays. All other options
stay as KIOSK_* env (env merges with config.yaml).

Note: a freshly-issued OWM key can take ~10min–2h to activate. Refs Phase 1.6b.

fix(dns): end the recurring DNS outages, and put pi3 in git

checks / fleet-roster (pull_request) Successful in 6s

Details

checks / kustomize-builds (pull_request) Successful in 7s

Details

da0aa2f9fa

AdGuard on pi3 had a single upstream (9.9.9.10) and an empty fallback list, so
whenever Quad9 was unreachable every query in the house burned upstream_timeout
twice and failed. The container log holds 14,716 such timeouts across 54
distinct days since 2026-03-05, most recently a 3-minute blackout on 09-01 that
swallowed teams.microsoft.com and dns.msftncsi.com.

Upstreams are now spread across two operators with a fallback pair behind them,
and upstream_timeout drops 10s -> 3s (the retry was what produced the observed
20.003s stalls). Verified by blackholing both Quad9 addresses on the Pi: names
that previously failed after 20s now resolve in 57-122ms with filtering intact.

Nothing about the box was in git - AdGuardHome.yaml existed only on the SD card,
its node_exporter was installed by hand, and .193 was a DYNAMIC lease that the
router hardcodes as its only DNS upstream. Adds ansible/roles/adguard rendering
both config files, pins the image (was :latest, frozen at v0.107.72 since
March), adds a healthcheck that resolves a local rewrite, caps the container
log, and pins the DHCP reservation. Converges clean on a second run.

Query log retention 90d -> 7d. querylog.json had reached 4.21 GB plus a 2.01 GB
rotation, 64% of an SD card that exposes no wear telemetry, appended at
~50 MB/day. Reclaimed 4 GB (27% -> 13% root). Logs are diagnostics, not records:
they are not backed up. Statistics go the other way, 24h -> 30d.

Two things were tried and REVERTED, both documented in the role README:

  A public resolver as a dnsmasq fallback cannot work. Measured over 18 queries
  for domains AdGuard blocks: with strict-order, 0/18 bypassed but dnsmasq never
  failed over (uncached names just fail after 15s); without it, 18/18 bypassed
  because dnsmasq prefers the faster server and public anycast always beats a
  Pi 3. There is no setting in between. pi3 stays a single point of failure,
  now a monitored one; removing it needs a SECOND FILTERING resolver.

  EDNS Client Subnet does not give per-device attribution. AdGuard uses ECS
  outbound only and keys clients off the transport source IP (AdGuardHome#1727
  still open); it also forwards an inbound ECS option upstream unstripped, and
  OpenWrt 23.05's dnsmasq has no addsubnet option, so the uci key is inert. The
  ratelimit side effect IS fixed: the house shared one 20 qps bucket, and the
  router is now whitelisted.

Monitoring, since it is the whole mitigation rather than a supplement: three
blackbox DNS probes against the ROUTER (previous checks only hit the Pi, so they
could not see the chain) and four alerts. The canary resolves a domain AdGuard
blocks and requires 0.0.0.0, which is the only signal separating "DNS works"
from "DNS works and is still filtering". DNSProbesAbsent covers the probes
themselves disappearing, and StandaloneNodeExporterAbsent closes the same gap
for pi3's metrics.

Also fixes two latent bugs found on the way:

  openwrt.sh mangled multi-value uci lists. `uci show` renders them on one line
  as key='a' 'b', and the parser stripped only the outer quotes, yielding
  "a' 'b". Invisible with one upstream; permanent false drift with two.

  headscale MagicDNS pointed tailnet clients at 10.43.199.233, a ClusterIP from
  when AdGuard ran in-cluster. No Service has held that address since.

docs/architecture.md claimed "OpenWRT falls back through its own dnsmasq" on an
AdGuard outage. It does not, and that line would have misled whoever read it
mid-outage.