kblack0610/home-config
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(host-access): runbook for reaching nodes when SSH is broken #21

Merged
kblack0610 merged 1 commit from docs/host-access into master 2026-06-03 14:48:17 +00:00
Conversation 0 Commits 1 Files changed 3 +161 -2
kblack0610 commented 2026-06-03 14:48:17 +00:00
Owner
Copy link

During the NUT rollout we hit a wall trying to SSH into worker nodes — the workstation's current ed25519 key wasn't in their authorized_keys (only an old ssh-rsa from a prior workstation was). Couldn't fix it via SSH (chicken-and-egg). Couldn't go through pi5-master (master has no SSH key of its own, can't reach workers either). The path that actually worked: kubectl debug node/<k8s-name> --profile=sysadmin -- chroot /host sh — privileged debug pod on the target node, host fs at /host, no SSH needed.

Documenting the pattern + the four-naming-systems trap that made it harder to find.

What's in this PR

  • docs/host-access.md: full runbook
    • Four-naming-systems table (Ansible alias / DHCP -lan / k8s node name / IP) with current fleet mapping
    • Decision tree: SSH vs kubectl debug
    • kubectl debug node/<k8s> pattern + gotchas (use k8s name not Ansible alias; debug pods don't auto-clean; stdout sometimes swallowed in loops)
    • SSH re-bootstrap recipe (single host + bulk version)
  • CLAUDE.md: "Host access naming" subsection expanded from 2 naming systems to 4 (k8s name was missing — real confusion when kubectl debug node/pi5-worker2 failed); points at the new doc
  • docs/README.md: new index entry under Core Operations

Why this matters

We've hit this twice now (worker1 was originally excluded from nut_clients based on this same wrong-key issue; turned out all 5 workers had the same problem). Without a runbook, the next person — or the next session — will rediscover it from scratch.

Not in scope (followups)

  • Ansible role for managing authorized_keys across the fleet. The kubectl-debug path is the rescue, not the standing-state. Once a role exists, demote the manual recipe to "use only when Ansible can't reach the host."
  • Renaming the Pis from raspberrypi[-suffix] to match their DHCP / Ansible names (collapses 4 names → 2). Bigger change, separate decision.
During the NUT rollout we hit a wall trying to SSH into worker nodes — the workstation's current ed25519 key wasn't in their authorized_keys (only an old ssh-rsa from a prior workstation was). Couldn't fix it via SSH (chicken-and-egg). Couldn't go through pi5-master (master has no SSH key of its own, can't reach workers either). The path that actually worked: `kubectl debug node/<k8s-name> --profile=sysadmin -- chroot /host sh` — privileged debug pod on the target node, host fs at /host, no SSH needed. Documenting the pattern + the four-naming-systems trap that made it harder to find. ## What's in this PR - `docs/host-access.md`: full runbook - Four-naming-systems table (Ansible alias / DHCP `-lan` / k8s node name / IP) with current fleet mapping - Decision tree: SSH vs kubectl debug - `kubectl debug node/<k8s>` pattern + gotchas (use k8s name not Ansible alias; debug pods don't auto-clean; stdout sometimes swallowed in loops) - SSH re-bootstrap recipe (single host + bulk version) - `CLAUDE.md`: "Host access naming" subsection expanded from 2 naming systems to 4 (k8s name was missing — real confusion when `kubectl debug node/pi5-worker2` failed); points at the new doc - `docs/README.md`: new index entry under Core Operations ## Why this matters We've hit this twice now (worker1 was originally excluded from `nut_clients` based on this same wrong-key issue; turned out all 5 workers had the same problem). Without a runbook, the next person — or the next session — will rediscover it from scratch. ## Not in scope (followups) - Ansible role for managing `authorized_keys` across the fleet. The kubectl-debug path is the rescue, not the standing-state. Once a role exists, demote the manual recipe to "use only when Ansible can't reach the host." - Renaming the Pis from `raspberrypi[-suffix]` to match their DHCP / Ansible names (collapses 4 names → 2). Bigger change, separate decision. 
Adds docs/host-access.md covering the kubectl-debug-as-host-shell
escape hatch we ended up using to fix the worker SSH keys during the
NUT rollout. Includes:

- Four-naming-systems table (Ansible alias / DHCP -lan name / k8s node
  name / IP) with the current pi5-master + workers mapping
- Decision tree: when to use SSH vs kubectl debug
- The `kubectl debug node/<k8s-name> --profile=sysadmin -- chroot /host`
  pattern + the gotchas we hit (k8s name vs Ansible alias confusion,
  swallowed stdout in loops, debug pod cleanup)
- Recipe for re-bootstrapping SSH on a node with a stale authorized_keys
- Bulk version for fixing all workers at once

Also updates CLAUDE.md "Host access naming" subsection: adds the k8s
node name column (was missing — caused real confusion when we first
tried `kubectl debug node/pi5-worker2`, which is not a node name),
points at the new doc. Adds index entry in docs/README.md.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kblack0610/home-config!21
No description provided.