gengwg·Oct 30SEV0 San Francisco 2026: Notes from an Infra EngineerI spent October 27 at SEV0, incident.io’s reliability conference, at the SFJAZZ Center in San Francisco. It’s a single-track day about…
gengwg·Oct 26AGNTCon + MCPCon 2026: Notes from an Infra EngineerI spent October 22 at AGNTCon + MCPCon North America in San Jose, the day after PyTorch Conference. I run GPU infrastructure, so I picked…A response icon3A response icon3
gengwg·Oct 24PyTorch Conference 2026: Notes from an Infra EngineerI spent October 20–21 at PyTorch Conference North America in San Jose. I run GPU infrastructure, so I picked talks on distributed training…
gengwg·Oct 24Stop typing eval $(op signin): 1Password CLI with desktop-app sign-in on LinuxFor a long time my workflow with the 1Password CLI was the same in every new terminal: run eval $(op signin), type the master password, run…
gengwg·Oct 15A missing cache tier turned one model rollout into an uplink floodWe run GPU Kubernetes clusters for AI inference workloads. On one of them, a routine model rollout saturated the site’s only 100 Gbps…
gengwg·Oct 15Sign PDFs on Linux with Just FirefoxA sweepstakes form showed up in my inbox today: fill it in, sign it, email it back. On Linux, that usually means printing, scanning, or…
gengwg·Oct 13Removing Public IPs from Kubernetes Control-Plane Nodes Without Locking Yourself OutOn one of our bare-metal Kubernetes clusters, the only machines with public IP addresses were the five control-plane nodes. That is…
gengwg·Oct 4Node NotReady, Host Alive: kubelet Crashlooping on a Cert/Key MismatchFault class: a node goes NotReady while the host is healthy. SSH works, uptime is unbroken, containerd is still running every pod. kubelet…
gengwg·Oct 4The Zeros Lie: Triaging Xid 94 and GPU Requires ResetTriage and escalation flow for an ECC-class GPU Xid, most often Xid 94 (Contained ECC error), and for the GPU requires reset repair-pending…
gengwg·Oct 2Your GPU Node Says Zero GPUs: The NVIDIA Driver Restart DeadlockFault class: a GPU node reports nvidia.com/gpu: 0 allocatable and capacity, while the GPUs are physically healthy and still visible to…
