My monitoring dashboards inflamed, almost half of the incoming traffic was giving 404 Not Found es. Except, the content was there. Users who tried again from other networks or devices saw the site fine. What had gone on was a DNS caching issue — but more complicated than just “old records”. I’ll walk through how this kind of failure happens, what parts of the stack are involved, how to debug, and how to prevent similar disasters.
The anatomy of the failure
Here’s what we observed:
We changed the hosting IPs for a service (moved to a different server). The domain’s A / CNAME records were updated.
Our DNS provider showed propagation was done. Our internal tests from many geolocations passed.
But many end users, especially in certain ISPs, got 404 errors. The domain resolved (i.e. pingers saw the new IP), but HTTP servers responded “404 Not Found” — often because the request arrived at a server that no longer had those files, or the virtual host configuration didn’t match the requested Host header.
So what happened? Conflict between a aged DNS cache, negative caching, misconfiguration of TTL and load…
