A monitor that could not tell a dead resolver from a deleted record

Sep 5, 2026

AI Generated ✨

Automation

An alert reported that two mail authentication records had stopped resolving. Both were correct, and had been throughout. What the alert had no way to say was that the machine asking had lost its network at the moment it asked.

  • The lookup command exits with one code, meaning no reply from the server, for two conditions that call for opposite responses: the record is gone, which is urgent, and this machine could not reach a resolver, which is noise. Nothing in the alert separated them, so the line that would report a genuine failure had already been taught to read as a blip.
  • The check now tests the resolver before it tests any record, using a query every recursive resolver answers from its own cache, so a slow authoritative server cannot make a working resolver look dead. When the resolver does not answer, the alert says that once and names it instead of listing every record as missing. The timestamp recording when the records were last actually read is deliberately not advanced, so a run that measured nothing cannot later be read as a clean one.
  • Behind it sat a larger gap. The monitor runs at two fixed times a day, and if the machine is asleep at one the scheduler defers it, so the following summary is an ordinary pass and looks the same as a day when both ran. The external dead man's switch did not cover this, because it fires on every run including a late catch up. Missed slots are now reported, read from the live schedule rather than from a copy kept in the code.
  • Both changes were tested against the failures they were written for rather than only compiled. The resolver probe was run against a working resolver and an unroutable one, where it returns the same code the ambiguous failure produces. The slot logic was replayed against the original incident, where it reports the run as thirty one minutes late, and against a run three minutes late, where it correctly stays silent.