Your Health Check Is Lying to You

There is an old pilot’s saying: the instruments don’t lie, but they only tell you what they’re built to measure. An altimeter will happily read 3,000 feet while you fly straight into a mountain, because measuring the mountain was never its job.

Most of our monitoring is an altimeter. It measures the wrong thing with total confidence.

Two questions, one answer

Every health check answers a question: is the service up? It pings an endpoint, gets a 200 OK, paints the dashboard green, and goes back to sleep. This is genuinely useful. When the box falls over, you want to know in seconds, not when a customer emails.

But “is the service up?” is not the question your users actually care about. They care about is the answer correct? And those two questions have almost nothing to do with each other.

A service can be up and serving garbage. A database can respond in 4 milliseconds with a number that is confidently, precisely wrong. The HTTP layer has no opinion about whether the bytes it returns are true. It only knows they were returned.

This is the gap where products quietly die. Not in a crash — a crash is loud, a crash pages you at 3am, a crash gets fixed. They die in the green. They die while every chart says fine.

A green dashboard over a rotting field

Let me make this concrete with a pattern I keep running into, anonymized.

Imagine a system that ingests messy real-world text — listings, tickets, support messages, whatever — and classifies each item into a clean category from a reference table. To do the classification it uses fuzzy matching: take the messy input, find the closest entry in the reference table by text similarity, assign it.

Fuzzy matching is wonderful until it isn’t. It has a failure mode that no health check on earth will catch: it always returns something. Ask it to match a string that has no good answer, and it won’t error. It won’t return null. It will hand you the least-bad option with a straight face and a similarity score that looks plausible.

So an input that should be classified as “Type 700” — a modern, distinct variant — gets matched against the closest string in the table, which happens to be “Type 600,” an older one that shares most of its name. The match scores high. The record is written. The endpoint returns 200 OK. The dashboard stays green.

And now your product is lying. Not crashing — lying. It’s showing users a category, a metric, a recommendation built on a label that is simply false. The aggregate even looks reasonable: the stats for “Type 600” come out plausible, because the things hiding under that label are real items with real values. The number is internally consistent and externally wrong. That is the worst kind of wrong, because nothing about it trips an alarm.

I have watched a system run for weeks in exactly this state, every check passing, while a meaningful share of its core records pointed at the wrong thing. The monitoring wasn’t broken. It was answering the question it was built to answer — is the service up? — perfectly. Yes. The service was up. The service was up the whole time it was wrong.

Why “green” is the most dangerous color

The reason this is dangerous, and not just annoying, is psychological. A red dashboard creates urgency. A green dashboard creates trust — and trust is exactly what you don’t want pointed at a system you haven’t actually verified.

Green tells the on-call engineer to stand down. Green tells the product manager the data’s fine, ship the feature. Green tells you, three weeks later when a user finally complains, to first suspect the user, because look — everything’s green. The dashboard becomes an alibi. It launders a data quality problem into “must be something on your end.”

Security people will recognize this immediately, because it’s the same failure as a SIEM that’s ingesting logs but parsing half of them into the wrong fields. The pipeline is “healthy.” Events per second look normal. And your detections are quietly blind, because the field that says user is actually holding a hostname, and nothing checks that the contents mean what the schema claims they mean. Availability is green. Correctness is on fire. Nobody’s looking at correctness.

The fix is a habit, not a tool

The good news is that the cure is cheap. It is not a new platform. It is not a vendor. It is a second question, asked on a schedule, in the same breath as the first.

After “is it up?”, add: “does the data still mean what it claims to mean?”

In practice that’s a handful of cheap assertions running next to your uptime check:

Cross-field consistency. If a record is labeled “Type 600,” does the source text it came from actually contain “600”? When the label and the evidence disagree, that’s a flag — no human review needed to raise it, just to resolve it. In the fuzzy-match case, this single check would have caught nearly every bad record, because the correct answer was sitting right there in the original text the whole time.
Distribution sanity. Did a category’s row count triple overnight? Did a price range suddenly include values that are physically impossible for that item? Outliers at the edges are often the tell that something upstream started mislabeling.
The “impossible value” trap. Pick the things that should never happen — a 1985 product priced like a 2025 one, or a field that’s suddenly null in most new rows when it used to be populated — and alert on those directly. They are far more honest than an uptime ping.
Concurrent stress, not single-request. A health check that fires one request and gets one 200 tells you the happy path works for one user. Fire twenty at once and you learn whether it works under the conditions your users actually create.

None of this is exotic. The reason it doesn’t get built is not difficulty — it’s that the first question is easy and feels complete. A green checkmark is satisfying. Writing the assertion that says “and by the way, prove the data isn’t garbage” requires admitting that up and correct are different, and that you’ve only been measuring one of them.

Measure the mountain

The altimeter isn’t wrong. It’s just answering a smaller question than the one that kills you. Adding a single instrument that looks forward, at the terrain — not down, at the number — is the difference between confident flight and a confident hole in the ground.

So the next time a dashboard tells you everything is fine, ask it the question it isn’t built to answer. Pull one record at random and check that its label matches its evidence. Count the things that should be impossible. If the system is healthy, you’ve spent five minutes confirming it honestly. If it isn’t, you just found out before your users did — which, in the end, is the only kind of monitoring worth the name.

Green is not a destination. It’s a claim. Make it prove itself.