Monitoring

Healthy, Degraded, Unhealthy: Defining Service Health Status

Abstract illustration of healthy, degraded, and unhealthy service status badges

Most status dashboards lie by omission. A service is showing green, so everyone assumes it is fine. But "green" is hiding a service that answers one request in five with a 500, or takes four seconds to load a page that used to take 400 milliseconds. The problem is not the dashboard. It is the model behind it. A binary service health status (up or down) cannot represent the state most real incidents actually live in: not down, but not right either.

Defining a richer service health status model is one of the highest-leverage things a team can do for reliability, and it costs nothing but clear thinking. This post makes the case for a graduated model and shows how to define, trigger, and respond to each state. It builds on the foundations in what is product health monitoring. If the term is new to you, start there.

Why binary "up/down" fails modern SaaS

Binary status made sense when "the service" was a single process on a single box that was either running or crashed. Modern SaaS products are nothing like that. They are compositions of APIs, queues, third-party integrations, background jobs, and caches, any of which can partially fail. In that world, "up" and "down" are the two states your product is least often in.

Consider what a binary model erases:

  • A service serving traffic but with a 12% error rate: up, but customers are hitting failures.
  • A service responding at 5× normal latency because a cache is cold: up, but effectively unusable.
  • A background reconciliation job that is silently three hours behind: the API is up, the product is drifting.
  • A monitoring check that timed out. Is the service down, or is the check broken? Binary can't tell you.

Each of these demands a different response, and a two-state model gives you no vocabulary to distinguish them. You end up either over-alerting (paging on the 12% error rate as if it were a full outage) or under-alerting (waiting for "down" that never technically arrives).

A graduated model: healthy, degraded, unhealthy, unknown

The fix is a small, opinionated set of states that map to distinct realities and distinct responses. Four states cover the vast majority of cases, and not coincidentally, these are the canonical statuses Sentrock uses:

  • Healthy. The service is meeting all its defined expectations. Nothing to do.
  • Degraded. The service is working, but outside its acceptable envelope: slower, partially failing, or missing a non-critical dependency. Customers may notice, but it is not yet an outage.
  • Unhealthy. The service is failing its core function. Customers are affected. This is the state that warrants urgency.
  • Unknown. The health check could not determine status (a timeout, an auth failure to the check itself, missing data). Not the same as healthy, and dangerous to treat as such.

Why four and not three? Because unknown is where silent failures hide. A team that collapses "unknown" into "healthy" will confidently show green while their monitoring is blind. Treating unknown as its own state forces the question: why can't we tell?

Defining each state for a real service

States are only useful if they are defined with concrete, measurable triggers, not vibes. The definitions belong to each service, but the shape is consistent. Here is a worked example for a hypothetical API:

  • Healthy: availability ≥ 99.9% over the last 5 minutes, p95 latency < 800ms, error rate < 1%.
  • Degraded: availability 99.0–99.9%, or p95 latency 800ms–3s, or error rate 1–5%. Any one condition trips it.
  • Unhealthy: availability < 99.0%, or p95 latency > 3s, or error rate > 5%.
  • Unknown: the check timed out, returned no data, or failed to authenticate.

Three principles make these definitions durable. First, use hysteresis: require a condition to persist for a few evaluation cycles before flipping state, so a single blip doesn't page anyone. Second, define states per capability, not per host, because "checkout" and "search" have different tolerances. Third, write the thresholds down where the system enforces them, not in a runbook nobody reads. Codifying these definitions is the whole point. We cover the mechanics of expressing custom conditions in writing custom health checks in Python.

Responding differently to degraded vs. unhealthy

The reason the states matter is that they gate different responses. If every non-green state pages a human at 3am, you have simply moved alert fatigue from "everything is an outage" to "everything is a page." The value of a graduated model is graduated response:

Degraded should notify, not page. Open a low-urgency ticket, post to the team channel, and keep an eye on it. Degraded that persists or worsens escalates to unhealthy on its own via the thresholds. Unhealthy should page the owner, because this is the state that justifies interrupting someone's sleep while customers are being harmed. Unknown should trigger an urgent investigation of the monitoring itself. Unknown for a critical service is a monitoring outage, and a monitoring outage means you are flying blind on something that matters.

Alert fatigue is not caused by too many alerts. It is caused by too many alerts that all demand the same response. Graduated status is the cure.

How a health rule system codifies service health status

Definitions in a wiki decay. Definitions enforced by a system stay true. The value of a product health monitoring platform here is that it turns your written state definitions into rules it evaluates on a schedule. It then rolls the individual rule results up into a single service health status you can trust. In Sentrock, each health rule you write resolves to one of the four canonical statuses (healthy, degraded, unhealthy, or unknown), and a product's overall status is derived from its rules. You set the interval, the thresholds, and the status logic per rule, so "degraded" means exactly what your team decided it means, consistently, every time. You can see how rules, intervals, and status roll-up fit together on the how it works overview.

Because the definitions live in the system, they are visible, reviewable, and shared. A new engineer can read what "unhealthy" means for the payments service instead of guessing, and the on-call response is the same whether the incident happens on a Tuesday afternoon or a Saturday night.

Where teams get service health status wrong

A graduated model is simple to describe and easy to get subtly wrong. The failure modes are consistent enough to name:

  • Too many states. Teams invent "warning," "critical," "minor," "major," and six colors. Every extra state is a new judgment call at 3am. Four states, precisely defined, beat ten states loosely defined.
  • Thresholds nobody agreed on. If the person who set "degraded = p95 > 800ms" never checked whether 800ms is actually a problem for this product, the status is noise. Thresholds should come from the product's real SLOs, not a default.
  • Per-host instead of per-capability. Rolling up health from individual servers tells you infrastructure is fine while the checkout capability is broken. Define status around what customers do, not where it runs.
  • No hysteresis. Without requiring a condition to persist, status flaps between healthy and degraded every evaluation, and the team learns to ignore it. That is the worst outcome.
  • Unknown rendered as green. The most dangerous mistake, because it is invisible. A dashboard that shows green when it actually means "we couldn't check" is worse than no dashboard, because it manufactures false confidence.

Every one of these traces back to the same root: definitions that live in someone's head or a stale doc rather than in a system that enforces them consistently. That is the argument for codifying status, not just discussing it.

Putting it into practice

You can adopt a graduated service health status model without changing a single line of product code:

  1. Pick the four states (healthy, degraded, unhealthy, unknown) as your standard. Resist the urge to invent more, because more states means more ambiguity.
  2. For each important service, write the trigger conditions for degraded and unhealthy in measurable terms. Start rough; refine with real data.
  3. Map each state to a response (notify vs. page vs. investigate) and make that mapping explicit.
  4. Encode the definitions in your monitoring so they are enforced, not aspirational.
  5. Treat unknown as a first-class problem, never as green.

Tip: If you only do one thing, split "down" into "degraded" and "unhealthy." That single distinction, working-but-wrong versus broken, will change how your team triages, and it is the smallest possible step toward a real service health status model.

Clear states are the vocabulary your whole reliability practice is built on. Once they exist, everything downstream (alerting, dashboards, SLAs, incident response) gets sharper, because every part of the organization is finally reasoning about the same well-defined thing. A support engineer, an on-call SRE, and an engineering manager can look at the same "degraded" and know it means the identical, agreed-upon condition rather than three different guesses. When you're ready to enforce them, you can define your first health rule and pick exactly what healthy, degraded, and unhealthy mean for your product.