Stop Paging on Error Counts: SLO Alerting for Shared Databases

What SLOs solve for a platform like Cassandra, and why burn-rate alerts replace volume and error-count alerts.

It is 3 a.m. and a page arrives: “Cassandra timeouts above 2,000 per minute.”

Is that bad? At peak traffic it is a rounding error. At 3 a.m., with one tenth of the traffic, it is a real outage. Nobody can tell from the alert. This post explains why that happens, what replaces it, and how to build it for a multi-region Cassandra cluster.

The problem with volume and error-count alerts

The traditional database alert is a threshold on a count or a rate: errors per minute, timeouts per second, CPU above 80%. These alerts fail in four ways.

1. The same number means different things at different times. Take a datacenter that serves 8,000 reads per second at peak and 800 at night. Suppose the alert fires at 2,000 timeouts per minute (about 33 per second). These numbers are illustrative.

• At peak, 33 failures per second is 0.42% of requests.
• At night, the same 33 per second is 4.2% of requests.

The threshold is ten times more sensitive at night than during the day, and neither number says whether users are hurt.

2. Blips page, slow leaks do not. A 30-second burst of errors crosses the threshold and wakes someone. A steady 0.3% failure rate at peak is only 24 failures per second. It never crosses the threshold, yet it uses the whole month’s reliability allowance in about ten days.

3. The alert has no link to a promise. “2,000 per minute” is a number someone chose. Nobody can say what happens to users or to the team’s commitments when it is crossed.

4. Every alert is “critical”. With many cause-based alerts (CPU, pending compactions, dropped messages, error rate), there is no shared way to rank them. Teams get noise, then they learn to ignore pages.

What an SLO changes

An SLO replaces absolute counts with a ratio and a target.

• SLI (indicator): the measurement, such as the share of requests that succeed.
• SLO (objective): the target over a window, such as 99.9% over 30 days.
• Error budget: what the target allows to fail. At 99.9% over 30 days, that is 0.1% of requests, or about 43 minutes of full outage.
• Burn rate: how fast the budget is being used. A burn rate of 1 uses exactly the whole budget in 30 days. A burn rate of 14.4 uses 2% of it in one hour.

A ratio does not depend on traffic. The same 0.3% failure rate is the same burn rate (3×) at peak and at night. The burn rate also answers the question that matters: at this pace, when does the budget run out?

Back to the example: the 0.3% leak that the count alert missed is a burn rate of 3. It uses the budget in 10 days, so it opens a ticket. The 30-second burst uses a tiny part of the budget, so nobody is woken.

Alert on burn rate, and only on burn rate

Each alert needs two windows to agree. The long window shows that the problem is real. The short window shows that it is still happening, so the alert clears soon after recovery. These numbers come from the Google SRE Workbook chapter on alerting on SLOs.

• Page at 14.4× over 1 hour and 5 minutes. This uses 2% of the budget in one hour.
• Page at 6× over 6 hours and 30 minutes. This uses 5% in six hours.
• Ticket at 1× over 3 days and 6 hours. This uses 10% in three days.

The rule: a human is paged only when the error budget is being used fast enough to be breached. Slow burns open a ticket for working hours. Everything else is a dashboard.

For a shared database this has three effects:

• Fewer pages, and each page is tied to budget at risk.
• A common priority language across clusters and teams.
• A budget that is a decision tool. When it reaches zero, the team stops risky change and spends time on reliability.

Databases are not applications

Application SLOs are well understood: pick the requests that matter, define success (no error, under a latency bound), set a target. A database makes this harder, because there is no one-to-one mapping between an application request and a database operation.

• One application request becomes zero, one or many database operations. A cache hit sends none. A page can issue several reads and a write, and the driver can retry.
• One database operation fans out. A client read or write goes to one coordinator node, then to replicas, possibly in other datacenters.
• The cluster is shared. Operations from many applications arrive at the same coordinators, and the metrics carry no application label.

So a database SLO cannot be “the share of application requests that succeed”. It is a promise on the database’s own interface: the share of client reads and writes that complete without a timeout, an unavailable or a failure, inside a latency bound. Application teams then build their own SLO on top and count database failures in their own budget.

This layering helps during an incident. If the application SLO is red and the database SLO is green, look outside the database. If both are red, start at the datacenter named in the alert.

The Cassandra example

The unit is the datacenter. A multi-region cluster is one logical cluster made of several datacenters. Clients connect to their local datacenter, so each one has its own errors, latency and budget. A single cluster-wide ratio is dominated by the busiest datacenter, so a failing small one barely moves it. The SLO is defined per cluster, datacenter and operation (read or write). A cluster-wide view is only a roll-up on the dashboard. Alerts fire per datacenter, so the page names where the problem is.

Get Sangeeth Talluri’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The metrics. Each node publishes JMX ClientRequest metrics for the requests it coordinates. A JMX exporter on each node turns them into Prometheus metrics, labelled with cluster, datacenter and operation. Each request is counted once, at its coordinator, so summing across a datacenter’s nodes does not double count replicas.

• Availability: 1 − (timeouts + unavailables + failures) / requests, from cassandra_clientrequest_timeouts_count, _unavailables_count, _failures_count and _latency_count.
• Latency: the share of 1-minute windows where the worst node’s p99 (cassandra_clientrequest_latency_micros{quantile="99thPercentile"}) is at or below a threshold. It is window-based because the exporter provides quantile gauges, not histograms.

Table, keyspace and replica metrics are not SLIs. They are the drill-down a person uses after a page.

How it is built

One spec in git drives the rules, the alerts and the dashboard.

1. A small spec per cluster. Teams pick a tier and SLIs from a library. They never write PromQL.

cluster: orders
project_id: my-project
lifecycle: shadow          # draft | shadow | ticket | page
tier: gold                 # 99.9% over 30 days
slos:
  - sli: availability
  - sli: read_latency
    threshold_ms: 500
overrides:
  - match: {datacenter: dc-west}
    slos:
      read_latency: {threshold_ms: 2500, reason: "older hardware, measured p99 tail about 1.5 s"}  

New datacenters are found from the metrics and inherit the defaults. An override that changes an objective needs a written reason.

2. Sloth generates the rules. Sloth turns an SLO definition (an error query, a total query, an objective) into Prometheus recording rules and multi-window burn-rate alerts. A generator expands the team’s spec into one Sloth SLO per datacenter and operation.

3. A rules evaluator runs them and writes to Google Managed Prometheus (GMP). The rules are not part of the database. A Prometheus-compatible evaluator reads the counters from GMP every 60 seconds, computes the rules and writes the results back as new series. Recording rules in GMP mean Grafana reads raw and recorded series through one data source, and a 30-day budget panel reads a precomputed ratio and does not scan 30 days of counters on every refresh.

4. GitOps. The generated rules are committed with the spec, so reviewers read the exact rules that will run. Each SLO moves through four states, one pull request per step.

• draft: nothing recorded.
• shadow: rules recorded, no alerts. Dashboard and backtest only.
• ticket: slow burns open tickets.
• page: tickets and pages.

Only ticket and page can notify a person, so a new SLO never wakes anyone before its target is checked against real traffic. Backtesting over past weeks shows whether the target would have been met.

5. One dashboard for every cluster. Every SLO produces the same series, so one Grafana dashboard serves all clusters: objective, 30-day attainment, remaining budget in minutes, burn rates against the page and ticket thresholds, the datacenter roll-up, and the drill-down to node and keyspace.

What an availability and latency SLO does not cover

Be clear about the limits:

• Volume weighting. A high-volume tenant dominates the ratio. A low-volume tenant can suffer while the cluster ratio stays green.
• Operations beyond Read and Write. Range scans and conditional operations are not counted.
• Consistency level. Stronger consistency levels see more unavailable errors, and the ratio does not split them.
• Requests that never reach a coordinator. If every coordinator in a datacenter is unreachable, the series disappears and does not turn red. The design alerts on missing data for this reason.

Cassandra has other failure modes that availability and latency do not show. These are proposed SLIs, not part of the design described above:

• Cross-datacenter freshness. Replicas can drift apart while reads and writes stay fast. Repair completeness and hint backlog are candidate indicators.
• Durability and recovery. Backup success and restore time against a recovery objective.

Some signals are early warnings and not objectives: pending compactions, dropped messages, GC pauses and disk headroom. Keep these as ticket-level alerts. They tell you the budget is about to burn.

Set an error budget policy

An SLO without a policy is a graph. Write down what happens at each level. For example:

• Budget below 50%: review recent changes.
• Budget at zero: freeze risky change on that cluster and spend the time on reliability.

Planned work counts too. Rolling restarts, node replacement and repair load can consume budget, so schedule them with the budget in view.

What this solves

• Pages that mean something. A page means the budget is at risk, in a named datacenter.
• Traffic-independent alerts. The same failure rate gives the same burn rate at peak and at night.
• Slow leaks get caught. They open a ticket days before they become an outage.
• A shared language. Platform owners publish a measured promise per cluster. Application teams design their own objectives against it.
• Reviewable reliability. Every target, override and alert state is a diff with an author and a reason.

Start small

Pick one cluster. Run per-datacenter availability SLOs in shadow with the generated rules and the dashboard, and leave alerts off. After a few weeks, compare attainment with the target, tune thresholds and overrides, then move to ticket and then page. Keep cause-based alerts for capacity as tickets, and let burn rate decide who gets woken up.