Prometheus SLO-Based Alerting

Alert fatigue kills incident response effectiveness. SLO-based alerting focuses on user-impacting symptoms rather than infrastructure causes.

Defining SLOs

Start with user-facing metrics:

Service	SLI	SLO Target
API Gateway	Request success rate	99.9%
Payment Service	Request latency p99 < 500ms	99.5%
Search	Availability	99.95%

Recording Rules

Create SLI recording rules in Prometheus:

groups: - name: slo-recording rules: - record: slo:request_success:ratio_rate5m expr: | sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Multi-Window Burn Rate Alerts

Google's SRE book recommends multi-window, multi-burn-rate alerts:

groups: - name: slo-alerts rules: - alert: HighErrorBudgetBurn expr: | slo:request_success:ratio_rate5m < 0.999 and slo:request_success:ratio_rate1h < 0.999 for: 2m labels: severity: critical annotations: summary: "High error budget burn rate detected"

Error Budget Policy

Define what happens when error budget is exhausted:

1. > 50% remaining — Normal development velocity
2. 25-50% remaining — Increase review rigor for risky changes
3. < 25% remaining — Freeze non-critical deployments
4. Exhausted — All hands on reliability improvements

Conclusion

SLO-based alerting transformed our on-call experience from hundreds of noisy alerts to a handful of actionable pages. Start with one service and expand the pattern gradually.