Six Alertmanager and Prometheus settings that cut pages before you buy anything

Start by measuring

Before changing anything, find out which rules make the noise. Prometheus keeps an ALERTS series for every pending and firing alert, so this shows which rules spent the most time firing last week:

topk(10, sum by (alertname) (count_over_time(ALERTS{alertstate="firing"}[7d])))

It counts evaluation samples, not pages, so treat it as a ranking. In most setups a handful of rules account for most of the noise, and fixing three of them beats tuning thirty. You can also paste a week of webhook payloads into the BlipLite Alert Noise Report, which runs in your browser.

1. group_by: one notification per problem

Alertmanager batches alerts that share the labels in group_by into one notification. Grouping by alertname alone merges unrelated services; grouping by every label (group_by: ['...']) turns grouping off, and fourteen crash-looping pods become fourteen pages.

route:
  receiver: default
  group_by: [alertname, service, env]

Pick the labels that describe one problem in your environment. alertname plus a service label and an environment label is a good default.

2. group_wait, group_interval, repeat_interval

These three decide how often a group can notify you. The defaults are 30 seconds, 5 minutes and 4 hours.

• group_wait is how long a new group waits for more alerts before the first notification. 30 to 60 seconds lets a burst land as one page.
• group_interval is how long to wait before notifying about new alerts added to a group that already notified. 5 minutes is sensible.
• repeat_interval is how often an unchanged, still-firing group is re-sent. For anything routed to a pager, 4 hours or more: PagerDuty and incident.io already know it's open.

A short repeat_interval on a warning that never gets fixed is one of the most common sources of overnight noise.

3. for: in the alert rule

for makes a condition hold for a while before the alert fires. Without it, one bad scrape or a 20-second spike pages someone for a problem that's gone before they find their laptop.

- alert: HighErrorRate
  expr: sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
      / sum by (service) (rate(http_requests_total[5m])) > 0.05
  for: 5m
  labels:
    severity: critical

If an alert regularly resolves within a few minutes of firing, its for is too short or its threshold is too tight.

4. keep_firing_for: stop the flapping

An alert that crosses its threshold, drops below, then crosses again sends firing, resolved, firing. Since Prometheus 2.42, keep_firing_for keeps an alert firing for a while after the condition clears, so a brief dip doesn't resolve and re-page.

- alert: TargetDown
  expr: up == 0
  for: 2m
  keep_firing_for: 10m
  labels:
    severity: warning

5. inhibit_rules: don't page for symptoms of a known cause

When a critical alert fires, the warnings that come with it rarely add anything. Inhibition mutes target alerts while a matching source alert is firing.

inhibit_rules:
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: [service, env]

Keep equal tight. Without it, one critical alert anywhere mutes warnings everywhere.

6. Route by severity, not by habit

If everything goes to the pager, severity labels are decoration. Send only what needs a human now to PagerDuty or incident.io, and the rest somewhere it will be read in the morning.

route:
  receiver: team-slack
  routes:
    - receiver: pagerduty
      matchers: [severity="critical"]
    - receiver: team-slack
      matchers: [severity=~"warning|info"]

Then make severity mean something: critical means a customer is affected or about to be, and someone should act now.

Where this stops working

All of this is per Alertmanager and per label set. It doesn't group across Alertmanager, Grafana and webhooks, it doesn't know about business hours unless you build time intervals for each route, and nobody can see why a page did or didn't happen without reading the config. That gap is what BlipLite is for: one paging policy across every source, with the reason recorded for every decision.