Alerts should start a response, not a group chat

An alert arrives: “Something is down.” Five people open five dashboards. Someone asks which environment it is. Someone else asks whether anyone is already looking. The notification has successfully created a meeting.

A useful alert should shorten the distance between a symptom and an action. RMON provides checks, notification channels, priorities and runbook links. The part that turns those controls into a workable response is deciding what the message should cause someone to do.

Start with the person and the action

Before selecting a destination, write down who owns the service and what they should inspect first. An internal DNS resolver, a public API and a branch-office connection may need different responders even if they are all monitored from the same RMON installation.

Give the check a name that identifies the service and environment. Add a useful description and, where appropriate, a runbook link. A runbook can be short: where to find the recent error, which dependency to inspect and when to involve another team. “Ask Alex” is a dependency, not a recovery procedure.

Choose a priority that matches the consequence of failure. Priority is context for your response workflow; it is not a replacement for defining who owns the destination and when it is watched.

Choose conditions you are willing to act on

Start with a concrete definition of failure. An HTTP endpoint may need to return an accepted status and a particular JSON value. A DNS check may need to resolve a record through a selected resolver. The notification is only as useful as the condition behind it.

RMON’s check interval, request timeout and retries affect how a problem is observed. More retries can help avoid treating a brief failed attempt as an incident, but they also affect how quickly a sustained failure is reported. Test the configured behavior rather than promising a detection time from the interval alone.

For checks with a response-time threshold, distinguish a slow successful response from a request that cannot complete. The threshold is in milliseconds; the check interval and request timeout use seconds. Unit confusion is a remarkably productive alert generator.

Connect a destination and assign it to checks

Under Channels, configure a supported destination such as Telegram, Slack, PagerDuty, Mattermost, IncidentRelay or email. Run the channel test and verify receipt at the destination. For email, the installation’s SMTP settings also need to work.

Then select the destination on the check’s Notifications step. Creating a channel does not automatically assign it to every check. This distinction is small enough to overlook and large enough to explain a very quiet incident.

RMON 1.4 no longer provides browser notifications. Use the external channels selected on the check and verify them. Keeping a Dashboard tab open is useful for inspection, but it is not an alert-delivery plan.

Test the incident, not just the Test button

A channel test proves that a test message can reach a destination. It does not exercise the whole path from an agent’s measurement through incident handling to recovery. Use a disposable check to test that path:

1. Confirm a fresh successful result and record the expected destination.
2. Make the disposable target fail the check’s configured condition.
3. Wait for the configured attempts and verify the failure in history and at the destination.
4. Restore the healthy response and verify a fresh result and the recovery notification.
5. Open the runbook as a responder and check that it is accessible and useful.

If a message is missing, compare the check’s assignments, incident history, provider access and receiver logs. The web application sends channel tests, while the result receiver handles incident delivery; their outbound connectivity can differ.

Keep the response current

Repeat the exercise after changing credentials, channel ownership or network access. Review the destinations selected on checks before deleting an old channel. An integration that worked last quarter has not signed a lifetime contract.

Public status pages can give customers a shared view of selected service states. Choose what to publish deliberately, and keep internal instructions and credentials out of public names and descriptions.

The notification guide covers provider setup and verification. Aim for a message that starts a response: a recognizable problem, a responsible destination and a clear first step.