Backup and recovery

Back up RMON so you can recover a failed host, restore an earlier configuration or roll back an unsuccessful update. A usable recovery set contains the database, its original application keys, configuration, certificates and the matching application versions.

What to save

Record the RMON, RMON Server and agent versions, database type, deployment directory, service locations and custom mount paths. Take a backup before an update, a host move or changes to authentication and certificates. Choose a routine backup interval based on how much recent monitoring history you can afford to lose.

Scroll horizontally to view the full table.

Keep dated recovery sets outside RMON's mounted data directories, with a protected copy on another host or backup service. Restrict access because backups contain credentials and private keys. Keep the latest verified recovery set until a newer one has passed a restore test.

Docker backup and recovery

Run these commands on the existing Docker host, from its deployment directory, in one shell session. Keep the same Compose project name and file selection. If the installation uses compose.server.yaml, keep COMPOSE_FILE=compose.yaml:compose.server.yaml in .env. Allow space for both a full backup and restored data.

Coordinate a maintenance window. Stop every service writing to the same database, including receivers and schedulers outside this Compose project, while taking the consistent backup. The commands below stop the local Compose services; stop external writers on their own hosts as well.

Prepare a backup

Record whether the current installation works and which services are running. These commands start with running web and proxy containers; if they cannot start, preserve the stopped installation and seek recovery assistance before changing its data. Allow enough space for the current images and a complete data backup. Keep the backup outside the directories mounted as RMON configuration, data or TLS storage.

set -eu
umask 077
RMON_BACKUP_DIR="$PWD/backups/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$PWD/backups"
mkdir -m 700 "$RMON_BACKUP_DIR"
docker compose config --quiet
docker compose ps

Save the images of the running containers before changing .env or pulling another version. Each command must succeed before continuing:

RMON_WEB_CONTAINER=$(docker compose ps -q web)
RMON_PROXY_CONTAINER=$(docker compose ps -q proxy)
test -n "$RMON_WEB_CONTAINER"
test -n "$RMON_PROXY_CONTAINER"
docker image tag "$(docker inspect --format '{{.Image}}' "$RMON_WEB_CONTAINER")" rmon-web:before-update
docker image tag "$(docker inspect --format '{{.Image}}' "$RMON_PROXY_CONTAINER")" rmon-proxy:before-update
docker image save --output "$RMON_BACKUP_DIR/images.tar" rmon-web:before-update rmon-proxy:before-update

When using the optional server in this Compose project, also run:

RMON_SERVER_CONTAINER=$(docker compose ps -q server)
test -n "$RMON_SERVER_CONTAINER"
docker image tag "$(docker inspect --format '{{.Image}}' "$RMON_SERVER_CONTAINER")" rmon-server:before-update
docker image save --output "$RMON_BACKUP_DIR/server-image.tar" rmon-server:before-update

Copy .env and every Compose file used by this installation into the backup directory. Include custom overrides and any separately supplied secret files. These files can contain credentials; keep the directory private.

for RMON_FILE in .env compose.yaml compose.server.yaml compose.override.yaml; do
    if [ -f "$RMON_FILE" ]; then
        cp -p "$RMON_FILE" "$RMON_BACKUP_DIR/"
    fi
done

Stop all RMON services that write to this database, including any result servers or separate schedulers outside this Compose project. Pause other administrators' update operations. Agents can retain undelivered results while receivers are stopped; keep the interruption short and check their disk space.

docker compose stop
docker compose ps --all
docker compose run --rm --no-deps -T --pull never --user 0:0 --entrypoint tar \
  web -C / -czf - etc/rmon var/lib/rmon > "$RMON_BACKUP_DIR/rmon-data.tar.gz"
docker compose run --rm --no-deps -T --pull never --entrypoint tar \
  proxy -C /etc/ssl/certs/rmon -czf - . > "$RMON_BACKUP_DIR/rmon-tls.tar.gz"
tar -tzf "$RMON_BACKUP_DIR/rmon-data.tar.gz" > /dev/null
tar -tzf "$RMON_BACKUP_DIR/rmon-tls.tar.gz" > /dev/null

For the optional server, save its mounted access token as well:

docker compose run --rm --no-deps -T --pull never --user 0:0 --entrypoint cat \
  web /run/secrets/rmon_server_token > "$RMON_BACKUP_DIR/server-token"
test -s "$RMON_BACKUP_DIR/server-token"

The data archive includes the standard configuration, application keys, and SQLite database with its accompanying files. All writers must remain stopped while copying SQLite. Add a separate backup for any custom database/key paths outside /etc/rmon and /var/lib/rmon, or certificates mounted outside the standard TLS directory. Logs are not included; archive them separately if needed.

For PostgreSQL or MySQL, also take and verify a database backup using your database administration tools while RMON writers remain stopped. The file archive above does not back up an external database. Do not continue with database updates until that backup is available.

Finish a routine backup or continue an update

After all archives and any external database backup have succeeded, copy the complete recovery set to protected backup storage. For a routine backup, start the services that were running before maintenance. In the standard web/proxy deployment:

docker compose start web proxy
docker compose ps

Start the optional server service if it was previously running, then restart the intended external writers. Follow recovery verification to confirm monitoring resumed. If you are about to update, leave writers stopped and continue with Apply the update instead.

Restore a Docker backup

Use this procedure to recover a failed installation or return to the version saved with a backup. Start the saved version with its matching database, configuration and application keys before considering an update.

On a replacement host, first install Docker and Compose and copy the recovery set to protected storage. Create a deployment directory with the saved Compose files and .env, keeping the original project name and file selection. Run the following commands from that directory. Prepare the restored storage and secret paths as described below before starting any containers; do not run initialization.

Stop RMON and all external writers again. If the current installation contains newer results, preserve a separate copy first so those results can be retained and reconciled. Restoring the earlier database returns history to that backup's time.

Choose the matching backup directory and load its saved images:

set -eu
umask 077
RMON_BACKUP_DIR=/absolute/path/to/the/chosen/backup
test -d "$RMON_BACKUP_DIR"
docker compose stop
docker image load --input "$RMON_BACKUP_DIR/images.tar"

Load server-image.tar from the same directory if the optional server was used. Restore the saved .env and Compose files to the original deployment directory. Set RMON_WEB_IMAGE=rmon-web:before-update and RMON_PROXY_IMAGE=rmon-proxy:before-update in .env; for the optional server, set RMON_SERVER_IMAGE=rmon-server:before-update. Clear conflicting shell overrides. Keep the current data directories intact while preparing restored data.

Extract your trusted backup into a new, empty directory, preserving ownership:

RMON_RESTORE_DIR="$PWD/backups/restore-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$PWD/backups"
mkdir -m 700 "$RMON_RESTORE_DIR"
sudo tar -xzf "$RMON_BACKUP_DIR/rmon-data.tar.gz" -C "$RMON_RESTORE_DIR"
sudo mkdir -m 700 "$RMON_RESTORE_DIR/tls"
sudo tar -xzf "$RMON_BACKUP_DIR/rmon-tls.tar.gz" -C "$RMON_RESTORE_DIR/tls"

In .env, set RMON_CONFIG_SOURCE to the absolute path of the extracted etc/rmon, RMON_DATA_SOURCE to the extracted var/lib/rmon, and RMON_TLS_SOURCE to the extracted tls directory. Restore separately backed-up files and mount paths too. For the optional server, restore server-token to its configured host path with owner 33:33 and permissions 0600.

For PostgreSQL or MySQL, restore the matching database backup before restarting RMON, preferably into a separate database; update the restored rmon.cfg to select it. Confirm that web and all result servers use that same database. Application keys must come from the same backup as the database.

Check that uid 33 can read the restored config/keys and write the data and log directories, then run:

docker compose config --quiet
docker compose run --rm --no-deps --pull never web check
docker compose up -d --no-build --pull never
docker compose ps

Verify sign-in and receiver readiness first, then restart the intended external result servers and verify fresh check results. Keep the newer data and backups until any results received after the restored snapshot have been reconciled. Do not delete volumes as part of an update or rollback.

Native installations

Back up a Linux package installation

1. Record the installed package versions and save any custom service configuration. Identify the actual configuration, database, key and certificate paths.
2. Stop the web service (apache2 on Ubuntu or httpd on Enterprise Linux), every rmon-server using this database, and any separately managed RMON scheduler. Confirm they have stopped.
3. Create a private, dated backup directory outside the application directories. For the standard configuration and data paths:

set -eu
umask 077
RMON_BACKUP_DIR="$HOME/rmon-backups/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$HOME/rmon-backups"
mkdir -m 700 "$RMON_BACKUP_DIR"
sudo tar -C / -czf "$RMON_BACKUP_DIR/rmon-native.tar.gz" etc/rmon var/lib/rmon
sudo tar -tzf "$RMON_BACKUP_DIR/rmon-native.tar.gz" > /dev/null

Add the configured web certificates (normally /etc/ssl/certs/rmon.crt and /etc/ssl/certs/rmon.key), custom configuration/key locations, service overrides and separate receiver secrets. For an external database, complete its backup before restarting writers. Copying these directories alone covers only a SQLite database stored within them.

After verifying the backup, restart the services that were previously running and check fresh results, or leave them stopped when continuing a planned update.

Restore a native installation

1. Keep all writers stopped and preserve the current data separately. Prepare the matching RMON package versions and their dependencies on the recovery host.
2. Extract the trusted archive into a new, empty staging directory, preserving ownership. Inspect its contents before replacing any installation files.
3. Restore the saved configuration, database and original application keys to their configured paths. Restore custom service overrides, certificates and secret files as part of the same recovery set.
4. For PostgreSQL or MySQL, restore the matching database backup and point the web application and every receiver at it. Keep an isolated recovery test connected only to a test database.
5. Check the service account's read access to configuration and keys, and write access to database and log directories. Start the saved version without running initialization, then complete verification.

Do not run an older application version against a database already changed by a newer release. Restore that older version's matching database first. Apply a later upgrade only after recovery succeeds, using the update guide.

External databases and metrics

For PostgreSQL or MySQL, use the database administrator's backup and restore procedure, including required roles, grants and database settings. Record the snapshot time and database server version. A copy of RMON configuration or Docker volumes does not back up a database on another service or host.

Restore into a separate database where possible, then update the restored web and receiver configuration to use it. Stop all production writers before switching them to the recovered database. Changes and history after the backup time will need separate reconciliation.

If you use VictoriaMetrics, keep its own backup and retention procedure. Enabling VictoriaMetrics or restoring the RMON database does not recreate missing measurements there. Preserve receiver and agent data separately when investigating undelivered results.

Verify recovery before relying on the backup

Test a recovery set on an isolated host with a restored database copy. Prevent the test installation from accepting production agent traffic or sending real incident notifications. Test access using a disposable target and notification destination.

1. Check archive integrity and confirm that all configuration, keys, certificates and external database backups belong to the selected recovery set.
2. Sign in with a local administrator account. Verify expected groups, users, checks, channels and status pages.
3. Test a saved SSH credential and an authenticated test check or notification destination to confirm stored secrets remain usable.
4. Confirm receiver readiness, agent access and a new result from each required location. For a standalone receiver, use its health probes.
5. Check the result timestamps, relevant historical charts and one test notification. An UP label from before recovery is insufficient.
6. Record the backup used, recovery duration and any manual steps. During a production recovery, restore normal traffic only after verification and keep the replaced data until reconciliation is complete.