Skip to content

Backup Verification

The Backup Verification feature in ClusterControl restores a backup onto a standalone Backup Verification Server (BVS), outside the source cluster, to confirm the backup is actually recoverable rather than just "successful." This is the standalone-host path introduced in Restore Backup → Restoring vs. Verifying; this page covers it in full.

Note

Backup Verification requires an Enterprise license.

There are two ways to trigger verification:

  • On demand, against an existing backup. Use this to spot-check a specific backup, recover a specific piece of data onto an isolated host, or confirm a configuration change doesn't break restorability.
  • Automatically, as part of a backup schedule, either immediately after the scheduled backup completes or on a delay of up to 24 hours. This is the preferred approach for most environments, since it continuously proves your backups are usable rather than checking them only when you happen to remember to. See Schedule Backup → Backup verification for the verify_backup/verify_backup_delay job_data fields used to configure this.

Supported backup methods

Cluster type Supported backup methods
MySQL-based clusters
  • mysqldump
  • Percona Xtrabackup (full, incremental)
MariaDB-based clusters
  • mariadb-dump
  • MariaDB Backup (full, incremental)
PostgreSQL-based clusters
  • pg_dump
  • pg_dumpall
  • pg_basebackup (full, incremental)
  • pgBackRest (full, incremental, differential)
Redis-based clusters
  • RDB - Sentinel topology only, not Cluster mode
Elasticsearch
  • Elasticsearch snapshot - verified differently; see Elasticsearch below

Note

MongoDB (mongodump, percona-backup-mongodb) and SQL Server (mssqlfull/mssqldiff/mssqllog) do not support backup verification.

Prerequisites

  • The backup must have completed successfully. Failed or incomplete backups cannot be verified.
  • The target Backup Verification Server (BVS) must be reachable by the ClusterControl controller, with the same OS-level access your database nodes use.
  • Sufficient disk space on the BVS to hold a full copy of the restored data.
  • Network connectivity between the ClusterControl controller and the source cluster where the backup is stored.
  • Appropriate user permissions and privileges on the BVS. If SELinux or AppArmor is enabled, either configure it to allow the restore or use the Disable SELinux/AppArmor option described below.
  • For best compatibility, use a BVS running the same OS and version as the source database server the backup was taken from.

Common Wizard Steps

Backup Verification uses the same wizard as Restore Backup, choosing Restore and verify on standalone host at Step 2 instead of Restore on node.

Opening the wizard

  1. Go to the Clusters dashboard and click your target cluster.
  2. Go to the Backups tab, keeping the view set to All Backups (default).
  3. Click the action menu (...) on the backup you want to verify and select Restore.
  4. Complete Step 1: Configuration as described in Restore Backup.

Step 2: How to restore

  • Choose Restore and verify on standalone host.

Click Continue.

Step 3: Settings

  • Restore backup on: FQDN or IP address of the target BVS. The host must not already be part of the cluster.
  • Temporary directory: Staging area for the backup files during restore. Allow a bit more space than the size of a full datadir.
  • Install software: Defaults to On. Installs the database software and its dependencies on the BVS from the package repository before restoring. Turn Off only if the BVS already has the correct database software installed (for example, a custom build, or an offline environment without a package repository).
  • Disable firewall: Defaults to On (recommended).
  • Disable SELinux/AppArmor: Defaults to On. If you turn this Off, make sure SELinux/AppArmor is configured to allow the database to create files and open sockets.
  • Shutdown the server after the backup is restored: Defaults to Off (recommended). When Off, the BVS stays up and online after verification, with its data and packages intact; it's then listed under ClusterControl GUI → Nodes as a Backup Verification Server, and you're responsible for decommissioning it yourself. When On, ClusterControl terminates the BVS once verification finishes and removes its datadir and configuration files; Install software must be On the next time you verify against the same BVS.
  • Start backup verification (after completion) in: Only shown when verification is configured as part of a backup schedule, not for on-demand verification. Number of hours to wait after the backup completes before starting verification, up to 24.

Click Continue.

Step 4: Summary

Review the selected backup and BVS settings, then click Finish to run the job.

On-demand verification of an existing backup uses s9s backup --verify with --backup-id and --test-server:

  • Verify backup ID 1 on cluster ID 1 against a standalone host:

    s9s backup \
    --verify \
    --cluster-id=1 \
    --backup-id=1 \
    --test-server=192.168.0.55 \
    --log

    ClusterControl installs the database software on the test server using the same settings as the source cluster, restores the backup there, and the job succeeds only if the restore completes successfully.

To verify a backup automatically right after it's created, add --test-server to s9s backup --create; see Schedule Backup → MySQL & MariaDB for examples. There is no dedicated CLI flag for Install software, Disable firewall, Disable SELinux/AppArmor, or Shutdown the server after the backup is restored; the CLI verify job uses ClusterControl's defaults for these (install software and disable firewall/SELinux on, shutdown off).

Backup Verification does not have a dedicated convenience operation. A verify job is created through the generic createJobInstance call on POST /v2/jobs, with job.job_spec.command set to verify. All calls require a TLS connection and, in most cases, an authenticated session or inline credentials. See ClusterControl RPC API → Authenticating with the API for details.

  • Verify backup ID 1 on cluster ID 1 against a standalone host:

    curl -k -XPOST \
-d '{
    "operation": "createJobInstance",
    "cluster_id": 1,
    "job": {
        "class_name": "CmonJobInstance",
        "job_spec": {
            "command": "verify",
            "job_data": {
                "backup_id": 1,
                "test_server": "192.168.0.55",
                "install_software": true,
                "disable_firewall": true,
                "disable_selinux": true,
                "terminate_db_server": false
            }
        }
    }
}' \
-b cookies.jar \
https://localhost:9501/v2/jobs

Note

The public RPC reference does not document the job_data fields accepted by the verify command. backup_id and test_server are inferred from their identically-named s9s backup --verify CLI flags. install_software, disable_firewall, disable_selinux, and terminate_db_server are carried over from the confirmed verify_backup object used by scheduleBackup (see Schedule Backup → Backup verification), since they configure the same BVS settings screen; the ad-hoc verify job may or may not nest them the same way. Verify against your controller version, or run s9s metatype --list-properties --type=CmonJobInstance --long for the authoritative property list, before relying on this in production.

Database-Specific Settings

MySQL & MariaDB

Covers MySQL & Percona & MariaDB standalone and primary-replica replication clusters, as well as Galera Cluster topologies (Percona XtraDB Cluster, MariaDB Galera Cluster). Settings are identical across topologies.

  • Incremental backups: Only shown when the selected full backup (xtrabackupfull or mariabackupfull) has incrementals chained to it. Select which incremental to verify up to.
  • Install software: Installs MySQL, Percona Server, or MariaDB on the BVS, matching the source cluster's vendor and version.

All other Settings fields are common to every method; see Common Wizard Steps above.

  • Verify a full xtrabackup:

    s9s backup \
    --verify \
    --cluster-id=99 \
    --backup-id=12 \
    --test-server=192.168.73.88 \
    --log

    Works the same way for mysqldump, mariabackupfull, and their incrementals.

All calls go to POST https://<CONTROLLER_HOST>:9501/v2/jobs.

  • Verify a full xtrabackup:

    curl -k -XPOST \
-d '{
    "operation": "createJobInstance",
    "cluster_id": 99,
    "job": {
        "class_name": "CmonJobInstance",
        "job_spec": {
            "command": "verify",
            "job_data": {
                "backup_id": 12,
                "test_server": "192.168.73.88",
                "install_software": true,
                "disable_firewall": true,
                "disable_selinux": true,
                "terminate_db_server": false
            }
        }
    }
}' \
-b cookies.jar \
https://localhost:9501/v2/jobs

PostgreSQL & TimescaleDB

  • Incremental backups: Only shown when the selected full backup (pgbackrestfull or pg_basebackupincr) has incrementals chained to it. Select which incremental to verify up to.
  • Install software: Installs PostgreSQL (or TimescaleDB, or EDB Postgres) on the BVS, matching the source cluster's vendor and version.

All other Settings fields are common to every method; see Common Wizard Steps above.

  • Verify a pgBackRest full backup:

    s9s backup \
    --verify \
    --cluster-id=125 \
    --backup-id=20 \
    --test-server=192.168.73.90 \
    --log

    Works the same way for pgdumpall, pgdump, pg_basebackup, pg_basebackupincr, and pgbackrestdiff/pgbackrestincr.

All calls go to POST https://<CONTROLLER_HOST>:9501/v2/jobs.

  • Verify a pgBackRest full backup:

    curl -k -XPOST \
-d '{
    "operation": "createJobInstance",
    "cluster_id": 125,
    "job": {
        "class_name": "CmonJobInstance",
        "job_spec": {
            "command": "verify",
            "job_data": {
                "backup_id": 20,
                "test_server": "192.168.73.90",
                "install_software": true,
                "disable_firewall": true,
                "disable_selinux": true,
                "terminate_db_server": false
            }
        }
    }
}' \
-b cookies.jar \
https://localhost:9501/v2/jobs

Redis Sentinel

Backup verification is only available for Redis Sentinel clusters. Redis Cluster and Valkey (Sentinel or Cluster) are not supported.

  • Install software: Installs Redis on the BVS, matching the source cluster's version.

All other Settings fields are common to every method; see Common Wizard Steps above.

  • Verify a backup:

    s9s backup \
    --verify \
    --cluster-id=84 \
    --backup-id=6 \
    --test-server=192.168.73.92 \
    --log

All calls go to POST https://<CONTROLLER_HOST>:9501/v2/jobs.

  • Verify a backup:

    curl -k -XPOST \
-d '{
    "operation": "createJobInstance",
    "cluster_id": 84,
    "job": {
        "class_name": "CmonJobInstance",
        "job_spec": {
            "command": "verify",
            "job_data": {
                "backup_id": 6,
                "test_server": "192.168.73.92",
                "install_software": true,
                "disable_firewall": true,
                "disable_selinux": true,
                "terminate_db_server": false
            }
        }
    }
}' \
-b cookies.jar \
https://localhost:9501/v2/jobs

Elasticsearch

Elasticsearch verifies snapshots differently from every other backup method above. Instead of restoring the backup to a standalone host, ClusterControl sends an HTTP request to the Elasticsearch snapshot API's _snapshot/_verify endpoint. This confirms that:

  • The snapshot repository (shared filesystem, S3 bucket, etc.) is reachable.
  • The repository is readable and writable.
  • Every node in the cluster can access it, if it's a shared location.
  • The repository is usable for snapshot and restore operations.

This does not verify that a specific snapshot's data is intact or restorable, only that the repository backing it is healthy. Elasticsearch clusters are often too large for a full restore-and-verify cycle to be practical, which is why this method is repository-level rather than snapshot-level.

There is no BVS, no Settings step, and no verify_backup/verify_backup_delay scheduling option for Elasticsearch: the check either passes or fails immediately.

  1. Go to the Clusters dashboard and click your target cluster.
  2. Go to the Backups tab, keeping the view set to All Backups (default).
  3. Click the action menu (...) on the backup you want to verify and select Restore.
  4. Complete Step 1: Configuration as described in Restore Backup.
  5. Click Continue.
  6. Step 2: How to restore: Choose Verify snapshot.
  7. Click Continue to proceed to Step 3: Summary.
  8. Click Finish to run the job.

  • Verify an Elasticsearch snapshot:

    s9s backup \
    --verify \
    --cluster-id=8 \
    --backup-id=4 \
    --log

    --test-server is not used; there is no standalone host involved.

All calls go to POST https://<CONTROLLER_HOST>:9501/v2/jobs.

  • Verify an Elasticsearch snapshot:

    curl -k -XPOST \
-d '{
    "operation": "createJobInstance",
    "cluster_id": 8,
    "job": {
        "class_name": "CmonJobInstance",
        "job_spec": {
            "command": "verify",
            "job_data": {
                "backup_id": 4
            }
        }
    }
}' \
-b cookies.jar \
https://localhost:9501/v2/jobs
Copyright © 2014-2026 Severalnines AB. All rights reserved.

Severalnines and ClusterControl are registered trademarks in the US, UK, and EU. The 3rd-party trademarks on this site are property of their respective owners and are used for referential purposes only.

Made with Material for MkDocs