Alerts and Notifications
ClusterControl provides multiple ways to view and send alerts when an alarm raised. The alarms can be viewed within ClusterControl GUI and CLI, and the corresponding alerts can be sent via email, messaging platform (Slack, Telegram), incident management systems (PagerDuty, OpsGenie, VictorOps, ilert) or web hooks.
Email notification is built-in into ClusterControl logic and requires a mail server configuration and recipients configuration. For other alerting endpoints like messaging platform, incident management and web hooks, ClusterControl relies on another module call cmon-events, provided by package clustercontrol-notifications. This module allows ClusterControl alarms to be forwarded to third-party services and easily configurable via Notification Services.
See also
To learn more about ClusterControl notification module, see ClusterControl Notifications.
Alarms
The alarms page provides administrators with an aggregated view of alarms raised for all clusters monitored by ClusterControl. Each alarm entry has a header, details on the issue, severity level, category, corresponding cluster name, corresponding host, and timestamp. Alarms indicate problems that could affect or degrade the cluster as a whole. Each alarms is categorized as:
- Cluster
- Cluster recovery
- Database health
- Database performance
- Host
- Node
- Network
View alarms
-
Log in to your ClusterControl GUI → Clusters → choose a database cluster → Alarms. This will open a page where you can view alarms for the particular cluster.
-
Alternatively, you can view all alarms for all clusters by going to ClusterControl GUI → Activity Center → Alarms.
-
Click on Actions → Details, to get a detailed explanation of the problem, together with the recommended action (if available) to resolve the problem.
-
An alarm can be acknowledged by clicking on Actions → Mute. When muted, no notification will be sent out for this type of alarm until it is unmuted. An alarm cannot be deleted or dismissed.
-
You can also see the most recent alarms summarized in the Home page under the Recent alarms section (bottom right).
Tip
Rollover on the Cluster entry will pop up a cluster information card summarizing the cluster status.
Email notifications
When an alarm is raised, ClusterControl can send an alert to any mail recipients. To receive an email from ClusterControl, one has to configure a remote SMTP mail server (or use local MTA) and enable email notifications for the particular cluster, as shown in the following sections.
Configure SMTP mail server
After installation, by default, ClusterControl does not send out email until a mail server is configured, or a local mail transfer agent (Postfix, Sendmail or Exim) is installed on activated on the ClusterControl server.
-
To configure an SMTP mail server, go to ClusterControl GUI → Settings → Configure now. It will open a wizard called "Configure mail server". Choose SMTP Server.
-
In the "Configure mail server: SMTP" window, configure the following:
- Server Address: The SMTP mail server address that you are going to use to send the email.
- Port: The SMTP port for the mail server. Usually, this value is 25 or 587, depending on your SMTP mail server configuration.
- User name: The SMTP user name. Leave empty if no authentication required.
- Password: The SMTP password for User name. Leave empty if no authentication required.
- Reply to/from: Specify the sender of the email. This will appear in the 'From' field of the mail header.
- TLS/SSL required: Check this box if ou want to use TLS/SSL for extra security. The mail server must support TLS/SSL.
-
Check the Send test email to test out the SMTP configuration. If you receive an email from ClusterControl, click Save and proceed to configure the mail recipients. See Configure mail recipients.
Configure local MTA
To use local MTA to send out emails, make sure the local MTA is installed and verified using the test email button available at ClusterControl GUI → Settings → Configure now → Sendmail → Send test email.
Sendmail
On ClusterControl server, install the following packages:
Start the Sendmail service:
Verify if it works:
$ echo "test message" | mail -s "test subject" [email protected]
[email protected] with your email address.
Postfix
Many Linux distributions come with Sendmail as the default MTA. To replace Sendmail and use other MTA, e.g Postfix, you just need to uninstall Sendmail, install Postfix and start the service. Following example shows commands that need to be executed on the ClusterControl node as the root user for RockyLinux 9:
systemctl stop sendmail
yum remove sendmail -y
yum install postfix mailx cronie -y
systemctl enable postfix
systemctl start postfix
Configure mail recipients
After a mail server is configured, or a MTA is installed and activated, one has to enable email notifications for the clusters.
-
Go to ClusterControl GUI → Clusters → choose a database cluster → Settings → Email notifications. It will open a wizard to help you configure the email recipients. This page configures email notifications for alarms generated for the corresponding database cluster.
-
Each recipient (chosen from a user group) must be enabled before you can configure the delivery settings. Choose one group under User group (only groups that has at least read permission to the cluster), select a user under the Users in the selected group dropdown and click Enable.
-
After enabled, configure the Digest delivery times which when to send a digested (summary) email every day to the selected recipient. The timezone will follow the user’s timezone and can be configured under the User profile setting.
-
Configure Digest delivery times which is the maximum number of non-digest email notifications should be sent per day to the selected recipient. The default is 100. This is to protect spamming the recipients for events that are set to "Deliver". See Events.
-
Next, configure the frequency of the emails to be sent for each event's category. See Events.
-
You can add an external email address (a user that does not exist in ClusterControl) by clicking on Add an external email.
Events
On every event severity, you may choose the frequency of raised alarms/events to be delivered to the recipient:
| Action | Description |
|---|---|
| Ignore | Ignore if an alarm is raised. |
| Deliver | Send notification immediately via email once an alarm is raised. |
| Digest | Send a summary of alarms raised every day at Digest delivery time. |
The following table shows the event category that you can forward to a specific recipient:
| Event | Description |
|---|---|
| Network related logs | Network-related messages, e.g. host unreachable, SSH issues. |
| Cmon database logs | Internal CMON database-related messages. |
| Mail logs | Mail system-related messages. |
| Cluster logs | Cluster-related messages, e.g. cluster failed. |
| Cluster configuration logs | Cluster configuration messages, e.g. software configuration messages. |
| Recovery logs | Recovery messages like cluster or node recovery failures. |
| Node related logs | Message related to nodes, e.g. node disconnected, missing GRANT, failed to start HAProxy, failed to start NDB cluster nodes. |
| Host logs | Host-related messages, e.g. CPU/disk/RAM/swap alarms. |
| Health logs | Database health-related messages, e.g. memory usage of MySQL servers, connections. |
| Database performance | Alarms for long-running transactions and deadlocks. |
| Installation logs | Software installation-related messages. |
| Backup logs | Messages about backups. |
| Other logsp | Other uncategorized messages. |
Forward alarms to third-party services
Alarms can be forwarded to third-party services like messaging platforms (Slack, Telegram), incident management systems (PagerDuty, ServiceNow, OpsGenie, VictorOps, ilert) and web hooks. See Integration → Notification Services.