Table of Contents
Provides an interface to manage cloud credentials, user settings, and licenses inside ClusterControl.
Profile
Manages current user profile settings. Click on the Settings button to configure the email address, first name, last name, and time zone for the current user. All timestamp values displayed by ClusterControl will be converted to the current user’s time zone accordingly. Click on the Change Password button to change the user password.
Cloud Storage Credentials
Manages resources and credentials for supported cloud providers. Note that this new feature requires two modules called clustercontrol-cloud
and clustercontrol-clud
. The former is a helper daemon that extends CMON’s capability of cloud communication, while the latter is a file manager client to upload and download files on cloud instances. Both packages are dependencies of the clustercontrol
UI package, which will be installed automatically if does not exist.
The credentials that have been set up here can be used to:
- Manage cloud resources (instances, virtual network, subnet) – ClusterControl GUI v1
- Deploy databases in the cloud – ClusterControl GUI v1
- Upload backup to cloud storage – ClusterControl GUI v1 and v2.
To create a cloud profile, click on Add cloud storage credentials and follow the wizard accordingly. Supported cloud providers are:
- Amazon Web Service
- Google Cloud Platform
- Microsoft Azure
- S3 Compatible Storage Provider
Amazon Web Services
The stored AWS credential will be used by ClusterControl to list out Amazon EC2 instances, spin new instances when deploying a cluster, and upload/download backups to AWS S3.
To create an access key for your AWS account root user:
- Use your AWS account email address and password to sign in to the AWS Management Console as the AWS account root user.
- On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials.
- If you see a warning about accessing the security credentials for your AWS account, choose to Continue to Security Credentials.
- Expand the Access keys (access key ID and secret access key) section.
- Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. After you close the dialog box, you can’t retrieve this secret access key again.
Field | Description |
---|---|
Name | Credential name. |
AWS Key ID | Your AWS Access Key ID as described on this page. You can get this from the AWS IAM Management console. |
AWS Key Secret | Your AWS Secret Access Key as described on this page. You can get this from the AWS IAM Management console. |
Default Region | Choose the default AWS region for this credential. |
Comment (Optional) | Description of the credential. |
Google Cloud
To create a service account:
- Open the Service Accounts page in the Cloud Platform Console.
- Select your project and click Continue.
- In the left navigation, click Service accounts.
- Look for the service account for which you wish to create a key, click on the vertical ellipses button in that row, and click Create key.
- Select
JSON
as the Key type and click Create
Field | Description |
---|---|
Name | Credential name. |
Read from JSON | The service account definition in JSON format. |
Comment (Optional) | Description of the credential. |
Microsoft Azure
In order to provide access to Azure services, you need to register an application and grant it access to your Azure resources.
- Create an application:
1. log in to Microsoft Azure portal → Azure Active Directory → App registrations → New application registration.
2. Provide a name and URL for the application. Select “Web app / API” for the type of application you want to create.
3. After specifying the values, click Create. - Get application ID and authentication key:
1. From App registrations in Azure Active Directory, select your application.
2. Copy the Application ID. You should pass that value asapplication_id
.
3. To generate an authentication key, select Manage → Certificates & secrets → New client secret.
4. Provide a description of the key and a duration for the key. When done, select Save.
5. After saving the key, the value of the key is displayed. Copy this value because you are not able to retrieve the key later. Pass this value asclient_secret
. - Get tenant ID:
1. Go to Microsoft Azure portal → Azure Active Directory → Properties for your Azure AD tenant.
2. Copy the Directory ID. Pass this value astenant_id.
- Get subscription ID:
1. Go to Microsoft Azure portal → Subscriptions.
2. Select your subscription from the list.
3. Copy the Subscription ID. Pass this value assubscription_id
. - Create a resource group:
1. Go to Microsoft Azure portal → Resource groups → Add.
2. Fill in the values and click Create.
3. Copy the Resource group name and use it asresource_group
in the credentials.
4. Wait until the Resource group is created and click Go to Resource group → Access control (IAM) → Add → Add Role Assignment.
5. Select Contributor as a Role then put your application’s name into the search input and select it from the list.
6. Click Save. - Create a storage account (for Upload Backup to Cloud feature):
1. Go to Microsoft Azure portal → Storage accounts → Add.
2. Fill in the Name and select Blob storage as Account kind.
3. Copy the Name value and use it asstorage_account
in credentials.
4. Select Enabled for Secure transfer required, select Subscription and Resource group (use the same Resource group as in the previous steps).
5. Select the storage Location and then click Create.
Finally, create a new text file on your workstation and copy all the required information retrieved from the previous steps in a JSON format. For example:
{
"application_id":"7f649053-xxxx-xxxx-xxxx-2179c1fa83b8",
"client_secret":"jbzW9tj4AyXHDkfO/KoTL9OP5EexpD6jeHROo2S4xxxx",
"tenant_id":"ce6b8358-xxxx-xxxx-xxxx-49b8c7a5cbc2",
"subscription_id":"6fafe95c-xxxx-xxxx-xxxx-1c33daa1c2c3",
"resource_group":"cc",
"storage_account":"mybackupazure"
}
Then, when configuring the Azure credentials, load the above text file under Read from JSON field. Take note that storage_account
value is optional.
The uploaded backup will be available under BLOB CONTAINERS storage. You can verify its existence in the cloud by going to Microsoft Azure portal → Storage Accounts → {your storage account} → Storage Explorer → BLOB CONTAINERS.
Field | Description |
---|---|
Name | Credential name. |
Read from JSON | The service account definition in JSON format. |
Comment (Optional) | Description of the credential. |
S3-Compatible Storage Provider
ClusterControl 1.9.0 introduces support for any cloud storage provider that supports AWS’s S3-compatible object storage API.
Field | Description |
---|---|
Name | Credential name. |
Endpoint | The S3-compatible endpoint in {host}:{port} format. For OpenStack Swift, see this. |
Region | The region name. This field is optional. |
Access Key | The access key given by the cloud provider identity access management console. |
Secret Key | The secret for the defined Access Key. |
Use SSL | Whether to use plain HTTP or HTTPS. |
Comment | Description of the credential. This field is optional. |
Notification Services
Configures third-party notifications on events triggered by ClusterControl, allowing users to integrate ClusterControl into your company’s communication channels, incident response systems, or workflow management systems. Users can configure for the notifications to be sent out only to specific clusters, or to specific alert severity like error or warning alerts. See Warning Events and Critical Events.
Introducing the ClusterControl Alerting Integrations.
Supported services are:
Incident management services | Chat services | Others |
---|---|---|
PagerDuty | Slack | Webhook |
VictorOps | Telegram | |
OpsGenie | ||
ServiceNow |
Note that events are only created when either of the following occurred:
- Alarm created
- Alarm ended
Thus, if you already have alerts raised and then you create, for example, a Slack (or any other) integration, you will never see events for the alerts already raised/created before you created the Slack integration. You will only see the new alerts created after the integration is configured and activated.
Field | Description |
---|---|
Add new integration |
|
Link an external notification service |
|
Service configuration |
|
Notification setting |
|
Edit |
|
Delete |
|
Event | Description |
---|---|
All Events | All ClusterControl events including warning and critical events. |
All Warning Events | All ClusterControl warning events, e.g. cluster degradation, network glitch. See Warning Events. |
All Critical Events | All ClusterControl critical events, e.g. cluster failed, host failed. See Critical Events. |
Network | Network-related events, e.g. host unreachable, SSH issues. |
CMON Database | Internal CMON database-related events, e.g. unable to connect to CMON database, datadir mounted as read-only. |
Mail system-related events, e.g. unable to send mail, mail server unreachable. | |
Cluster | Cluster-related events, e.g. cluster failure, cluster degradation, time drifting. |
Cluster Configuration | Cluster configuration events, e.g. SST account mismatch. |
Cluster Recovery | Recovery events, e.g. cluster or node recovery failures. |
Node | Node-related events, e.g. node disconnected, missing GRANT, failed to start HAProxy, failed to start NDB cluster nodes. |
Host | Host-related messages, e.g. CPU/disk/RAM/swap exceeds thresholds, memory full. |
Database Health | Database health-related events, e.g. memory usage of MySQL servers, connections, missing primary key. |
Database Performance | Alarms for long-running transactions, replication lag, and deadlocks. |
Software Installation | Software installation-related events, e.g. license expiration. |
Backup | Backup-related events, e.g. backup failed. |
Warning Events
ClusterControl uses the same alarms’ code name for other database clusters which produces a similar result. For example, if a PostgreSQL slave is lagging behind, the alarm’s internal code name is also called “MySqlReplicationLag”. However, the actual alarm’s response will have proper PostgreSQL relevant texts. This is handled internally.
Area | Alarms | Severity | Description |
---|---|---|---|
Node | MySqlReplicationLag | Warning | MySQL replication slave lag, default 10 seconds. |
MySqlReplicationBroken | Warning | The SQL thread has stopped. For PostgreSQL, it means the slave gets disconnected from the master. | |
CertificateExpiration | Warning | SSL certificate expiration time (<=31 days, >7 days). | |
MySqlAdvisor | Warning | Raised by wsrep_sst_method.js and wsrep_node_name.js advisors. |
|
MySqlTableAnalyzer | Warning | Raised by schema_check_nopk.js advisor. |
|
StorageMyIsam | Warning | Raised by schema_check_myisam.js advisor. |
|
MySqlIndexAnalyzer | Warning | Raised by schema_check_dupl_index.js advisor. |
|
MySqlReplicationLooseServer | Warning | A slave is found but its master can’t be determined, or it is not part of the cluster. | |
Host | HostSwapV2 | Warning | If a configurable number of pages has been swapped in/out during a configurable period of time. Default 20 pages in 10 minutes. |
HostSwapping | Warning | >5% swap space has been used. | |
HostCpuUsage | Warning | >80%, <90% CPU used. | |
HostRamUsage | Warning | >80%, <90% RAM used. | |
HostDiskUsage | Warning | >80%, <90% disk space used on a monitored_mountpoint. | |
ProcessCpuUsage | Warning | >95 % CPU used on average by a process for 15 minutes. | |
Backup | BackupFailed | Warning | The backup job fails. |
Recovery | GaleraWsrepMissing | Warning | wsrep_cluster_address or wsrep_provider is missing. |
GaleraSstAuth | Warning | SST settings (user/pass are wrong). | |
Network | HostFirewall | Warning | The host is not responding to ping after 3 cycles. |
HostSshSlow | Warning | It takes 6-12 seconds to SSH into a host. | |
Cluster | ClusterTimeDrift | Warning | Time drift between ClusterControl and database nodes. |
ClusterLicenseExpire | Warning | The license is about to expire. | |
ClusterInconsitentView | Warning | The load balancer or ClusterControl sees a different set of working nodes (the master is down from the ClusterControl point-of-view, while the load balancer or the slave reports the master working.) |
Critical Events
ClusterControl uses the same alarms’ code name for other database clusters which produces a similar result. For example, if a PostgreSQL server goes down, the alarm’s internal code name is also called “MySqlDisconnected”. However, the actual alarm’s response will have proper PostgreSQL relevant texts. This is handled internally.
Area | Alarms | Severity | Description |
---|---|---|---|
Node | MySqlDisconnected | Critical | The database server cannot be reached. |
MySqlGrantMissing | Critical | Node does not have the correct privileges set for the cmon user. | |
MySqlLongRunningQuery | Critical | If queries are running for too long time. Only used if configured, by default it is not. | |
ProcFailedRestart | Critical | A process (HAProxy, ProxySQL, Garbd, MaxScale) could not be restarted after a failure. | |
CertificateExpiration | Critical | (<= 7 days), SSL Certificates expiration time. | |
MySqlReplicationMultiMaster | Critical | Multiple writable masters detected. | |
Host | HostSwapV2 | Critical | If a configurable number of pages has been swapped in/out during a configurable period of time. Default 20 pages in 10 minutes. |
HostSwapping | Critical | >20% swap space has been used. | |
HostCpuUsage | Critical | >90% CPU used. | |
HostRamUsage | Critical | >90% RAM used. | |
HostDiskUsage | Critical | >90% disk space used on a monitored_mountpoint . |
|
ProcessCpuUsage | Critical | >99 % CPU used on average by a process for 15 minutes. | |
Backup | BackupVerificationFailed | Critical | Backup verification fails. |
Recovery | GaleraWsrepMissing | Critical | wsrep_cluster_address or wsrep_provider is missing, and still missing after 20 sample cycles which are ~ 100 seconds in this case) |
GaleraClusterSplit | Critical | There is a split-brain. | |
ClusterRecoveryFail | Critical | Recovery has failed. | |
GaleraConfigProblem1 | Critical | A configuration issue preventing the node from starting. | |
GaleraNodeRecoveryFail | Critical | Automatic recovery has failed 3 consecutive times. | |
ReplicationFailoverBlacklistError | Critical | In the case of automatic failover, the only possible candidate is blacklisted, then this alarm is raised with critical severity. | |
Network | HostUnreachable | Critical | The host is not responding to ping after 3 cycles. |
HostSshFailed | Critical | Please check SSH access to the host. The host may also be down. | |
HostSshAuth | Critical | Please check whether the configured SSH key is authenticated on the host. | |
HostSudoError | Critical | sudo command error on the host. |
|
HostSshSlow | Critical | It takes >12 seconds to SSH into a host. | |
Cluster | ClusterFailure | Critical | The cluster failed. |
ClusterLicenseExpire | Critical | The license is expired. |
Webhook Integrations
For webhook integrations, ClusterControl will raise an alarm by sending the JSON data using the HTTP POST method to the configured endpoint. The webhook endpoint should receive the following example event if a new alarm is created:
{
"id": 470,
"status": "CREATED",
"component": "Node",
"hostname": "192.168.20.62",
"title": "Server disconnected",
"message": "PostgreSQL server on 192.168.20.62:5432 disconnected: Connect failure: Watchdog: Failed connection to [email protected]:5432. timeout expired\n",
"recommendation": "Check node status on UI and error log of failed server.",
"severity": "CRITICAL"
}
The above event was created and sent out to the webhook endpoint after ClusterControl detected that one of the database servers (192.168.20.62) was down. After the server came back online, ClusterControl would then send another event with the same alarm ID (470) with a different status ENDED
, indicating the above alarm has ended and the node is back operational:
{
"id": 470,
"status": "ENDED",
"component": "Node",
"hostname": "192.168.20.62",
"title": "Server disconnected",
"message": "PostgreSQL server on 192.168.20.62:5432 disconnected: Connect failure: Watchdog: Failed connection to [email protected]:5432. timeout expired\n",
"recommendation": "Check node status on UI and error log of failed server.",
"severity": "CRITICAL"
}
Commonly, the “ended” event has an almost identical response text with the “created” event except for the “status” value.
You may use https://webhook.site/ to test out the webhook integration with ClusterControl.
Certificate Management
Key Management allows you to manage a set of SSL certificates and keys that can be provisioned on your clusters. This feature allows you to create Certificate Authority (CA) and/or self-signed certificates and keys. Then, it can be easily enabled and disabled for MySQL and PostgreSQL client-server connections using the SSL encryption feature.
By default, the generated keys and certificates will be created under the default repository at /var/lib/cmon/ca
.
Field | Description |
---|---|
Create Folder |
|
Refresh |
|
Generate Self-signed Certificate
Generate a self-signed Certificate Authority and key. You can use this Certificate Authority (CA) to sign your client and server certificates.
Field | Description |
---|---|
Path |
|
Certificate Authority and Key Name |
|
Description |
|
Country |
|
State |
|
Locality |
|
Organization |
|
Organization unit |
|
Common name |
|
|
|
Key length (bits) |
|
Expiration Date (days) |
|
Generate |
|
Reset |
|
Generate Client/Server Certificate
Sign with an existing CA or generate a self-signed certificate. ClusterControl generates certificates and keys depending on the type, server or client. The generated server’s key and certificate can then be used by the Enable SSL Encryption feature.
Field | Description |
---|---|
Certificate Authority |
|
Type |
|
Certificate and Key Name |
|
Description |
|
Country |
|
State |
|
Locality |
|
Organization |
|
Organization unit |
|
Common name |
|
|
|
Key length (bits) |
|
Expiration Date (days) |
|
Generate |
|
Reset |
|
Import
Import keys and certificates into ClusterControl’s certificate repository. The imported keys and certificates can then be used to enable SSL encryption for server-client connection, replication, or backup at a later stage. Before you perform the important action, bear in mind to:
- Upload your certificate and key to a directory in the ClusterControl Controller host
- Uncheck the Self-signed Certificate checkbox if the certificate is not self-signed
- You need to also provide a CA certificate if the certificate is not self-signed
- Duplicate certificates will not be created
Field | Description |
---|---|
Destination Path |
|
Save As |
|
Certificate File |
|
Private Key File |
|
Self-signed Certificate |
|
Import |
|
License
ClusterControl introduces a new license format on v1.7.1 (the new license key format contains only a long-encrypted string). If you are having the older format, contact the account manager, or email our sales department at [email protected] to get a new license.
For users with a valid subscription (Advanced and Enterprise), enter your license key by clicking on the Enter a license key button to unlock additional features based on the subscription. The license string contains information about the license type, company/affiliation, email, expiration date, and the total number of licensed nodes.
The following example is license information that one would get from us:
Email: [email protected]
Company: Severalnines AB
Expiration Date: 2019-04-23
Licensed Nodes: 1
License Key: ewogICAgImNvbXBhbnkiOiAiU2V2ZXJhbG5pbmPkIEFCIiwKICAgICJlbWFpbF9hZGRyZXNzIjog
InRlc3RAc2VRZXJhbG5pbmVzLmNvbSIsCiAgICAiZXhwaXJhdGlvbl9kYXRlIjogIjIwMTktMDQt
MjNUMDA6MDA6MDAuMDAxWiIsCiAgICAibGljZW5zZWRfbm9kZXMiOiAtMSwKICAgICJ0eXBlIjog
IkVudGVycHJpc2UiCn0Ke3M5cy1zaWduYXR1cmUtc2VwYXJhdG9yfQp0RUew5OZKV8SqmmwiQEzT
P+qTNnmCphirIVm7MriA/aCdlJYQcr1NJr4nvTNcSmgu4uFVf3Ufv4olHr4wrBq0/Js9Rm8bJWZo
BO8svHzQhCmIVEWcTYub362krjRyREnOGXaqWwUnvkZ0uUCT+WDaM1P9qn/HawoNd0e8E0+7WiZK
CpwjH+ESqSEppeu/Ewzf3p0C0e8WZbwHtZ9UmX2qJNQq9NDlByrO8FtbVjOOL4zTbc8jV0W2DWzY
1swzOgeyk+7N2eGVRWfdUSzudQSXkT3LA4cdV2HAsU5QLnmKxSCgg+jq+RQJiPwdPXEr3gzjzJzV
+qhmOZ5tTN+WABPy9l3kpztlCbkfzO84/4lM7Z3c4rQ8snMTu6RvD2M+oh/4lhvR8M9RrQQcl8JF
RX2Ak1ZAKxAXkJ97Z5U7nIzuyUGuMTCXdKGEtQkBXzpIcYFvXDeWu0MUks+EULpqG+OFnl+rSZa0
nNTSW3mR/f9B+4e2mK4y2OpJhh4rWPXR1DLpLVLk/2p0o64aEizA+IPe0TP+ox7bFzEfAXirVWfC
/Ol7m1k6arRbl8PSV1DRRcefM+UsABa6jypoiit+JXNPOajdjY1WBgEekCn/jeXBBoPM2k26274u
br0BuHULLkxGSpC8I2/nW6s84E653FO1Kpbvyx+2SKJxwUxLiuEZ2g==
Only paste the license key string (starting after the License Key:
part) into the license text field. Once applied, restart the CMON service to load the new license information:
$ systemctl restart cmon # systemd
$ service cmon restart # SysVinit
When the license expires, ClusterControl defaults back to the Community Edition. For features comparison, please refer to the ClusterControl product page.
If you would like to force the existing enterprise edition to the community edition (commonly to test out and compare different editions during trial), you can truncate the license table on the ClusterControl host manually. On the ClusterControl server, run:
$ mysql -uroot -p cmon -e "TRUNCATE TABLE cmon.license"
$ systemctl restart cmon
Once a trial license is truncated and cmon is restarted, there is no way to re-activate the same trial license anymore.