1. Home
  2. Docs
  3. ClusterControl
  4. User Guide (GUI v2)
  5. Settings

Settings

Provides an interface to manage cloud credentials, user settings, and licenses inside ClusterControl.

Profile

Manages current user profile settings. Click on the Settings button to configure the email address, first name, last name, and time zone for the current user. All timestamp values displayed by ClusterControl will be converted to the current user’s time zone accordingly. Click on the Change Password button to change the user password.

Cloud Storage Credentials

Manages resources and credentials for supported cloud providers. Note that this new feature requires two modules called clustercontrol-cloud and clustercontrol-clud. The former is a helper daemon that extends CMON’s capability of cloud communication, while the latter is a file manager client to upload and download files on cloud instances. Both packages are dependencies of the clustercontrol UI package, which will be installed automatically if does not exist.

The credentials that have been set up here can be used to:

  • Manage cloud resources (instances, virtual network, subnet) – ClusterControl GUI v1
  • Deploy databases in the cloud – ClusterControl GUI v1
  • Upload backup to cloud storage – ClusterControl GUI v1 and v2.

To create a cloud profile, click on Add cloud storage credentials and follow the wizard accordingly. Supported cloud providers are:

  • Amazon Web Service
  • Google Cloud Platform
  • Microsoft Azure
  • S3 Compatible Storage Provider

Amazon Web Services

The stored AWS credential will be used by ClusterControl to list out Amazon EC2 instances, spin new instances when deploying a cluster, and upload/download backups to AWS S3.

To create an access key for your AWS account root user:

  1. Use your AWS account email address and password to sign in to the AWS Management Console as the AWS account root user.
  2. On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials.
  3. If you see a warning about accessing the security credentials for your AWS account, choose to Continue to Security Credentials.
  4. Expand the Access keys (access key ID and secret access key) section.
  5. Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. After you close the dialog box, you can’t retrieve this secret access key again.
Field Description
Name Credential name.
AWS Key ID Your AWS Access Key ID as described on this page. You can get this from the AWS IAM Management console.
AWS Key Secret Your AWS Secret Access Key as described on this page. You can get this from the AWS IAM Management console.
Default Region Choose the default AWS region for this credential.
Comment (Optional) Description of the credential.

Google Cloud

To create a service account:

  1. Open the Service Accounts page in the Cloud Platform Console.
  2. Select your project and click Continue.
  3. In the left navigation, click Service accounts.
  4. Look for the service account for which you wish to create a key, click on the vertical ellipses button in that row, and click Create key.
  5. Select JSON as the Key type and click Create
Field Description
Name Credential name.
Read from JSON The service account definition in JSON format.
Comment (Optional) Description of the credential.

Microsoft Azure

In order to provide access to Azure services, you need to register an application and grant it access to your Azure resources.

  1. Create an application:
    1. log in to Microsoft Azure portal → Azure Active Directory → App registrations → New application registration.
    2. Provide a name and URL for the application. Select “Web app / API” for the type of application you want to create.
    3. After specifying the values, click Create.
  2. Get application ID and authentication key:
    1. From App registrations in Azure Active Directory, select your application.
    2. Copy the Application ID. You should pass that value as application_id.
    3. To generate an authentication key, select Manage → Certificates & secrets → New client secret.
    4. Provide a description of the key and a duration for the key. When done, select Save.
    5. After saving the key, the value of the key is displayed. Copy this value because you are not able to retrieve the key later. Pass this value as client_secret.
  3. Get tenant ID:
    1. Go to Microsoft Azure portal → Azure Active Directory → Properties for your Azure AD tenant.
    2. Copy the Directory ID. Pass this value as tenant_id.
  4. Get subscription ID:
    1. Go to Microsoft Azure portal → Subscriptions.
    2. Select your subscription from the list.
    3. Copy the Subscription ID. Pass this value as subscription_id.
  5. Create a resource group:
    1. Go to Microsoft Azure portal → Resource groups → Add.
    2. Fill in the values and click Create.
    3. Copy the Resource group name and use it as resource_group in the credentials.
    4. Wait until the Resource group is created and click Go to Resource group → Access control (IAM) → Add → Add Role Assignment.
    5. Select Contributor as a Role then put your application’s name into the search input and select it from the list.
    6. Click Save.
  6. Create a storage account (for Upload Backup to Cloud feature):
    1. Go to Microsoft Azure portal → Storage accounts → Add.
    2. Fill in the Name and select Blob storage as Account kind.
    3. Copy the Name value and use it as storage_account in credentials.
    4. Select Enabled for Secure transfer required, select Subscription and Resource group (use the same Resource group as in the previous steps).
    5. Select the storage Location and then click Create.

Finally, create a new text file on your workstation and copy all the required information retrieved from the previous steps in a JSON format. For example:

{
  "application_id":"7f649053-xxxx-xxxx-xxxx-2179c1fa83b8",
  "client_secret":"jbzW9tj4AyXHDkfO/KoTL9OP5EexpD6jeHROo2S4xxxx",
  "tenant_id":"ce6b8358-xxxx-xxxx-xxxx-49b8c7a5cbc2",
  "subscription_id":"6fafe95c-xxxx-xxxx-xxxx-1c33daa1c2c3",
  "resource_group":"cc",
  "storage_account":"mybackupazure"
}

Then, when configuring the Azure credentials, load the above text file under Read from JSON field. Take note that storage_account value is optional.

The uploaded backup will be available under BLOB CONTAINERS storage. You can verify its existence in the cloud by going to Microsoft Azure portal → Storage Accounts → {your storage account} → Storage Explorer → BLOB CONTAINERS.

Field Description
Name Credential name.
Read from JSON The service account definition in JSON format.
Comment (Optional) Description of the credential.

S3-Compatible Storage Provider

ClusterControl 1.9.0 introduces support for any cloud storage provider that supports AWS’s S3-compatible object storage API.

Field Description
Name Credential name.
Endpoint The S3-compatible endpoint in {host}:{port} format. For OpenStack Swift, see this.
Region The region name. This field is optional.
Access Key The access key given by the cloud provider identity access management console.
Secret Key The secret for the defined Access Key.
Use SSL Whether to use plain HTTP or HTTPS.
Comment Description of the credential. This field is optional.

Notification Services

Configures third-party notifications on events triggered by ClusterControl, allowing users to integrate ClusterControl into your company’s communication channels, incident response systems, or workflow management systems. Users can configure for the notifications to be sent out only to specific clusters, or to specific alert severity like error or warning alerts. See Warning Events and Critical Events.

See also

Supported services are:

Incident management services Chat services Others
PagerDuty Slack Webhook
VictorOps Telegram
OpsGenie
ServiceNow

Note that events are only created when either of the following occurred:

  • Alarm created
  • Alarm ended

Thus, if you already have alerts raised and then you create, for example, a Slack (or any other) integration, you will never see events for the alerts already raised/created before you created the Slack integration. You will only see the new alerts created after the integration is configured and activated.

Field Description
Add new integration
  • Opens the service integration configuration wizard.
Link an external notification service
  • Choose a service that you want to configure. Different service requires a different set of options.
Service configuration
  • Specify a name for this integration together with the corresponding service key, authentication string, or service token. This information can be retrieved from the provider’s website. Click on the “Test” button to verify if ClusterControl is able to connect with the service provider.
Notification setting
  • Specify the cluster name together with ClusterControl events that you would like to trigger for the incident. You can define multiple values for both fields. Details on the events are described in Warning Events and Critical Events.
Edit
  • Edit the selected integration.
Delete
  • Remove the selected integration.
Event Description
All Events All ClusterControl events including warning and critical events.
All Warning Events All ClusterControl warning events, e.g. cluster degradation, network glitch. See Warning Events.
All Critical Events All ClusterControl critical events, e.g. cluster failed, host failed. See Critical Events.
Network Network-related events, e.g. host unreachable, SSH issues.
CMON Database Internal CMON database-related events, e.g. unable to connect to CMON database, datadir mounted as read-only.
Mail Mail system-related events, e.g. unable to send mail, mail server unreachable.
Cluster Cluster-related events, e.g. cluster failure, cluster degradation, time drifting.
Cluster Configuration Cluster configuration events, e.g. SST account mismatch.
Cluster Recovery Recovery events, e.g. cluster or node recovery failures.
Node Node-related events, e.g. node disconnected, missing GRANT, failed to start HAProxy, failed to start NDB cluster nodes.
Host Host-related messages, e.g. CPU/disk/RAM/swap exceeds thresholds, memory full.
Database Health Database health-related events, e.g. memory usage of MySQL servers, connections, missing primary key.
Database Performance Alarms for long-running transactions, replication lag, and deadlocks.
Software Installation Software installation-related events, e.g. license expiration.
Backup Backup-related events, e.g. backup failed.

Warning Events

Note

ClusterControl uses the same alarms’ code name for other database clusters which produces a similar result. For example, if a PostgreSQL slave is lagging behind, the alarm’s internal code name is also called “MySqlReplicationLag”. However, the actual alarm’s response will have proper PostgreSQL relevant texts. This is handled internally.

Area Alarms Severity Description
Node MySqlReplicationLag Warning MySQL replication slave lag, default 10 seconds.
MySqlReplicationBroken Warning The SQL thread has stopped. For PostgreSQL, it means the slave gets disconnected from the master.
CertificateExpiration Warning SSL certificate expiration time (<=31 days, >7 days).
MySqlAdvisor Warning Raised by wsrep_sst_method.js and wsrep_node_name.js advisors.
MySqlTableAnalyzer Warning Raised by schema_check_nopk.js advisor.
StorageMyIsam Warning Raised by schema_check_myisam.js advisor.
MySqlIndexAnalyzer Warning Raised by schema_check_dupl_index.js advisor.
MySqlReplicationLooseServer Warning A slave is found but its master can’t be determined, or it is not part of the cluster.
Host HostSwapV2 Warning If a configurable number of pages has been swapped in/out during a configurable period of time. Default 20 pages in 10 minutes.
HostSwapping Warning >5% swap space has been used.
HostCpuUsage Warning >80%, <90% CPU used.
HostRamUsage Warning >80%, <90% RAM used.
HostDiskUsage Warning >80%, <90% disk space used on a monitored_mountpoint.
ProcessCpuUsage Warning >95 % CPU used on average by a process for 15 minutes.
Backup BackupFailed Warning The backup job fails.
Recovery GaleraWsrepMissing Warning  wsrep_cluster_address or wsrep_provider is missing.
GaleraSstAuth Warning SST settings (user/pass are wrong).
Network HostFirewall Warning The host is not responding to ping after 3 cycles.
HostSshSlow Warning It takes 6-12 seconds to SSH into a host.
Cluster ClusterTimeDrift Warning Time drift between ClusterControl and database nodes.
ClusterLicenseExpire Warning The license is about to expire.
ClusterInconsitentView Warning The load balancer or ClusterControl sees a different set of working nodes (the master is down from the ClusterControl point-of-view, while the load balancer or the slave reports the master working.)

Critical Events

Note

ClusterControl uses the same alarms’ code name for other database clusters which produces a similar result. For example, if a PostgreSQL server goes down, the alarm’s internal code name is also called “MySqlDisconnected”. However, the actual alarm’s response will have proper PostgreSQL relevant texts. This is handled internally.

Area Alarms Severity Description
Node MySqlDisconnected Critical The database server cannot be reached.
MySqlGrantMissing Critical Node does not have the correct privileges set for the cmon user.
MySqlLongRunningQuery Critical If queries are running for too long time. Only used if configured, by default it is not.
ProcFailedRestart Critical A process (HAProxy, ProxySQL, Garbd, MaxScale) could not be restarted after a failure.
CertificateExpiration Critical (<= 7 days), SSL Certificates expiration time.
MySqlReplicationMultiMaster Critical Multiple writable masters detected.
Host HostSwapV2 Critical If a configurable number of pages has been swapped in/out during a configurable
period of time. Default 20 pages in 10 minutes.
HostSwapping Critical >20% swap space has been used.
HostCpuUsage Critical >90% CPU used.
HostRamUsage Critical >90% RAM used.
HostDiskUsage Critical >90% disk space used on a monitored_mountpoint.
ProcessCpuUsage Critical >99 % CPU used on average by a process for 15 minutes.
Backup BackupVerificationFailed Critical Backup verification fails.
Recovery GaleraWsrepMissing Critical wsrep_cluster_address or wsrep_provider is missing, and still missing after 20 sample cycles which are ~ 100 seconds in this case)
GaleraClusterSplit Critical There is a split-brain.
ClusterRecoveryFail Critical Recovery has failed.
GaleraConfigProblem1 Critical A configuration issue preventing the node from starting.
GaleraNodeRecoveryFail Critical Automatic recovery has failed 3 consecutive times.
ReplicationFailoverBlacklistError Critical In the case of automatic failover, the only possible candidate is blacklisted, then this alarm is raised with critical severity.
Network HostUnreachable Critical The host is not responding to ping after 3 cycles.
HostSshFailed Critical Please check SSH access to the host. The host may also be down.
HostSshAuth Critical Please check whether the configured SSH key is authenticated on the host.
HostSudoError Critical sudo command error on the host.
HostSshSlow Critical It takes >12 seconds to SSH into a host.
Cluster ClusterFailure Critical The cluster failed.
ClusterLicenseExpire Critical The license is expired.

Webhook Integrations

For webhook integrations, ClusterControl will raise an alarm by sending the JSON data using the HTTP POST method to the configured endpoint. The webhook endpoint should receive the following example event if a new alarm is created:

{
  "id": 470,
  "status": "CREATED",
  "component": "Node",
  "hostname": "192.168.20.62",
  "title": "Server disconnected",
  "message": "PostgreSQL server on 192.168.20.62:5432 disconnected: Connect failure: Watchdog: Failed connection to [email protected]:5432. timeout expired\n",
  "recommendation": "Check node status on UI and error log of failed server.",
  "severity": "CRITICAL"
}

The above event was created and sent out to the webhook endpoint after ClusterControl detected that one of the database servers (192.168.20.62) was down. After the server came back online, ClusterControl would then send another event with the same alarm ID (470) with a different status ENDED, indicating the above alarm has ended and the node is back operational:

{
  "id": 470,
  "status": "ENDED",
  "component": "Node",
  "hostname": "192.168.20.62",
  "title": "Server disconnected",
  "message": "PostgreSQL server on 192.168.20.62:5432 disconnected: Connect failure: Watchdog: Failed connection to [email protected]:5432. timeout expired\n",
  "recommendation": "Check node status on UI and error log of failed server.",
  "severity": "CRITICAL"
}

Commonly, the “ended” event has an almost identical response text with the “created” event except for the “status” value.

See also

You may use https://webhook.site/ to test out the webhook integration with ClusterControl.

Certificate Management

Key Management allows you to manage a set of SSL certificates and keys that can be provisioned on your clusters. This feature allows you to create Certificate Authority (CA) and/or self-signed certificates and keys. Then, it can be easily enabled and disabled for MySQL and PostgreSQL client-server connections using the SSL encryption feature.

By default, the generated keys and certificates will be created under the default repository at /var/lib/cmon/ca.

Field Description
Create Folder
  • Create a new directory under the default repository.
Refresh
  • Refresh the certificate and directory list.

Generate Self-signed Certificate

Generate a self-signed Certificate Authority and key. You can use this Certificate Authority (CA) to sign your client and server certificates.

Field Description
Path
  • Certification repository path. To change the path, click on the file browser left-side menu. Default value is /var/lib/cmon/ca.
Certificate Authority and Key Name
  • Enter a name without an extension. For example MyCA, ca-cert
Description
  • Put some description for the certificate authority.
Country
  • Choose a country name from the dropdown menu.
State
  • State or province name.
Locality
  • City name.
Organization
  • Organization name.
Organization unit
  • Unit or department name.
Common name
  • Specify server fully-qualified domain name (FQDN) or your name.
  • Common Name value used for the server and client certificates/keys must each differ from the Common Name value used for the CA certificate. Otherwise, the certificate and the key files will not work for the servers compiled using OpenSSL.
Email
  • Email address.
Key length (bits)
  • The key length in bits. 2048 and higher is recommended. The larger the public and private key length, the harder it is to crack
Expiration Date (days)
  • Certificate expiration in days.
Generate
  • Generate certificate and key.
Reset
  • Reset the form.

Generate Client/Server Certificate

Sign with an existing CA or generate a self-signed certificate. ClusterControl generates certificates and keys depending on the type, server or client. The generated server’s key and certificate can then be used by the Enable SSL Encryption feature.

Field Description
Certificate Authority
  • Select an existing CA (by clicking on any existing CA on the left-hand side menu) or leave it empty to generate a self-signed certificate.
Type
  • Server – Generate a certificate for server usage.
  • Client – Generate a certificate for client usage.
Certificate and Key Name
  • Enter the certificate and key name. The same name will be used by ClusterControl to generate certificates and keys. For example, if you specify the name as Severalnines, ClusterControl will generate severalnines.key and severalnines.crt respectively.
Description
  • Put some description for the certificate and key.
Country
  • Choose a country name from the dropdown menu.
State
  • State or province name.
Locality
  • City name.
Organization
  • Organization name.
Organization unit
  • Unit or department name.
Common name
  • Specify server’s fully-qualified domain name (FQDN) or your name.
  • Common Name value used for the server and client certificates/keys must each differ from the Common Name value used for the CA certificate. Otherwise, the certificate and the key files will not work for the servers compiled using OpenSSL.
Email
  • Email address.
Key length (bits)
  • The key length in bits. 2048 and higher is recommended.
Expiration Date (days)
  • Certificate expiration in days.
Generate
  • Generate certificate and key.
Reset
  • Reset the form.

Import

Import keys and certificates into ClusterControl’s certificate repository. The imported keys and certificates can then be used to enable SSL encryption for server-client connection, replication, or backup at a later stage. Before you perform the important action, bear in mind to:

  1. Upload your certificate and key to a directory in the ClusterControl Controller host
  2. Uncheck the Self-signed Certificate checkbox if the certificate is not self-signed
  3. You need to also provide a CA certificate if the certificate is not self-signed
  4. Duplicate certificates will not be created
Field Description
Destination Path
  • Where you want the certificate to be imported to. Click on the file explorer window on the left to change the path.
Save As
  • Certificate name.
Certificate File
  • Absolute path to the certificate file. For example: /home/user/ssl/file.crt.
Private Key File
  • Absolute path to the key file. For example: /home/user/ssl/file.key.
Self-signed Certificate
  • Uncheck the checkbox if the certificate is not self-signed.
Import
  • Start the import process.

License

Attention

ClusterControl introduces a new license format on v1.7.1 (the new license key format contains only a long-encrypted string). If you are having the older format, contact the account manager, or email our sales department at [email protected] to get a new license.

For users with a valid subscription (Advanced and Enterprise), enter your license key by clicking on the Enter a license key button to unlock additional features based on the subscription. The license string contains information about the license type, company/affiliation, email, expiration date, and the total number of licensed nodes.

The following example is license information that one would get from us:

Email: [email protected]
Company: Severalnines AB
Expiration Date: 2019-04-23
Licensed Nodes: 1
License Key: ewogICAgImNvbXBhbnkiOiAiU2V2ZXJhbG5pbmPkIEFCIiwKICAgICJlbWFpbF9hZGRyZXNzIjog
InRlc3RAc2VRZXJhbG5pbmVzLmNvbSIsCiAgICAiZXhwaXJhdGlvbl9kYXRlIjogIjIwMTktMDQt
MjNUMDA6MDA6MDAuMDAxWiIsCiAgICAibGljZW5zZWRfbm9kZXMiOiAtMSwKICAgICJ0eXBlIjog
IkVudGVycHJpc2UiCn0Ke3M5cy1zaWduYXR1cmUtc2VwYXJhdG9yfQp0RUew5OZKV8SqmmwiQEzT
P+qTNnmCphirIVm7MriA/aCdlJYQcr1NJr4nvTNcSmgu4uFVf3Ufv4olHr4wrBq0/Js9Rm8bJWZo
BO8svHzQhCmIVEWcTYub362krjRyREnOGXaqWwUnvkZ0uUCT+WDaM1P9qn/HawoNd0e8E0+7WiZK
CpwjH+ESqSEppeu/Ewzf3p0C0e8WZbwHtZ9UmX2qJNQq9NDlByrO8FtbVjOOL4zTbc8jV0W2DWzY
1swzOgeyk+7N2eGVRWfdUSzudQSXkT3LA4cdV2HAsU5QLnmKxSCgg+jq+RQJiPwdPXEr3gzjzJzV
+qhmOZ5tTN+WABPy9l3kpztlCbkfzO84/4lM7Z3c4rQ8snMTu6RvD2M+oh/4lhvR8M9RrQQcl8JF
RX2Ak1ZAKxAXkJ97Z5U7nIzuyUGuMTCXdKGEtQkBXzpIcYFvXDeWu0MUks+EULpqG+OFnl+rSZa0
nNTSW3mR/f9B+4e2mK4y2OpJhh4rWPXR1DLpLVLk/2p0o64aEizA+IPe0TP+ox7bFzEfAXirVWfC
/Ol7m1k6arRbl8PSV1DRRcefM+UsABa6jypoiit+JXNPOajdjY1WBgEekCn/jeXBBoPM2k26274u
br0BuHULLkxGSpC8I2/nW6s84E653FO1Kpbvyx+2SKJxwUxLiuEZ2g==

Only paste the license key string (starting after the License Key:  part) into the license text field. Once applied, restart the CMON service to load the new license information:

$ systemctl restart cmon # systemd
$ service cmon restart # SysVinit

When the license expires, ClusterControl defaults back to the Community Edition. For features comparison, please refer to the ClusterControl product page.

If you would like to force the existing enterprise edition to the community edition (commonly to test out and compare different editions during trial), you can truncate the license table on the ClusterControl host manually. On the ClusterControl server, run:

$ mysql -uroot -p cmon -e "TRUNCATE TABLE cmon.license"
$ systemctl restart cmon
Warning

Once a trial license is truncated and cmon is restarted, there is no way to re-activate the same trial license anymore.

Was this article helpful to you? Yes No