ClusterControl Domain Specific Language (CCDSL)

CCDSL is the language used to write ClusterControl advisors, programs, and widgets, small scripts that run inside the ClusterControl cmon controller process and are packaged into tarballs (such as s9s.tar.gz and common.tar.gz) that cmon loads on first start, or after an upgrade.

The syntax is JavaScript-flavored, with extensions for accessing ClusterControl's internal data structures and functions, for example, executing SQL statements, running shell commands across cluster hosts, reading historical monitoring data, editing configuration files, raising alarms, enqueueing controller jobs, and rendering graphs. The results are then processed into advisors, alerts, or other actions.

1. Language basics

1.1 How CCDSL differs from JavaScript

Semicolons are mandatory, as in C/C++.
Numbers are not all double-precision floats: Int, Ulonglong, and Double are distinct types, needed to represent disk sizes and network traffic measured in bytes without losing precision.
Associative arrays are a dedicated Map type.
Arrays are two-dimensional but can be used as one-dimensional (a[10, 11] and a[10] are both valid).
There is a distinct List type.
Namespaced/static-style functions use C++ notation (JSON::parse(text), cluster::hosts()) rather than JavaScript's dot notation.
Variables created on the fly inside a function are local to that function, not global.
The language implements a C-like #include preprocessor directive.

1.2 Types

var a = new Int;
var b = new CmonHost();
passed = a === 0 && b.typeName() == "CmonHost";

Built-in types: Int, Bool, Double, Ulonglong, String, Error, Map, List, Array, CmonHost, CmonMySqlHost, CmonGaleraHost, CmonGroupReplHost, CmonPostgreSqlHost, CmonNdbHost, CmonMongoHost, CmonMaxScaleHost, CmonProxySqlHost, CmonAdvice, CmonClusterConfig, CmonFile, CmonGraph, CmonJob, CmonRegExp, CmonDateTime.

1.3 Literals

Boolean literals: true and false.

String literals — single or double quoted; single-quoted strings may contain double quotes and vice versa. Adjacent string literals concatenate at parse time, C-style:

var carName1 = "Volvo XC60";
var carName2 = 'Volvo XC60';
var answer1  = "It's alright";
var answer2  = "He is called 'Johnny'";

var a = "one "
        "two "
        "three";
passed = a == "one two three";

Integer literals, including hex:

var1 = 0xff;
var2 = 0XFFFF;
passed = var1 == 255 && var2 == 65535;

Unsigned long long literals — an integer literal too large for Int is automatically stored as Ulonglong. A literal can also be forced to this type with an ull suffix:

var a = 91872698761001;
var b = 10ull;
passed = a.typeName() == "Ulonglong" && b.typeName() == "Ulonglong";

Double literals — any number not fitting an integer type:

var a = 10.2;
var b = 10.8E11;
var c = 2.8e-10;

Error literals — sentinel values representing "not a real value" conditions:

Literal	Meaning
`#NULL`	Null value
`#DIV/0!`	Division by zero
`#VALUE!`	Type mismatch, e.g. the log of a string
`#REF!`	Invalid variable reference, missing variable
`#NAME?`	Function name not found
`#NUM!`	Invalid numerical value, e.g. `sqrt(-1)`
`#N/A`	Value is not available — this is what an unpassed argument to `main()` evaluates to
`#SYNTAX!`	Syntax error
`#ARGS!`	Wrong argument count for a function call

Map literals — associative arrays that can hold any value type, including nested maps, indexed by string keys:

var a = {};
a["one"] = {};
a["one"]["two"] = "value";
passed = a.typeName() == "Map" && a["one"]["two"] == "value";

var testmap = {};
testmap["key1"] = "test";
testmap["key2"] = "test";
keys = testmap.keys();
for (i = 0; i < keys.size(); ++i) {
    print(keys[i] + ": " + testmap[keys[i]]);
}

Regular expression literals — same syntax as JavaScript, typed as CmonRegExp:

var regexp = /([0-9]+)x([0-9]+)/i;
var string = "s: 640x480";
string.replace(regexp, "$2x$1");
// string === "s: 480x640"

1.4 Preprocessor directives

#include "filepath"
#pragma once

#include works as in C/C++. Two kinds of path are used:

Built-in headers — cmon/alarms.h, cmon/io.h, cmon/graph.h — bring named constants into scope (severities and alarm types, toString() format specifiers, graph plot styles).
Script includes, such as common/mysql_helper.js or common/helpers.js — shared library files, included by their path relative to the project root, that provide reusable helper functions.

#pragma once protects an include file from being included multiple times.

2. Program structure

2.1 Entry points

main(...) — if a script defines main(), it runs after any top-level statements. The arguments passed to main() come from the calling environment; the return value of main() becomes the script's exit status, unless exit() was called first.

var global1 = 10;
function main(arg1) {
    return arg1 == "UtCmonImperative" && global1 === 10 && global2 === 11;
}
var global2 = 11;

exit(exitstatus) — ends script execution and returns the given exit status to the Cmon environment; a normal program termination.
abort() — aborts execution and presents a backtrace showing exactly where abort() was called.

If a function is called with the wrong number of arguments, the call evaluates to an #ARGS! error rather than raising an exception.

2.2 Input/output functions

print([value]...) — prints all arguments as one message at info severity, and returns the printed string.
warning([value]...) — same, at warning severity.
error([value]...) — same, at critical severity.

print("ERROR: ", retval["errorMessage"]);
print(host, ": ", msg);

3. Advisors, programs, and widgets

A CCDSL script is one .js file. Its location within the bundle and its calling convention determine what kind of script it is.

3.1 Advisors

An advisor lives under a category directory (for example mysql/innodb/, host/, ndbcluster/), runs on a schedule, and produces CmonAdvice for the cluster it examines. An advisor takes no arguments (or, occasionally, one optional flag) and follows this shape:

#include "common/mysql_helper.js"
#include "cmon/alarms.h"

var DESCRIPTION = "One or two sentences describing what this checks and why it matters.";
var TITLE = "Short human-readable title";
var OK_THRESHOLD = 999;
var WARNING_THRESHOLD = 990;

function main()
{
    var hosts = cluster::mySqlNodes();
    var advisorMap = {};

    for (idx = 0; idx < hosts.size(); ++idx)
    {
        host = hosts[idx];
        if (!host.connected())
            continue;

        var advice = new CmonAdvice();
        advice.setHost(host);
        advice.setTitle(TITLE);

        // ... compute a value, decide a severity ...

        advice.setSeverity(Ok);   // Ok, Warning, or Critical
        advice.setJustification("measured value and how it was computed");
        advice.setAdvice("what the administrator should do, if anything");

        advisorMap[idx] = advice;
    }
    return advisorMap;
}

DESCRIPTION, TITLE, and any threshold or message constants are declared as top-level vars before main(). main() returns a Map keyed by loop index, with one CmonAdvice per examined host — advisorMap[idx] = advice for each host, return advisorMap; at the end. This is the return contract every advisor implements, including ones that only ever produce a single entry.

cluster::hosts() can return the same physical host more than once — for example once by its controller role and once by its database role. When an advisor must examine each physical host only once, it deduplicates by host name:

var examinedHostnames = "";
for (idx = 0; idx < hosts.size(); ++idx) {
    host = hosts[idx];
    if (!host.connected()) continue;
    if (examinedHostnames.contains(host.hostName())) continue;
    examinedHostnames += host.hostName();
    ...
}

Rate- or ratio-based checks should confirm the server has been running long enough, and handling enough traffic, for the computed value to be meaningful, before comparing it against a threshold — see checkPrecond() in section 21.1.

Some advisors also remediate: they read a configuration value, compute a corrected one, apply it immediately with setGlobalVariable() (a runtime SET GLOBAL), and persist it with host.config() → config.setVariable() → config.save() (a config-file change, which only takes effect after a restart). Doing both keeps the running value and the on-disk value consistent.

An advisor that only applies to certain cluster types can check cluster::type() and return a single Ok advice with an explanatory justification for cluster types it does not support:

var clusterType = cluster::type().toString();
if (clusterType != "REPLICATION") {
    var advice = new CmonAdvice();
    advice.setSeverity(Ok);
    advice.setJustification("Advisor supports only REPLICATION clusters.");
    advice.setAdvice("Nothing to do.");
    advisorMap[0] = advice;
    return advisorMap;
}

An advisor is not required to produce CmonAdvice at all — a script whose purpose is a human-readable report can simply print() its output and return true;.

3.2 Programs

A program lives under a programs/ directory, is invoked on demand with explicit arguments (rather than on a schedule), and validates every argument before doing any work:

/**
 * explain query
 * Format:   db  query
 * db : "mydb"
 * query : "select 1"
 */
function main(db, q)
{
    var result = [];

    if (db.toString() == "" || db == #N/A || db.empty()) {
        result["error_msg"] = "Argument 'db' not specified";
        print(result["error_msg"]);
        exit(result);
    }
    if (q.toString() == "" || q == #N/A || q.empty()) {
        result["error_msg"] = "Argument 'query' not specified";
        print(result["error_msg"]);
        exit(result);
    }
    // ... perform the work, then exit(result) ...
}

An argument is checked against all three conditions — x.toString() == "", x == #N/A, and x.empty() — since any one check alone can miss a case depending on how the caller passed or omitted the argument. A doc comment above main() documenting the expected argument order and format is standard practice.

3.3 Widgets

A widget lives under a widgets/ directory and is invoked by a user-interface element rather than a schedule. Widgets follow the same argument-validation convention as programs and are typically grouped into subdirectories by the UI surface that calls them (for example widgets/config/, widgets/qm/ for query-monitor actions such as explain, kill, and processlist, widgets/schema/, widgets/usermgmt/). Because a widget backs a specific UI element, its header comment should note that modifying it can affect that UI feature.

4. Cluster and host objects

4.1 Cluster functions

Function	Returns
`cluster::hosts()`	every host considered part of the cluster, including the controller's own host, as plain `CmonHost` objects (see below)
`cluster::mySqlNodes()`	the `CmonMySqlHost`s of the cluster
`cluster::galeraNodes()`	the `CmonGaleraHost`s of the cluster
`cluster::groupReplNodes()`	the `CmonGroupReplHost`s of the cluster (MySQL Group Replication)
`cluster::postgreSqlNodes()`	the `CmonPostgreSqlHost`s of the cluster
`cluster::mongoNodes()`	the `CmonMongoHost`s of the cluster
`cluster::ndbdNodes()`	the `CmonNdbHost`s of the cluster (NDB Cluster data nodes)
`cluster::maxscaleNodes()`	the `CmonMaxScaleHost`s of the cluster
`cluster::proxysqlNodes()`	the `CmonProxySqlHost`s of the cluster
`cluster::id()`	the cluster's numeric ID
`cluster::name()`	the cluster's display name
`cluster::type()`	the cluster type, as a string (for example `"REPLICATION"`, `"GALERA"`, `"MONGODB"`)
`cluster::state()`	the numeric cluster state
`cluster::vendor()`	the database vendor of the cluster, as a string (for example `"percona"`, `"mariadb"`)
`cluster::rollingRestart()`	restarts the nodes of the cluster one at a time, without stopping it

cluster::hosts() always builds its result as literal CmonHost objects, regardless of the underlying node's real type — including for node types that have no dedicated cluster::*Nodes() accessor of their own (Redis/Valkey, MSSQL, Elasticsearch, and ClickHouse nodes; see section 4.4). Every CmonHost function documented in section 4.2 works on the result, but subtype-specific functions (section 4.3) require going through the matching cluster::*Nodes() call instead.

4.2 `CmonHost` tag functions

Identity and status:

host.hostName()          // string
host.port()              // int, usually the SQL server port
host.ipAddress()         // string, IPv4 address
host.dataHostName()      // string, the hostname used for data-layer connections
host.internalHostName()  // string, the hostname used for internal cluster communication
host.clusterId()         // int
host.nodeType()          // "controller" | "mysql" | "galera" | "mongo" | "maxscale" | "proxysql" |
                          // "redis" | "valkey" | "mssql" | "elastic" | "clickhouse" | ...
host.hostStatus()        // string, human-readable node status
host.role()              // "controller" | "master" | "slave" | ...
host.connected()         // boolean
host.message()           // human-readable status description
host.serverVersion()     // SQL server version string, or the Cmon version for the controller host
host.isAtLeastVersion(versionString)   // boolean, minimum-version check
host.dataDir()           // string, database data directory
host.distributionName() / .distributionCodeName() / .distributionRelease()
host.toMap()             // the host's properties as a Map
host.toJSonString()      // the host as a JSON string

host.toMap() returns a map of the host's properties; keys commonly used include connected, nodetype, isgalera, readonly, and, for Galera hosts, a nested galera map with galerastatus and localstatusstr (for example "Synced").

Statistics:

host.memoryInfo()                              // latest memory statistics, as a Map
host.memoryStats(startTime, endTime)           // memory statistics over a period, as a List
host.sqlInfo()                                  // latest SQL server statistics, as a Map
host.sqlStats(startTime, endTime)              // SQL server statistics over a period, as a List
host.mongoStats(startTime, endTime)            // MongoDB statistics over a period, as a List
host.networkInfo()                              // one Map per monitored network interface
host.diskInfo()                                 // one Map per monitored disk partition
host.diskStats(startTime, endTime, [deviceName]) // disk statistics over a period, as a List
host.cpuInfo()                                  // one Map per CPU core
host.cpuStats(startTime, endTime, [coreId])    // CPU statistics over a period, as a List

Each *Stats(startTime, endTime) call returns a List of per-sample Maps (each including a created timestamp field). Convert a List to an Array for statistical processing with .toArray(fieldNames) — see section 6.

Execution and configuration:

host.system(command)                        // shell command; {success, errorMessage, result}
host.executeSqlQuery(query)                 // SELECT-style query; {success, errorMessage, result}
host.executeSqlCommand(sqlCommand)          // INSERT/UPDATE/DDL-style command; {success, errorMessage}
host.executeMongoQuery(dbname, query)       // Mongo query; {success, errorMessage, result}
host.sqlPing([timeoutSeconds])              // boolean
host.sqlSystemVariable(name) / host.sqlSystemVariables()   // cached SHOW GLOBAL VARIABLES (SHOW ALL on PostgreSQL)
host.sqlStatusVariable(name) / host.sqlStatusVariables()   // cached SHOW GLOBAL STATUS
host.config([fileName])                     // configuration as a CmonClusterConfig, see [section 9](#9-configuration-files-cmonclusterconfig)

Alarms:

host.checkValue(type, value)     // checks a value against a registered check type, raising/clearing as needed
host.raiseAlarm(type, severity, [message])
host.clearAlarm(type)
host.alarms()                    // active alarms for this host

4.3 Subtype tag functions

CmonMySqlHost inherits all CmonHost properties and functions, and adds:

host.isGalera()   // boolean
host.uptime()     // ulonglong, SQL server uptime in seconds
host.readOnly()   // boolean, the 'read_only' SQL variable

Everywhere a CmonHost function is documented, it is also usable on CmonMySqlHost, CmonGaleraHost, CmonGroupReplHost, CmonPostgreSqlHost, CmonNdbHost, CmonMongoHost, CmonMaxScaleHost, and CmonProxySqlHost — this is the complete set of CmonHost subclasses a script can obtain a typed handle for, via the matching cluster::*Nodes() function in section 4.1 or via getSelf() inside a subtype-specific tag function.

4.4 Node types without a typed host class

Redis/Valkey, MSSQL (Availability Group or standalone), Elasticsearch, and ClickHouse nodes are fully managed cluster types, but none of them has a cluster::*Nodes() accessor or a corresponding entry in section 1.2's built-in type list — there is no cluster::redisNodes(), cluster::mssqlNodes(), cluster::elasticNodes(), or cluster::clickhouseNodes(), and no way to construct a typed handle for one of these hosts from a script.

These nodes are still reachable through cluster::hosts(), which returns them as plain CmonHost objects, distinguishable by host.nodeType() ("redis", "valkey", "mssql", "elastic", or "clickhouse"). Every generic CmonHost function from section 4.2 works on them normally — hostName(), port(), connected(), system(), raiseAlarm()/clearAlarm(), diskInfo()/diskStats(), cpuInfo()/cpuStats(), memoryInfo()/memoryStats(), networkInfo(), config(). What isn't available is a type-specific query method equivalent to executeSqlQuery() or executeMongoQuery(), so engine-specific data has to come from shelling out with host.system() to that engine's own CLI or REST API:

var hosts = cluster::hosts();
for (idx = 0; idx < hosts.size(); ++idx) {
    host = hosts[idx];
    if (host.nodeType() != "redis" && host.nodeType() != "valkey")
        continue;

    var retval = host.system("redis-cli -p " + host.port() + " INFO replication");
    if (retval["success"])
        print(retval["result"]);
}

The same pattern (a real, managed node type reached only through cluster::hosts() plus host.system(), with no dedicated query method) already applies to MaxScale-adjacent tooling such as maxscale/ms_admin.js, which shells out to maxctrl rather than calling a typed MaxScale query function.

5. Executing SQL commands and queries

host.executeSqlQuery(), host.executeSqlCommand(), and host.system() all return a Map with a success boolean, an errorMessage string, and, for queries and shell commands, a result value.

function getSqlVariable(host, variableName)
{
    var query = "SHOW GLOBAL STATUS LIKE '" + variableName + "'";
    var retval = host.executeSqlQuery(query);
    if (!retval["success"]) {
        print("ERROR:", retval["errorMessage"]);
        return #N/A;
    }
    var value = retval["result"][0, 1];   // two-dimensional indexing: row 0, column 1
    if (value.looksInteger()) return value.toInt();
    if (value.looksULongLong()) return value.toULongLong();
    if (value.looksDouble()) return value.toDouble();
    return value;
}

host.executeSqlQuery() and host.executeSqlCommand() work identically on CmonMySqlHost, CmonGaleraHost, CmonGroupReplHost, and CmonPostgreSqlHost — a PostgreSQL advisor can run the same query/command pattern shown here against a CmonPostgreSqlHost, substituting PostgreSQL SQL for MySQL SQL (for example SELECT * FROM pg_stat_activity in place of SHOW GLOBAL STATUS).

Executing a shell command:

function listFiles(host)
{
    var retval = host.system("ls -lha /home");
    if (!retval["success"])
        error("ERROR: ", retval["errorMessage"]);
    print("Host    : ", host.hostName());
    print("Result  : ", retval["result"]);
    print("Success : ", retval["success"]);
    return retval["success"];
}

Executing a query against the Cmon database itself (rather than a managed host):

var retval = CmonDb::executeSqlQuery("select * from mysql_states;");
if (!retval["success"])
    error("Executing SQL query failed: ", retval["errorMessage"]);

for (idx = 0; idx < retval["result"].rows(); ++idx) {
    print(retval["result"][idx, 2]);
}

6. Obtaining and processing statistical information

#include "cmon/io.h"
function toGigaBytes(value) { return value / (1024 * 1024 * 1024); }
function main()
{
    var host = cluster::hosts()[0];
    var endTime = CmonDateTime::currentDateTime();
    var startTime = endTime - 10 * 60;
    var stats = host.memoryStats(startTime, endTime);
    for (idx = 0; idx < stats.size(); ++idx) {
        map = stats[idx];
        created = CmonDateTime::fromUnixTime(map["created"]);
        ramfree = toGigaBytes(map["ramfree"]);
        print(created.toString(LongTimeFormat), " ", ramfree.toString(TwoDecimalNumber), "GBytes");
    }
    return true;
}

For most tasks, converting a List to an Array with .toArray(fieldName) and using the high-level statistical functions is more efficient than looping over individual samples:

function printUtil(host, startTime, endTime)
{
    var list = host.memoryStats(startTime, endTime);
    var array = list.toArray("memoryutilization");
    var min = min(array);
    var max = max(array);
    var ninth = percentile(array, 0.9);
    print(host.hostName(), " ",
        min.toString(TwoDecimalPercent), " - ",
        ninth.toString(TwoDecimalPercent), " - ",
        max.toString(TwoDecimalPercent));
    return true;
}

.toArray() also accepts a comma-separated list of fields to produce a multi-column array, useful for graphing or forecasting aligned x/y series:

var stats = host.sqlStats(startTime, endTime);
var array = stats.toArray("created,interval,COM_SELECT,COM_INSERT");

Linear forecasting from historical data:

var list = host.diskStats(startTime, endTime, device);
var freeOverTime     = list.toArray("free");
var timeOfEachSample  = list.toArray("created");
var expectedFree = forecast(futureTime.toInt(), freeOverTime, timeOfEachSample);
expectedFree = expectedFree < 0 ? 0 : expectedFree;   // linear regression can predict below zero

7. Advice: `CmonAdvice`

CmonAdvice represents an action to be taken by the administrator, together with the information needed to understand why it was raised.

var advice = new CmonAdvice();
advice.setHost(host);
advice.setTitle("Short title");
advice.setSeverity(Warning);          // Ok, Warning, or Critical — from #include "cmon/alarms.h"
advice.setJustification("Measured 96% CPU over the last 60 minutes.");
advice.setAdvice("Investigate top queries or scale up the instance.");

Function	Description
`advice.setTitle(title)` / `advice.title()`	a short description of the advice
`advice.setCreator(creator)` / `advice.creator()`	the owner that created the advice; defaults to the source file
`advice.setJustification(text)` / `advice.justification()`	a detailed, human-readable description of the measured value and why an action is suggested
`advice.setAdvice(text)` / `advice.advice()`	the recommended action
`advice.setSeverity(severity)` / `advice.severity()`	the severity level: `Ok` (0), `Warning` (1), or `Critical` (2)
`advice.setHost(host)`	the host the advice was generated for

advice.toString(format) supports these format specifiers:

Specifier	Meaning
`%t`	the title
`%j`	the justification
`%a`	the advice text
`%c`	the creator
`%h`	the host name, if set
`%E`	a multi-line summary of all of the above

An advisor's main() returns a Map keyed by loop index, holding one CmonAdvice per examined host: advisorMap[idx] = advice;, then return advisorMap;.

8. Alarms

#include "cmon/alarms.h" provides the severity constants (Ok, Warning, Critical) and a set of predefined alarm-type constants that can be raised directly, without registration:

#include "cmon/alarms.h"
host.raiseAlarm(HostDiskUsage, Warning, "message");
host.clearAlarm(HostDiskUsage);

Predefined alarm types include HostDiskUsage, StorageUtilizationPrediction, MySqlReplicationConfigProblem, ClusterSystemCheck, MySqlAdvisor, MySqlTimeZoneCheck, StorageMyIsam, MySqlTableAnalyzer, MySqlIndexAnalyzer, and MySqlAutoIncrementAnalyzer, among others. A condition specific to an engine that has no matching predefined type (for example PostgreSQL table bloat) needs a custom alarm type registered as shown below before it can be raised with host.raiseAlarm(); until such a type exists, an advisor can only report the condition through CmonAdvice severity.

For conditions that don't match a predefined type, register a custom one:

#include "cmon/alarms.h"
function myAlarm() {
    return Alarm::alarmId(
        Node, true,
        "Computer is on fire",
        "The computer is on fire, it is on flames.",
        "Pour some water on it.");
}
var myAlarmId = myAlarm();
var host = cluster::hosts()[0];
host.raiseAlarm(myAlarmId, Critical);

Alarm::alarmId(category, isHost, title, message, recommendation) registers a new alarm type (or returns the existing one if an identical type is already registered) and returns its ID.
Alarm::checkId(category, isHost, warningLevel, criticalLevel, title, message, recommendation) registers a check type — an alarm with warning and critical numeric thresholds — for use with host.checkValue(type, value).

Reading active alarms:

#include "cmon/alarms.h"
var alarms = host.alarms();
var found = false;
for (idx = 0; idx < alarms.size(); ++idx) {
    if (alarms[idx]["title"] == "MySQL advisor alarm") {
        found = true;
        break;
    }
}

An alarm that is raised in a failing branch should be cleared with a matching clearAlarm() call in the corresponding healthy branch, so that it does not remain active once the underlying condition is resolved.

9. Configuration files: `CmonClusterConfig`

function getConfiguredClientPort(host)
{
    var config   = host.config();
    var variable = config.variable("port");
    for (idx = 0; idx < variable.size(); ++idx) {
        print("*** section  : ", variable[idx]["section"]);
        print("*** value    : ", variable[idx]["value"]);
        print("*** location : ", variable[idx]["filepath"], ":", variable[idx]["linenumber"]);
        if (variable[idx]["section"] == "client")
            return variable[idx]["value"].toInt();
    }
    return #N/A;
}

Function	Description
`host.config([fileName])`	loads the configuration from the host, together with its include files and directories, as a `CmonClusterConfig`
`config.errorMessage()`	a human-readable message describing the state of the last operation
`config.variable([variableName])`	a `List` of `Map`s (`variablename`, `linenumber`, `value`, `filepath`, `section`); returns every variable in the file if no name is given
`config.setVariable(section, variableName, value)`	sets (or adds) a variable in the given section; changes the in-memory configuration object only
`config.save()`	saves the configuration back to the original host(s) and file name(s); returns `{success, errorMessage}`

Changing a variable with config.setVariable() and config.save() updates the configuration file; it does not change the running value. To apply a dynamic variable immediately as well, issue a SET GLOBAL through host.executeSqlCommand() in addition to saving the configuration file.

10. Files: `CmonFile`

CmonFile represents a single file — either local to the controller, or on a managed host — and provides read-only inspection and reading functions. It is a different tool from CmonClusterConfig (section 9): CmonClusterConfig understands a config file's section/variable/value structure, while CmonFile reads any file as raw text or lines, or just inspects its metadata.

var file = new CmonFile("/etc/mysql/my.cnf", host);   // a file on a managed host
// var file = new CmonFile("/etc/cmon.cnf");           // a file on the controller itself

if (!file.exists()) {
    print("File not found: ", file.fullPath());
} else if (file.isReadable()) {
    var content = file.readTxtFile();
    var lines   = file.readLines();
    print(file.fileName(), " is ", content.length(), " bytes, ", lines.size(), " lines");
}

Function	Description
`new CmonFile(path, [host])`	opens `path`; with no `host` argument the file is on the controller, otherwise on the given `CmonHost`
`file.fileName()`	the file name, without its directory
`file.baseName()`	the file name without its extension
`file.fullPath()`	the full, absolute path
`file.isRemote()`	true if the file is on a managed host rather than the controller
`file.exists()`	true if the file exists
`file.isDirectory()`	true if the path is a directory
`file.isExecutable()`	true if the file has execute permission
`file.isReadable()`	true if the file has read permission
`file.readTxtFile()`	the file's contents as a `String`
`file.readLines()`	the file's contents as a `List` of lines
`file.errorString()`	a human-readable message describing the last failed operation

11. Jobs: `CmonJob`

A CmonJob represents a task the Cmon controller executes, usually asynchronously: the script builds a CmonJob, sets its properties, and enqueues it; the controller then executes it and reports the results to the UI.

var job = CmonJob::createBackupJob("127.0.0.1", "/var/tmp");
job.setBackupMethod("mysqldump");
var passed = job.enqueue();
if (passed !== true) {
    error("ERROR: ", job.errorString());
    exit(false);
}

Function	Description
`job.enqueue()`	checks the job for consistency and sends it to the execution queue; returns `true` on success
`job.errorString()`	a human-readable error, or an empty string if there was none
`job.jobId()`	the job's numeric ID; valid (greater than 0) only after a successful `enqueue()`
`CmonJob::getJob(jobId)`	reads a previously enqueued job from the Cmon database; `#N/A` if not found
`CmonJob::createBackupJob(host, dir)`	creates a job that backs up a host's data
`CmonJob::createDoCheckJob()`	creates a job that runs schema/index checks on the controller
`CmonJob::createAddNodeJob(hostName, [install], [configFile])`	creates a job that adds a new node to the cluster
`job.setBackupMethod(method)`	`null`/`"auto"`/`"none"` (default method), `"mysqldump"`, `"xtrabackupfull"`, `"xtrabackupincr"`, or `"pgdump"`
`job.setCompression(compression)`	whether the backup file should be compressed
`job.setIncludeDatabases(value)`	which databases to include
`job.setIsCcStorage(value)`	whether the controller should store the backup file
`job.setNetcatPort(value)`	the netcat port used when copying the backup file over the network

12. Graphs: `CmonGraph`

#include "cmon/graph.h"
var host      = cluster::hosts()[0];
var endTime   = CmonDateTime::currentDateTime();
var startTime = endTime - 10 * 60;
var stats     = host.sqlStats(startTime, endTime);
var array     = stats.toArray("created,interval,COM_SELECT,COM_INSERT");

for (idx = 0; idx < array.columns(); idx++) {
    array[5, idx] = 1000 * array[2, idx] / array[1, idx];
    array[6, idx] = 1000 * array[3, idx] / array[1, idx];
}

var graph = new CmonGraph;
graph.setXDataIsTime();
graph.setTitle("SQL Statistics " + host.toString());
graph.setSize(800, 600);
graph.setPlotLegend(1, "Select (1/s)");
graph.setPlotColumn(1, 0, 5);
graph.setPlotStyle(1, Impulses);
graph.setPlotLegend(2, "Insert (1/s)");
graph.setPlotColumn(2, 0, 6);
graph.setPlotStyle(2, Impulses);
graph.setData(array);
exit(graph);

Function	Description
`graph.setTitle(title)`	text shown at the top of the graph image
`graph.setSize(width, height)`	size of the generated graph, in pixels
`graph.setXDataIsTime([boolean])`	whether X-axis values should be printed as date/time
`graph.setPlotLegend(plotIdx, legend)`	legend text for the given plot
`graph.setPlotColumn(plotIdx, xColumn, yColumn)`	which array columns feed the given plot
`graph.setPlotStyle(plotIdx, style)`	plot rendering style; available styles are defined in `cmon/graph.h`

Plot indices are 1-based.

13. JSON functions and the Cmon database

var req = new Map;
req["operation"] = "clusters";
var retval = JSON::postRequest("http://localhost:9500/0/clusters", req);
print("retval of clusters:\n" + retval);

Function	Description
`JSON::parse(text)`	parses a JSON string, returns it as a `Map`
`JSON::toString(map)`	converts a `Map` to a well-formatted JSON string
`JSON::postRequest(url, map)`	sends a POST request to the given URL with the JSON-encoded map as its body, returning the reply
`CmonDb::executeSqlQuery(query)`	executes a query against the Cmon database (the controller's own metadata store); returns `{success, errorMessage, result}`

14. Date and time: `CmonDateTime`

CmonDateTime is distinct from JavaScript's Date type. It supports direct arithmetic in seconds:

var endTime    = CmonDateTime::currentDateTime();
var startTime  = endTime - 60 * 60;          // one hour ago
var futureTime = endTime + 2 * 24 * 3600;    // two days from now

Function	Description
`CmonDateTime::currentDateTime()`	the current real-time clock time from the controller host
`CmonDateTime::fromUnixTime(time)`	converts a Unix timestamp to a `CmonDateTime`
`CmonDateTime::fromString(string)`	parses a date/time string
`dateTime.second()` / `.minute()` / `.hour()`	time-of-day components
`dateTime.month()`	the month, 1–12
`dateTime.year()`	the year, e.g. 2024
`dateTime.weekday()`	day of the week, Sunday = 1 … Saturday = 7
`dateTime.timeZone()`	seconds to add to local time to get UTC
`dateTime.dayLight()`	seconds to add for daylight saving time

toString([format]) accepts either a named format constant from cmon/io.h, or a strftime()-style format string:

#include "cmon/io.h"
var dateTime = CmonDateTime::fromUnixTime(1424686003);
str01 = dateTime.toString(FileNameFormat);       // "2015-02-23_110643"
str03 = dateTime.toString(LogFileFormat);        // "Feb 23 11:06:43"
str04 = dateTime.toString(MySqlLogFileFormat);   // "2015-02-23 11:06:43"
str09 = dateTime.toString(ShortDateFormat);      // "02/23/15"
str10 = dateTime.toString(LocalDateTimeFormat);  // "Mon Feb 23 11:06:43 2015"
str11 = dateTime.toString(EmailDateTimeFormat);  // "Mon, 23 Feb 2015 11:06:43 +0100"

15. General value functions

Available on any value:

Function	Description
`value.typeName()`	the type name of the value
`value.toString([formatId])`	converts to string, optionally using a format specifier from `cmon/io.h`
`value.empty()`	true if a string has no characters, or a container has no items
`value.size()`	character count for strings, item count for containers; for an `Array`, the number of columns
`value.isNull()`	true for a null string, e.g. an unset SQL value
`value.isInvalid()`	true if the value was never set
`value.isString()` / `.isInt()` / `.isULongLong()` / `.isDouble()` / `.isBoolean()` / `.isNumber()` / `.isError()` / `.isMap()` / `.isList()` / `.isArray()`	type checks
`value.toInt()` / `.toULongLong()` / `.toDouble()` / `.toBoolean()`	conversions

#include "cmon/io.h"
var theDouble = 42.0;
var str1 = theDouble.toString(TwoDecimalNumber);   // "42.00"
var str2 = theDouble.toString(FourDecimalNumber);  // "42.0000"

16. String functions

Tag functions on a string value:

Function	Description
`string.length()`	number of characters
`string.indexOf(substring, [start])`	position of the first match, or -1
`string.split(separator)`	array of substrings
`string.substr(begin, length)`	substring extraction
`string.trim()`	removes leading/trailing whitespace
`string.toLowerCase()` / `.toUpperCase()`	case conversion
`string.contains(substring)`	boolean
`string.replace(from, to)`	substring replacement
`string.startsWith(other)` / `.endsWith(other)`	boolean, prefix/suffix check
`string.leftAlign(length)` / `.rightAlign(length)`	pads with trailing/leading spaces up to `length` (capped at 1000) — left- or right-aligns the text within a fixed-width field
`string.search(pattern)`	position of the first match of `pattern` (a string or a `CmonRegExp`), or -1
`string.looksInteger()` / `.looksULongLong()` / `.looksDouble()` / `.looksBoolean()` / `.looksEmail()` / `.looksIpAddress()`	format checks
`string.toInteger()` / `.toULongLong()`	conversions

Free functions: asc(text), char(number) / chr(number), left(text, n), right(text, n), startswith(text1, text2), endwith(text1, text2), mid(text, start, length), escape(text) / unescape(text), upper(text) / lower(text), trim(text), len(text), concatenate(text, [text]...).

17. Mathematical, array, and statistical functions

Mathematical: rand(), pi(), degrees(n), radians(n), sign(n), sin/asin/sinh, cos/acos/cosh/acosh, tan/atan/tanh/atanh, fisher(n)/fisherinv(n), log(n, [base]), sqrt(n), abs(n), exp(n), floor(n, [significance]), ceiling(n, [significance]), round(n, digits), roundup(n, digits), rounddown(n, digits), mround(n, multiple), even(n), iseven(n), odd(n), isodd(n), convert(n, fromUnit, toUnit) (byte, kbyte, mbyte, gbyte, tbyte, celsius, kelvin, fahrenheit, hz, mhz, ghz).

Array: choose(position, value, [value]...), transpose(array), filter rows(array, column, value), columns(array), rows(array), vlookup(value, array, column, [notExact]), hlookup(value, array, column, [notExact]), match(value, array, [matchType]).

Statistical — accept either individual arguments or arrays:

b = [10, 8, 5];
c = average(b);
d = average(10, 11, 12);

count, countblank, min, max, sum, sumsq, product, average, geomean, mode, emaverage(alpha, ...), median, percentile(array, [n]), small(array, n), large(array, n), stdev, avedev, pearson(array1, array2), correl(array1, array2), covar(array1, array2), devsq, var, forecast(x, knownYValues, knownXValues), linest(knownYValues, knownXValues), slope(knownYValues, knownXValues), intercept(knownYValues, knownXValues).

Value classification: iserr(value), isnumeric(value), istext(value), isnumber(value), isarray(value).

18. Regular expression functions

CmonRegExp tag functions:

Function	Description
`regexp.test(string)`	true if the pattern matches
`regexp.lastIndex()`	index where the next match check begins; 0 without the "global" modifier
`regexp.match(string)`	a `List` with the matched text at index 0, and sub-expression matches from index 1
`regexp.exec(string)`	a `Map` with detailed match information

toString() format specifiers for CmonRegExp: %r (the pattern itself), %j (pattern and modifiers in JavaScript notation, e.g. /[0-9]+/ig), %m (the matched string, if any), %nm (the nth matched sub-expression).

19. Cmon, license, and introspection functions

Function	Description
`cmon::version()`	the Cmon version, as a string
`cmon::build()`	the Cmon build number
`cmon::uptime()`	seconds since Cmon started managing this cluster
`cmon::running()`	true if Cmon is managing this cluster
`cmon::hostname()`	hostname of the controller's computer
`cmon::domainname()`	domain name of the controller's computer
`license::statustext()`	a short description of the license status
`license::status()`	true if the cluster has a valid license
`license::expires()`	days remaining on the license; negative if expired or not found
`conf::values()`	the controller's own `cmon.cnf`-level configuration, as a `Map`
`CmonMetaType::getParamSpec(className, propertyName)`	the metadata (as a `Map`) describing one property of one internal class
`CmonClusterInfo::getClusterInfo([clusterId])`	a `CmonClusterInfo` for the given cluster, or the current one if omitted
`CmonServer::containers()`	the containers known to this controller (LXC/cloud), as a `CmonContainer`
`CmonUser::currentUser()`	the `CmonUser` the script is running as

conf::values() is how a script checks a controller-level toggle before doing expensive work — for example, several information_schema-based MySQL advisors check enable_is_queries before running:

var cmonConfig = conf::values();
if (cmonConfig.keys().contains("enable_is_queries") &&
    !cmonConfig["enable_is_queries"].toBoolean())
{
    print("Information_schema queries are not enabled.");
    exit(result);
}

20. Advisor bundle structure

An advisor bundle is a tarball (for example s9s.tar.gz) containing a project.conf file at its root and a directory tree of .js scripts organized by category.

20.1 Directory layout

s9s.tar.gz
└── s9s/
    ├── project.conf
    ├── host/                      # OS-level checks
    │   └── widgets/
    ├── mysql/
    │   ├── auto_tuners/           # advisors that also change configuration
    │   ├── connections/
    │   ├── galera/
    │   ├── general/
    │   ├── group_repl/
    │   ├── health/
    │   ├── i_s/                   # information_schema-based checks
    │   ├── innodb/
    │   ├── p_s/                   # performance_schema-based checks
    │   ├── programs/              # on-demand scripts, not scheduled
    │   ├── query_cache/
    │   ├── replication/
    │   ├── schema/
    │   ├── security/
    │   ├── table_cache/
    │   └── widgets/
    │       ├── config/
    │       ├── qm/                 # query monitor: explain, kill, processlist
    │       ├── schema/
    │       └── usermgmt/
    ├── mongodb/
    │   ├── connections/, mmap/, replication/, sharding/
    │   └── widgets/qm/
    ├── maxscale/
    ├── ndbcluster/
    ├── predictions/                # forward-looking checks
    └── reports/                    # print-only output, no CmonAdvice

common.tar.gz
└── common/
    ├── project.conf
    ├── helpers.js                  # generic, non-database-specific helpers
    ├── mysql_helper.js             # SQL helpers shared by MySQL/Galera advisors
    └── cmonrpc.js                  # helpers for calling the Cmon RPC API from a script

A shared library such as common/ contains helper .js files, included by the other bundles with #include "common/mysql_helper.js", and is not itself scheduled.

Neither tarball currently contains a category directory for PostgreSQL, Redis/Valkey, MSSQL, Elasticsearch, or ClickHouse; project.conf already reserves [schedules-postgresql_single] (see below) but it has no entries.

20.2 `project.conf`

[project]
# Increase this value to force cmon to reinstall the scripts (a backup
# of the previous version is made automatically) on the next start/upgrade.
version=20210922
# Increase this value when only the [schedules*] sections below changed.
schedule_version=1029

[schedules]
# Applies to every cluster type.
s9s/host/cpu_usage.js="*/30 * * * *"
s9s/host/disk_space_usage.js="*/30 * * * *"
s9s/host/swappiness.js="0 1 * * *"

[schedules-mysqlcluster]
s9s/mysql/connections/connections_used_pct.js="*/1 * * * *"
s9s/mysql/schema/schema_check_nopk.js="0 1 * * *"

[schedules-replication]
...

[schedules-group_replication]
...

[schedules-galera]
...

[schedules-mysql_single]
...

[schedules-postgresql_single]
...

[schedules-mongodb]
...

[project] holds a version and a schedule_version. Increasing version causes cmon to reinstall the bundle's scripts (backing up the previous versions first); increasing schedule_version is sufficient when only the schedule sections changed.
[schedules] lists scripts scheduled for every cluster type.
[schedules-<clustertype>] lists scripts scheduled only for that cluster type; <clustertype> is the same cluster-type identifier used when the cluster was created (for example mysqlcluster, replication, group_replication, galera, mysql_single, postgresql_single, mongodb).
Each entry's key is the script's path relative to the tarball root, including the top-level directory name, and its value is a quoted five-field cron expression.
A script that is present in the bundle but commented out of its [schedules*] entry is shipped but not scheduled by default.
Scripts under a programs/ or widgets/ directory are not listed in any [schedules*] section; they are invoked on demand instead.
common/project.conf contains only a [project] section with a version, since the common bundle is a shared library and has nothing to schedule.

21. Shared helper library (`common/*.js`)

21.1 `common/mysql_helper.js`

Function	Description
`readVariable(host, name)`	runs `SHOW GLOBAL VARIABLES LIKE '<name>'`; returns the value, or `false` on error
`readStatusVariable(host, name)`	runs `SHOW GLOBAL STATUS LIKE '<name>'`; returns the value, or `false` on error
`getSingleValue(host, query)`	runs a query, returns the single scalar from its first row/column
`getValueMap(host, query)`	runs a query, returns the result set as a `{row: {column: value}}` map
`getResultSet(host, query)`	runs a query, returns the raw `executeSqlQuery()` reply
`executeSqlCommand(host, query)`	runs a command, prints any error, and returns `true`/`false`
`executeSqlCommand2(host, query)`	runs a command and returns the raw reply map
`setGlobalVariable(host, variable, value)`	builds and runs the appropriate `SET GLOBAL variable=value` statement, quoting the value correctly for numeric or string types
`mySleep(host, seconds)`	runs `SELECT SLEEP(seconds)`
`checkPrecond(host)`	returns true when the host has been up for more than an hour and is handling more than 10 queries per second — used to avoid evaluating rate- or ratio-based checks against a freshly started or idle server
`isSystemTable(name)`	true for `performance_schema`, `information_schema`, `mysql`, or `sys`
`checkHostVersion(host, versionPrefix)`	true if the host's server version starts with the given prefix
`isMySqlHost(host)`	true for MySQL or Galera node types
`isMySql80Host(host)` / `isMySql55Host(host)`	version-specific MySQL checks
`isMariaDb100Host(host)` / `isMariaDb101Host(host)` / `isMariaDb102Host(host)` / `isMariaDb103Host(host)` / `isMariaDb104Host(host)` / `isMariaDb10xHost(host)`	version-specific MariaDB checks

Despite the filename, getSingleValue(), getValueMap(), and getResultSet() are just thin wrappers around host.executeSqlQuery() and carry no MySQL-specific logic, so they are safe to reuse from a PostgreSQL advisor. readVariable() and readStatusVariable() are MySQL-specific (SHOW GLOBAL VARIABLES/SHOW GLOBAL STATUS syntax) and have no PostgreSQL equivalent in this file.

21.2 `common/helpers.js`

Function	Description
`hostMatchesFilter(host, hostAndPort)`	true if a host matches a `"hostname:port"` filter string, or if the filter is empty or `"*"`
`executeOnController(command)`	runs a shell command specifically on the host whose node type is `controller`
`isMongoDbPrimary(host)`	true if the given MongoDB host is currently the primary, using the version-appropriate admin command (`hello` on newer releases, `isMaster` on older ones)

21.3 `common/cmonrpc.js`

Helpers for calling the Cmon RPC API from within a script, by running curl on the controller host:

Function	Description
`getCmonController()`	returns the `CmonHost` whose role is `controller`, or `false` if none is found
`curl(controller, path, json)`	POSTs a JSON payload to the controller's RPC endpoint for the given cluster and path, returning the result
`mapToJSON(map)` / `JSONToMap(text)`	converts between `Map` and JSON text
`addNode(node)`	enqueues an `addnode` job for the given hostname via the RPC API
`setCmonrpcToken(json, clusterId)`	attaches the RPC authentication token for the given cluster ID to a request payload
`findUnusedHost()`	returns the first configured free host not already part of the cluster

CMONRPC_HOST, CMONRPC_PORT, and CMONRPC_TOKEN are the controller's RPC host, port, and per-cluster authentication tokens, referenced by these helpers when building a request.