Usage Metering and Billing

ClusterControl Usage Metering provides billing-grade accounting of the database-server estate under management. For every collection tick, it records one snapshot per eligible node (hardware class, RAM, vCPU, disk, tags), and at report time it computes per-period high-watermarks, a filtered or estate-wide summary, and a cryptographic seal that makes the report tamper-evident. These sealed reports can back invoices, tenant chargebacks, and audit trails — a typical use case is a Managed Service Provider (MSP) reporting its billable estate.

Metering runs in a standalone service, cmon-telemetry, which polls ClusterControl (via cmon-proxy) on a configurable interval, stores per-node snapshots in a local SQLite database, and seals each generated report with two signatures: HMAC-SHA256 (internal integrity) and Ed25519 (third-party verifiable operator signature). A collection tick is two HTTP calls and a database write — there is no streaming pipeline to operate.

Note

Usage metering requires ClusterControl 2.5.0 or newer, with cmon-proxy, cmon-telemetry, and the ClusterControl controller all upgraded together.

Architecture at a glance

ClusterControl GUI ──► cmon-proxy ──► cmon-telemetry (:9520 REST) ──► SQLite
                          ▲                │
                          │                │ POST /proxy/controllers/status   (discovery)
                          │ Bearer         │ POST /v2/clusters {getMeteringData}
                          │ service token  │
                          └────────────────┘

cmon-proxy ──► controllers   (using the bound service user's cmon keyfile)

cmon-telemetry (the service) polls cmon-proxy each tick for snapshots, stores them in SQLite, produces sealed reports on demand, and exposes a REST API on port 9520.
cmon-proxy (the credential gate) holds the controller-side RSA keyfiles in service_users[] and issues a Bearer service token for cmon-telemetry to call back through. Calls to the controllers happen as the bound service user, not as any operator session.
Billing UI lives at /billing in the ClusterControl web application. It is the operator interface to generate, view, verify, download, and delete reports.

Prerequisites

Component	Requirement
ClusterControl	2.5.0 or newer (`cmon-proxy` + `cmon-telemetry` + controller upgraded together)
`clustercontrol-telemetry` package	Installed; typically on the same host as `cmon-proxy` (a collocated install is the documented path)
`cmon-proxy` configuration	`billing_enabled: true` in `ccmgr.yaml`; service user + token provisioned via `ccmgradm serviceauth-bootstrap` (see below)
Ports	`9520` inbound on the `cmon-telemetry` host (REST). Outbound from `cmon-telemetry` to `cmon-proxy`'s `19051` (`/proxy/` and `/v2/`).
Operating system	Debian/Ubuntu or RHEL-based distributions (DEB/RPM packages)
Go runtime	None — `cmon-telemetry` is a single static binary

Installing cmon-telemetry

Pick one host. It can be the same host as cmon-proxy for collocated installs, or a separate host for split deployments.

Debian/UbuntuRHEL/CentOS/Rocky

sudo apt-get update
sudo apt-get install -y clustercontrol-telemetry

sudo yum install -y clustercontrol-telemetry

The package lays down:

/usr/bin/cmon-telemetry — the binary
/etc/cmon-telemetry/config.yaml — configuration template (preserved on upgrade)
/var/lib/cmon-telemetry/ — SQLite data directory and the default home for the Ed25519 key file
/lib/systemd/system/cmon-telemetry.service — systemd unit

Generate a signing key and Bearer token

# Signing key for sealed reports (SHA-256 + HMAC)
SIGNING_KEY=$(openssl rand -hex 32)
echo "Signing key: $SIGNING_KEY"

# Bearer token for REST auth
API_TOKEN=$(openssl rand -hex 24)
echo "API token:   $API_TOKEN"

Keep both values safe — you will paste them into the configuration file below. The signing key only needs to survive the retention window (12 months by default); the Bearer token can be rotated at any time.

Note

You do not generate the Ed25519 keypair by hand. cmon-telemetry writes a fresh Ed25519 keypair to ed25519_key_file (default /var/lib/cmon-telemetry/signing.ed25519) on first start, with permissions 0600. The public-key fingerprint then shows up in the startup log and on the /billing/status endpoint.

Write the configuration file

Edit /etc/cmon-telemetry/config.yaml:

# REST listen socket — Billing UI + curl /billing/* hit here
api_listen: "127.0.0.1:9520"

# Storage
db_path: /var/lib/cmon-telemetry/metering.db
retention_months: 12
log_level: "info"
# logfile: ""             # Empty = stdout/stderr (journald). Set to a path for on-disk logging.

# --- Internal integrity seal (HMAC-SHA256, symmetric) ----------------
signing_key: "<SIGNING_KEY generated above>"
key_id:      "key-2026-Q2"
verification_keys:
  key-2026-Q2: "<same SIGNING_KEY>"
  # Add previous keys here during rotation; the list is used by verifyReport

# --- Third-party verifiable export (Ed25519, asymmetric) -------------
# Auto-generated on first start if the file is missing. Never share the
# file — only the public-key fingerprint (visible in /billing/status + UI).
ed25519_key_file: /var/lib/cmon-telemetry/signing.ed25519

# --- REST auth -------------------------------------------------------
# Required for /billing/{reports,snapshots,controllers/status}.
# /billing/status stays open as a health probe.
api_token: "<API_TOKEN generated above>"
# api_tls_cert: /etc/cmon-telemetry/api-server.crt
# api_tls_key:  /etc/cmon-telemetry/api-server.key

# --- Billing collection ----------------------------------------------
# cmon-telemetry holds NO controller credentials. All traffic flows
# through cmon-proxy. Provision cmon-proxy's side as described in the
# next section; the bootstrap drops the token at the default token_file
# path so a collocated install needs zero pasting.
billing:
  enabled: true
  cmon_proxy:
    url: "https://localhost:19051"
    insecure: false                          # set true ONLY for testing with self-signed certs
    # token: "<paste-if-not-using-token_file>"
    token_file: /usr/share/ccmgr/service-tokens/cmon-telemetry
  interval:               60m                # production default
  parallelism:            4
  per_controller_timeout: 30s
  period_months:          1                  # default report period when generateReport is called without dates
  min_active_hours:       24                 # nodes must clear this to count as billable

Start the service

sudo systemctl enable --now cmon-telemetry
sudo systemctl status cmon-telemetry

Expected output ends with active (running).

Check the journal for the keypair line on first start:

sudo journalctl -u cmon-telemetry | grep -i ed25519 | head -2
# generated Ed25519 keypair: fingerprint=sha256:27dccd92548741af,
#   file=/var/lib/cmon-telemetry/signing.ed25519, public_key=962d9fb98a3ed1c2...

On subsequent restarts, the line reads loaded Ed25519 keypair: fingerprint=sha256:.... The fingerprint is what you share with the party that verifies your exported reports (see Report seals and verification).

Verify the service is listening:

sudo ss -tlnp | grep -E ':9520\s'
# LISTEN … 127.0.0.1:9520 …

Provisioning the cmon-proxy service token

cmon-telemetry needs a Bearer token to call cmon-proxy. The token resolves on cmon-proxy's side to a service_user (a controller-side RSA keypair) that authenticates against each controller. There are two provisioning paths — Path A is the standard, Path B is for manual or scripted installs.

Path A — `ccmgradm serviceauth-bootstrap` (recommended)

This provisions everything end to end: it generates an RSA keypair for the controller-side service user, registers the public key on each controller listed in instances: via the createUser RPC, writes the service_users[] + service_tokens[] entries to ccmgr.yaml, and drops the token at the default location cmon-telemetry reads from.

Run on the cmon-proxy host, as root (so the generated files end up owned by the daemon user):

CCMGR_ADMIN_PASSWORD='<your-admin-password>' \
  sudo -E ccmgradm serviceauth-bootstrap --name cmon-telemetry --admin-user admin

The environment-variable form keeps the password off the shell-history line. The admin credentials are required at bootstrap time only and are never persisted to ccmgr.yaml.

Per-controller outcomes print at the end (✓ created / · user already existed (pubkey re-registered) / ✗ error). The token is printed once with a "save this" note; cmon-telemetry's default token_file (/usr/share/ccmgr/service-tokens/cmon-telemetry) reads it automatically — no pasting is needed on a collocated install.

sudo systemctl restart cmon-proxy    # reload the new ccmgr.yaml

Multi-controller pools

Bootstrap iterates the primary URLs in instances:. The controller replicates the user record across pool members, but per-user public keys do not propagate — on a multi-controller pool, only the primary's metering fetch succeeds initially. Workaround — re-run bootstrap after adding or replacing pool members:

ccmgradm serviceauth-bootstrap --name cmon-telemetry --admin-user admin
# then on each controller pool member, if the user got suspended during the gap:
s9s user --unsuspend cmon-telemetry

Path B — manual via `s9s`

For deployments without ccmgradm, or when you want explicit control over the keypair and ccmgr.yaml edits:

# 1. On each controller in the pool: create a dedicated service user and a fresh keypair.
sudo -u cmon s9s user --create --generate-key --group=admins cmon-telemetry
# This writes the private key to /home/cmon/.s9s/cmon-telemetry.key and registers
# the public half against the controller-side user. Repeat on every pool member —
# per-user public keys do not propagate (see the multi-controller caveat above).

# 2. Copy the private key to a location cmon-proxy can read.
sudo install -m 640 -o s9s_cc -g severalnines \
  /home/cmon/.s9s/cmon-telemetry.key /usr/share/ccmgr/service-users/cmon-telemetry.key

# 3. Generate a Bearer token and write the hint file.
TOKEN=$(openssl rand -hex 32)
sudo install -m 640 -o s9s_cc -g severalnines \
  /dev/stdin /usr/share/ccmgr/service-tokens/cmon-telemetry <<< "$TOKEN"

Then add two blocks to /usr/share/ccmgr/ccmgr.yaml:

service_users:
  - name: cmon-telemetry
    cmon_username: cmon-telemetry
    cmon_keyfile: /usr/share/ccmgr/service-users/cmon-telemetry.key

service_tokens:
  - name: cmon-telemetry
    token: <the $TOKEN from step 3>
    service_user: cmon-telemetry
    endpoints:
      - path: /proxy/controllers/status
        methods: [GET, POST]
      - path: /v2/clusters
        methods: [POST]

sudo systemctl restart cmon-proxy

Verify provisioning landed

sudo journalctl -u cmon-telemetry -n 100 | grep '\[collect\]'
# expect: tick start controllers=N captured=...
#         (per-controller lines, no auth errors)

curl -sf -H "Authorization: Bearer $API_TOKEN" \
     http://localhost:9520/billing/controllers/status | jq '.outcome, (.results[] | {name, status, error})'
# expect: "outcome": "ok", every result status="ok" or "empty"

If controllers/status shows error rows with "AccessDenied: User cmon-telemetry is suspended...", see the multi-controller caveat above and the troubleshooting table below.

Verifying the pipeline end to end

Wait one billing.interval tick (default 60 minutes) after starting cmon-telemetry and provisioning the token, then:

# Health probe — no auth
curl -sf http://localhost:9520/billing/status | jq .

Expected (trimmed):

{
  "collector_running": true,
  "collection_healthy": true,
  "health_status": "ok",
  "total_snapshots": 22,
  "billing_period_months": 1,
  "min_active_hours": 24,
  "retention_months": 12,
  "ed25519_public_key": "962d9fb98a3ed1c24406bcd7a94637d2f15bb3dd802d5ad87c666e6a7077bb5f",
  "ed25519_fingerprint": "sha256:27dccd92548741af"
}

Record the fingerprint (sha256:…) somewhere you can reach later — the party verifying your exported reports will cross-check it against any bundle you send them.

Deeper checks — both require authentication:

# Live per-controller fetch outcome (in-memory state of the last collector tick)
curl -sf -H "Authorization: Bearer $API_TOKEN" \
     http://localhost:9520/billing/controllers/status | jq .

# Paginated raw snapshot read (the event store cmon-telemetry writes each tick)
curl -sf -H "Authorization: Bearer $API_TOKEN" -H "Content-Type: application/json" \
     -X POST http://localhost:9520/billing/snapshots \
     -d '{"limit": 5}' | jq '.total, .snapshots[0]'

Troubleshooting if data is not flowing

Symptom	Check
`total_snapshots: 0` after more than one tick	`curl /billing/controllers/status` — `results[]` shows the per-controller outcome with the verbatim controller error string. Audit the token allow-list and service-user provisioning.
`outcome: "discovery_error"` on `/billing/controllers/status`	`cmon-telemetry` cannot reach `cmon-proxy`. Check `billing.cmon_proxy.url`, the token in `token_file`, and the service token's `endpoints:` allow-list in `ccmgr.yaml`.
Some controllers report `status: "empty"` while others produce snapshots	Per-controller. `empty` means the fetch succeeded but the controller owned no clusters this tick — common in multi-controller pools where ownership migrates. Not an error.
Some controllers report `status: "error"` with "User cmon-telemetry is suspended..."	Pool-member key propagation gap (see the multi-controller caveat). Re-run `ccmgradm serviceauth-bootstrap`, then `s9s user --unsuspend cmon-telemetry` on each controller that locked out.
`collection_healthy: false`	`journalctl -u cmon-telemetry -f` — collection errors, disk full, corrupted database.
`401 Unauthorized` from `/billing/reports`	Bearer token mismatch with `api_token` in `/etc/cmon-telemetry/config.yaml`. `/billing/status` stays open (health probe); other endpoints require the token.
`ed25519_fingerprint` missing from `/billing/status`	You are running a pre-2.5.0 binary. Reinstall `clustercontrol-telemetry`.

The Billing page

Open the ClusterControl GUI and click Billing in the left navigation (visible only when billing_enabled: true on cmon-proxy).

The page has four logical blocks, top to bottom:

Current period card. Live high-watermarks for the in-progress billing period (billable nodes, total vCPUs, total RAM, total disk). Totals update as new snapshots arrive. The card header holds the Show public key and Generate report buttons.
Sealed reports heading and filter bar. The Clusters and Tags selects narrow the reports table. Matching is strict — rows without filter descriptors are hidden when a selection is active.
Sealed reports table. One row per sealed report, sorted newest-sealed first:
- Period — the billing period the report covers (for example, Apr 2026).
- Billable — nodes that met min_active_hours during the period.
- Generated — when the report was sealed, in your browser timezone.
- Filter — estate-wide or descriptor chips (for example, tags: customer-acme or cluster_ids: 3,2).
- Seal — a check mark means the report carries a valid cryptographic seal.
- Actions — a menu with View details and Delete.
Detail drawer. Slides in from the right when a row is clicked. The seal status (Sealed or Seal invalid) reflects the hash_valid / signature_valid flags returned by the service. For a deeper check, export the CSV bundle and run cmon-report-verify (see below).

The Show public key button

Click the key-icon Show public key button to open a modal with:

Fingerprint (for example, sha256:27dccd92548741af) with a copy-to-clipboard button. This is the short string you share with whoever verifies your reports.
Public key (hex) — the full 64-character hex, also copyable. Useful for automated verification workflows.

Both values come straight from /billing/status. The private key lives only in ed25519_key_file on the cmon-telemetry host and never surfaces in the UI, API, or logs.

Generating a report

Click Generate report on the current-period card.

Pick the period. Monthly billing periods are the default; pick the first and last day of the month. Shorter periods work, but billable counts will be 0 if nodes do not accumulate min_active_hours.
Scope (optional). Leave Clusters and Tags empty for an estate-wide report. Pick specific clusters or tags to scope to a per-customer or per-tenant report. Tags are free-form (for example, customer-acme).
Force regenerate. Off by default — the service returns 409 Conflict when an estate-wide report already exists for the period, with the existing report_id echoed in the body. Flip it on to overwrite (a new report version is sealed; the previous version stays in the table).
Generate. On success, the page switches back to the table with the fresh row prepended.

Viewing a report

Click any row in the Sealed reports table, or choose View details from the row's actions menu. Things worth knowing:

Timestamps render as YYYY-MM-DD HH:MM:SS TZ in the operator's local timezone.
Total vCPUs / RAM / disk are period high-watermarks (peak concurrent usage across billable nodes), not simple sums.
Seal tag. Green Sealed means the stored hash_valid and signature_valid flags are both true. Red Seal invalid means either the report's JSON body was altered after sealing or the current verification_keys list does not cover the report's signing_key_id. For a deeper, third-party-grade check, use the CSV export with cmon-report-verify.

Downloading JSON / CSV

Download JSON — the raw report structure (summary + by-type-and-vendor + node details + seal metadata). Useful for downstream billing pipelines that read JSON directly.
Download CSV — a ZIP with four files. This is the format to send for third-party verification.

The CSV ZIP contents:

File	Purpose	Signed?
`report.json`	The canonical JSON bytes the service signed, byte-for-byte verbatim. Do not re-format.	Yes — this is what the Ed25519 signature covers.
`SIGNATURE.json`	Manifest: `report_sha256`, `ed25519_signature`, `ed25519_public_key`, `ed25519_fingerprint`, `hmac_signature`, `hmac_key_id`, `signed_at`, `report_id`, `report_version`, `period_start`, `period_end`.	Contains the signature, not signed itself.
`summary.csv`	Human-readable per-cluster-type + vendor aggregate. Useful for invoices.	Derived view — not cryptographically covered.
`node_details.csv`	Human-readable per-node breakdown.	Derived view — not cryptographically covered.

Third parties verify using report.json + SIGNATURE.json; the CSVs are for humans to read.

Deleting a report

From the row's actions menu, choose Delete. Sealed reports are immutable; deleting removes the record permanently. The delete operation is idempotent — calling it for a report ID that is already gone succeeds silently, so retries are safe. Because seals are tamper-evident, delete is the only way to "undo" a report; there is no edit.

Scoping reports to a customer or tenant

Set a tag on each cluster that belongs to the customer (in the ClusterControl GUI: the cluster's Settings → Tags). A customer code is the recommended convention — for example, customer-acme.

Then scope a report by tag:

# Via the UI: Clusters or Tags picker in the Generate form.
# Via curl:
curl -sf -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "operation":    "generateReport",
    "period_start": "2026-04-01T00:00:00Z",
    "period_end":   "2026-04-30T23:59:59Z",
    "tags":         ["customer-acme"]
  }' http://localhost:9520/billing/reports | jq '.report.summary'

The response's summary block is scoped to the customer's nodes only. The sealed report is stored as a separate row (a separate report_id) so it is independently verifiable. Filtered requests bypass the duplicate-gate that estate-wide reports use — multiple per-customer slices of the same period are expected to coexist.

REST API reference

All routes live under /billing/*. /billing/status is unauthenticated (health probe); everything else requires Authorization: Bearer $API_TOKEN.

BASE=http://localhost:9520
AUTH="Authorization: Bearer $API_TOKEN"
CT="Content-Type: application/json"

# Health (no auth)
curl -sf $BASE/billing/status | jq .

# --- /billing/reports (operation discriminator) -------------------------
curl -sf -H "$AUTH" -H "$CT" -d '{"operation":"listReports"}' $BASE/billing/reports | jq '.reports[:3]'

# generateReport — creation-only. 409 with existing_report_id on duplicate.
curl -sf -H "$AUTH" -H "$CT" -d '{
  "operation":    "generateReport",
  "period_start": "2026-04-01T00:00:00Z",
  "period_end":   "2026-04-30T23:59:59Z"
}' $BASE/billing/reports | jq '.report_id, .report.summary'

# Overwrite (re-seal an existing period)
curl -sf -H "$AUTH" -H "$CT" -d '{
  "operation":         "generateReport",
  "period_start":      "2026-04-01T00:00:00Z",
  "period_end":        "2026-04-30T23:59:59Z",
  "force_regenerate":  true
}' $BASE/billing/reports

# Per-customer slice (filter bypasses the duplicate-gate)
curl -sf -H "$AUTH" -H "$CT" -d '{
  "operation":    "generateReport",
  "period_start": "2026-04-01T00:00:00Z",
  "period_end":   "2026-04-30T23:59:59Z",
  "tags":         ["customer-acme"]
}' $BASE/billing/reports | jq '.report_id, .report.filter'

# Use getReport to fetch a specific row by id (preserves filter descriptor)
curl -sf -H "$AUTH" -H "$CT" -d '{"operation":"getReport","report_id":82}' \
  $BASE/billing/reports | jq '.report.summary'

# verifyReport — re-runs HMAC + Ed25519 against the local keys.
curl -sf -H "$AUTH" -H "$CT" -d '{"operation":"verifyReport","report_id":82}' \
  $BASE/billing/reports | jq '{hash_valid, signature_valid, verified_at}'

# exportReport (JSON)
curl -sf -H "$AUTH" -H "$CT" -d '{"operation":"exportReport","report_id":82,"format":"json"}' \
  $BASE/billing/reports -o report-82.json

# exportReport (CSV zip — the third-party verifiable bundle)
curl -sf -H "$AUTH" -H "$CT" -d '{"operation":"exportReport","report_id":82,"format":"csv"}' \
  $BASE/billing/reports -o report-82.zip
unzip -l report-82.zip
# report.json + SIGNATURE.json + summary.csv + node_details.csv

# deleteReport (idempotent)
curl -sf -H "$AUTH" -H "$CT" -d '{"operation":"deleteReport","report_id":82}' \
  $BASE/billing/reports | jq '{report_id, deleted}'

# --- /billing/snapshots — paginated raw event-store read ---------------
# "What's in storage, per controller, since when?" — the data view.
curl -sf -H "$AUTH" -H "$CT" -d '{"limit":5}' $BASE/billing/snapshots \
  | jq '.total, .snapshots[0]'

# Filter by controller, last 7 days
curl -sf -H "$AUTH" -H "$CT" -d '{
  "controller_id": "<controller-xid>",
  "since": "2026-05-11T00:00:00Z",
  "limit": 200
}' $BASE/billing/snapshots | jq '.total'

# --- /billing/controllers/status — live last-tick outcome ---------------
# "Is each controller being collected from right now, and why not if not?"
curl -sf -H "$AUTH" $BASE/billing/controllers/status | jq '
  .outcome,
  (.results[] | {name, controller_id, role, status, error})
'

Report seals and verification

Every sealed report carries two signatures, each serving a distinct trust model.

HMAC-SHA256 — internal integrity

Protects cmon-telemetry's own database against tampering by an attacker who reads rows but does not hold the signing key. HMAC is symmetric — the same key signs and verifies — so it is never shared with third parties.

A SHA-256 hash is computed over the canonical JSON of the report body.
An HMAC-SHA256 signature is computed over the hash, using the signing_key active at seal time.
signing_key_id on the report row identifies which key signed it.
verifyReport rebuilds the hash and re-runs the HMAC using verification_keys[signing_key_id]. A red Seal invalid tag in the UI means either:
- the report's JSON body was altered after sealing (hash_valid: false), or
- the current verification_keys map does not contain the key that signed it (signature_valid: false) — a key-rotation operational issue, not actual tampering.

During HMAC key rotation:

# /etc/cmon-telemetry/config.yaml
signing_key: "<new key>"       # newly sealed reports use this
key_id:      "key-2026-Q3"
verification_keys:
  key-2026-Q3: "<new key>"
  key-2026-Q2: "<previous key>"  # keep old keys while reports signed with them still exist
  key-2026-Q1: "<older key>"

Old reports remain HMAC-verifiable as long as their signing_key_id still appears in verification_keys.

Ed25519 — asymmetric, third-party verifiable

Ed25519 lets anyone with the public key verify a report — without ever having access to the signing key. This is what MSPs use to prove to a billing counterparty that the numbers they are sending have not been modified after generation.

On first start, cmon-telemetry writes a fresh Ed25519 keypair to ed25519_key_file (default /var/lib/cmon-telemetry/signing.ed25519, permissions 0600). The file is reused forever after; do not delete it.
The public key (64 hex characters) and its fingerprint (sha256: + the first 16 hex characters of SHA256(pubkey)) appear in GET /billing/status and the Billing UI's Show public key modal.
At seal time, an Ed25519 signature is produced over the canonical JSON bytes and stored with the report.
Export ZIPs include report.json + SIGNATURE.json, which together are sufficient for offline verification.

Verifying an exported bundle with cmon-report-verify

The recipient of a billing ZIP (for example, a billing counterparty or auditor) can confirm it has not been tampered with between generation and arrival.

Step 1 — Record the sender's public key once. Ask the operator for their Ed25519 fingerprint (for example, sha256:27dccd92548741af) and full public key hex. They can copy both from the Billing UI's Show public key modal or from /billing/status. Save the public-key hex to a file, for example ~/pubkeys/acme-msp.pubkey.

This is the trust anchor

Everything else flows from this first handshake. If the operator rotates their key (rare — the keypair file is persistent across restarts), ask them to re-send the new fingerprint and update your records.

Step 2 — Install the verifier. The cmon-report-verify binary ships in the same clustercontrol-telemetry package, at /usr/bin/cmon-report-verify.

Step 3 — Sanity-check the bundle.

unzip -l metering-report-105.zip
# report.json · SIGNATURE.json · summary.csv · node_details.csv

If those four files are not all present, the bundle is not a verifiable export — ask the sender to re-generate via the Billing UI's Download CSV button (not JSON — JSON-format exports have no signature manifest).

Step 4 — Verify against the trusted public key.

cmon-report-verify verify metering-report-105.zip \
  --pubkey-file ~/pubkeys/acme-msp.pubkey

Expected output when the bundle is trustworthy (exit code 0):

[PASS] sha256: 5071c7bc7352deb8ec15cb42cc9c5be148fc32dd24b2cd4a9750855932a68c4b matches manifest
[PASS] ed25519 signature valid
[PASS] fingerprint: sha256:27dccd92548741af matches manifest
[PASS] public key matches expected

Report ID:            105
Report Version:       3
Period:               2026-04-01 → 2026-04-20
Generated at:         2026-04-20T10:57:09Z
Total billable nodes: 23
Fingerprint:          sha256:27dccd92548741af

Step 5 — Interpret a failed check. If any check fails, the CLI exits non-zero (exit code 1) and prints which stage failed:

Failure	What it means	What to do
`[FAIL] sha256: computed … manifest claims …`	`report.json` was modified after export.	Reject the bundle. Ask the sender to re-download from their Billing UI and forward the untouched ZIP.
`[FAIL] ed25519 signature invalid`	The signature was forged or `report.json` was modified.	Same as above: reject, ask for a fresh export.
`[FAIL] fingerprint … does not match manifest`	Internal inconsistency within the bundle (rare — points to a broken or malicious repack).	Reject the bundle.
`[FAIL] public key mismatch: bundle has X, expected Y`	The bundle was signed by a different installation than the one on file.	Could be a key rotation, could be a spoofed bundle. Confirm the fingerprint with the sender out-of-band before accepting anything.

The exit code is always 2 on I/O or usage errors (missing SIGNATURE.json, unreadable file, and so on).

Verifier command reference:

# Self-consistency check only (bundle matches its own manifest)
cmon-report-verify verify metering-report-105.zip

# Trust-anchored verification (recommended — anchors to a known fingerprint)
cmon-report-verify verify metering-report-105.zip \
  --expected-pubkey 962d9fb98a3ed1c24406bcd7a94637d2f15bb3dd802d5ad87c666e6a7077bb5f

# Same, with the pubkey in a file
cmon-report-verify verify metering-report-105.zip \
  --pubkey-file ~/pubkeys/acme-msp.pubkey

# Inspection only — no cryptographic check
cmon-report-verify show metering-report-105.zip

What verification proves — and what it does not

Does prove	Does not prove
The `report.json` bytes have not been altered between generation and receipt.	That the numbers in the report are factually correct — the service signs whatever snapshots it has in its database. Input-tampering inside the source environment is a separate trust issue.
The signature was produced by the holder of the private key matching the bundle's public key.	That the key holder is who you think they are — that is what `--expected-pubkey` enforces by anchoring to the fingerprint recorded during onboarding.
The bundle was sealed at the timestamp in `SIGNATURE.json`.	That the timestamp is "recent" — cross-check `period_start`/`period_end` against the billing cycle.

Retention and data lifecycle

retention_months (default 12) — a daily job inside cmon-telemetry deletes node snapshots older than the retention window. Sealed report rows are not deleted automatically.
/billing/status surfaces last_retention_cleanup + last_cleanup_deleted_rows so you can confirm cleanup is running.
Dedup guarantee: (captured_at, node_id) is unique in the snapshot store, enforced as a database constraint. Wall-clock-aligned tick boundaries keep restart-storms idempotent — a process that crashes mid-tick and restarts in the same interval bucket produces snapshots with the same captured_at, which the unique constraint silently drops.
Rough sizing: 22 eligible nodes × 1 snapshot per 60-minute tick × 12 months ≈ 230k rows, well under 50 MB of SQLite storage in practice. Tighter intervals scale linearly.

Frequently asked questions

Q: My first report shows total_billable_nodes: 0 even though I see snapshots in /billing/status. A: Nodes must accumulate min_active_hours (default 24 hours) of active-or-stopped state to count as billable. Wait a day after the first collection tick, or lower min_active_hours temporarily for a test. The active-hours formula is gap-based — it reads timestamps directly, so changing the collection interval mid-run does not retroactively inflate or deflate counts.

Q: Some hosts have vcpu: 0 in the first report. A: vCPU is resolved in two stages: the host's stat collector first, then a fallback via cmon-proxy for hosts that have not been sampled yet. Hosts that are new and have not had a single stat tick will stay at 0 until then.

Q: I scaled out — added a second cmon-proxy. Will I double-count? A: Not if cmon-telemetry is configured with a single billing.cmon_proxy.url. There is one poller; controller discovery goes through that one cmon-proxy, and the (captured_at, node_id) constraint dedups regardless. If you front multiple cmon-proxy instances with a load balancer, each tick lands on whichever proxy the balancer picked — still one logical poller. Multiple cmon-telemetry processes against the same database is not supported.

Q: I selected a cluster in the filter bar and the table is empty. A: Filter matching is strict — only rows with a matching filter descriptor pass. Estate-wide reports are excluded when a selection is active. Clear the filter bar to see them again.

Q: The Billing tab is missing from the navigation. A: billing_enabled: false on the cmon-proxy serving the UI. The /billing menu entry and route are both gated on that flag; a direct link to /billing when disabled redirects to the overview page. Flip billing_enabled: true in ccmgr.yaml and restart cmon-proxy.

Q: I tried to generate a report for the same period twice and got 409 Conflict. A: Expected — report generation is creation-only. The response body includes the existing report_id so the UI can offer to open it. To re-seal, pass force_regenerate: true.

Q: /billing/controllers/status shows status: "empty" for some controllers but "ok" on others. A: empty is not an error — it means the fetch succeeded and the controller owned zero clusters in this tick. This is common in multi-controller pools where cluster ownership migrates between members.

Q: /billing/controllers/status shows "User cmon-telemetry is suspended for previous authentication failures" on a pool member. A: Pool-member key propagation gap (see the multi-controller caveat under provisioning). The controller replicates the user record across pool members, but per-user public keys do not propagate, so a fresh bootstrap leaves the other members with stale keys, authentication fails repeatedly, and the brute-force lockout suspends the user. Re-run ccmgradm serviceauth-bootstrap and s9s user --unsuspend cmon-telemetry on each suspended member.

Q: A report's seal is shown as invalid in the UI. A: Verify the current verification_keys map covers the report's signing_key_id. If it does and the seal still fails, the stored JSON was modified outside of cmon-telemetry — consider re-generating the report and investigating the database for tampering.

Q: How do I rotate the HMAC signing key without invalidating old reports? A: Add the new key to signing_key and verification_keys, update key_id, but keep the old key in verification_keys until the last report signed with it ages out.

Q: Should I share my signing_key with a billing counterparty? A: No — never. The HMAC signing key is symmetric: anyone holding it can sign fake reports. Share only the Ed25519 fingerprint / public key. The whole point of Ed25519 is that you prove authenticity without disclosing the key.

Q: What happens if I delete ed25519_key_file? A: On the next start, cmon-telemetry generates a fresh keypair with a new fingerprint. Existing reports remain signed by the old key, so verifiers anchored to the old fingerprint will reject new exports until you also update them with the new fingerprint. Do not delete the file in production without a plan.

Q: My exported ZIP does not have SIGNATURE.json or report.json. A: You are on a pre-2.5.0 cmon-telemetry, or you downloaded the JSON export instead of the CSV ZIP. Only the CSV ZIP carries the third-party-verifiable bundle.