Dashboard, alerts and metrics

Dashboard

The dashboard is a web interface served by the cloud itself. It is off by default and listens only on 127.0.0.1:8780.

dashboard on
dashboard password <new password>
dashboard

The password needs at least eight characters. All changes apply from the next start of the cloud; dashboard shows the current state and, if something holds the dashboard back, the reason as well.

There are two ways to log in. If the player name field is left empty, the shared password applies, and whoever knows it may do everything the dashboard offers. With a player name, that user's own password from user password <name> <password> applies, and they may only do what their roles in access.toml allow. A forgotten password can only be reset in the console.

Section	Content
Overview	groups with start, restart, stop, edit and, for static groups, backup; creating groups; servers with log, drain, restart and stop
Players	who is online, on which server and since when
Users	accounts with roles, roles with rights, existing tokens
Ranks	player ranks and who holds which
Backups	the last twenty backups with restore
Network	broadcast, maintenance and exceptions
Console	enter commands, with exceptions

The page refreshes every three seconds. The dashboard console deliberately lacks the commands that change access itself or stop the cloud: stop, groups <name> delete, service <name> screen, dashboard, alerts, token, update, data, audit, addon and publish. Every action in the dashboard is recorded in audit.log with source dashboard.

Reachable from outside

The cloud accepts an address outside the machine only under certain conditions; otherwise it falls back to 127.0.0.1 and states the reason.

Address	Requirement
`127.0.0.1`	none
private network or Tailscale	a password is set
public address	password, domain and port 443 for Let's Encrypt
behind your own reverse proxy	password and `allow_insecure = true`

dashboard bind 0.0.0.0:443
dashboard domain cloud.example.com
dashboard contact [email protected]

The domain's A record must point to the machine. The cloud obtains the certificate itself and stores it under cache/acme; on port 80 it then redirects to https. dashboard staging on uses the Let's Encrypt staging environment for testing, whose certificates no browser accepts. An ordinary user may not open ports below 1024; the cloud prints the matching setcap command for that.

Further fields exist only in the file:

mutecloud.toml

[dashboard]
session_hours = 12
trusted_proxies = ["127.0.0.1"]
allow_insecure = false

session_hours sets how long a login stays valid. trusted_proxies lists the reverse proxies whose X-Forwarded-For the cloud trusts, so that the lockout after failed attempts hits the correct address.

After five failed logins from one address within a minute, that address is locked for a minute; after twenty across all addresses combined, login is locked for everyone. In both cases a login-blocked alert is sent.

Alerts

The cloud reports events to a Discord webhook or to any other URL that accepts JSON.

alerts webhook https://discord.com/api/webhooks/...
alerts test
alerts events service-crashed service-hanging node-lost backup-failed
alerts events alle
alerts quiet 10
alerts off

alerts webhook enables alerts at the same time, alerts test sends a test message and shows the other side's response. Without alerts events all events are sent.

Event	Level	When
`service-crashed`	error	a server ends unexpectedly, with the path to the crash report
`service-start-failed`	error	a server cannot be started
`service-hanging`	error	the watchdog considers a server frozen
`group-paused`	error	three crashes in a row, the group pauses for a minute
`backup-failed`	error	a scheduled backup failed
`node-lost`	error	a worker lost its connection
`node-back`	info	a node has connected
`memory-tight`	warning	no node has enough memory for the next server
`login-blocked`	warning	too many wrong logins or tokens
`rollout-waiting`	warning	a changed template is waiting for approval
`update-available`	info	a new version is available
`cloud-started`, `cloud-stopped`	info	the cloud starts or stops

The same event for the same server, group or node is sent only once within quiet minutes; alerts quiet 0 turns this off. URLs other than Discord receive a flat JSON with event, level, node, subject, detail and text.

Metrics

GET /metrics on the API address returns Prometheus text. The token needs the right groups.view; a dedicated one for Prometheus is quickly created:

token create prometheus groups.view

prometheus.yml

scrape_configs:
  - job_name: mutecloud
    authorization:
      credentials: <token>
    static_configs:
      - targets: ["127.0.0.1:8770"]

Metric	Labels	Content
`mutecloud_up`		always 1
`mutecloud_build_info`	`version`, `node`	cloud version
`mutecloud_memory_mb`	`node`, `art`	`budget` and `belegt` (used)
`mutecloud_nodes`	`node`, `rolle`	1 for every connected node
`mutecloud_players`		players in the network
`mutecloud_group_services`	`gruppe`	running servers
`mutecloud_group_scale`	`gruppe`, `grenze`	`min` and `max`
`mutecloud_group_players`	`gruppe`	players in the group
`mutecloud_group_maintenance`	`gruppe`	1 in maintenance
`mutecloud_service_players`	`server`, `gruppe`, `node`	players on the server
`mutecloud_service_memory_mb`	as above	allocated memory
`mutecloud_service_rss_mb`	as above	memory actually used
`mutecloud_service_cpu_percent`	as above	CPU load
`mutecloud_service_uptime_seconds`	as above	uptime
`mutecloud_service_ready`	as above	1 when it accepts players
`mutecloud_service_states`	`zustand`	number of servers per state
`mutecloud_rollout_pending`	`gruppe`, `freigabe`	servers on an old template

Log files

logs shows which log files exist and how long they are kept.

File	Content
`logs/mutecloud.<date>.log`	everything, new each day
`logs/fehler.<date>.log`	warnings and errors only
`audit.log`	who ran what from outside
`update.log`	every installed version
`dumps/<server>-<time>/`	crash reports