Operations

Setting a cloud up takes an afternoon. Running it for a year is a different job, and this page is about that job.

Autostart

The installer writes mutecloud.service next to the binary. Put it in place once, then the cloud comes back after a reboot and after a crash:

sudo cp ~/mutecloud/mutecloud.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now mutecloud

The unit restarts the cloud ten seconds after an unexpected exit and gives it two minutes to shut down cleanly, which is enough for running servers to save their worlds. A cloud started by hand from a shell survives neither a reboot nor a closed session, and the network stays down until somebody notices.

What runs on its own

Every two seconds the cloud compares the target state from groups/ with reality and corrects the difference. That covers both directions: missing servers are started, and servers that are no longer needed are drained and removed, one at a time, youngest ordinal first, never below scale.min and never one that still has players on it. A server younger than three minutes is left alone so that a brief spike does not cause start-stop churn. Persistent groups are never shrunk automatically, because their world lives in their directory.

A crashed server is noticed immediately, its last output is written to dumps/ and a replacement is up within seconds.

A hung server is noticed as well. The bridge reports every ten seconds how long ago the server's main thread last ticked. If that stall, or the silence of the bridge itself, passes the limit, the cloud raises service-hanging and renews the server. Without this a frozen server keeps its port open and the proxy keeps sending players into it.

[watchdog]
enabled = true
hang_seconds = 90
restart = true

Set restart = false to be told without being acted upon.

Memory is a budget, not a wish

A node starts servers only while its budget allows it:

[resources]
memory = 4096
reserve = 1024

With these defaults 3072 MB are available for servers. A group asking for 64 MB per server therefore gets 48 of them, and the next start is refused with a warning instead of pushing the machine into swap. Raise memory to what the machine really has, minus what the operating system and your databases need, and leave reserve alone.

Backups

groups <name> backup            write one now
groups <name> backup list       what is there
groups <name> backup restore    put one back

Backups land under backups/. Test the restore path once while nothing is on fire, and copy the directory off the machine; a backup on the same disk is a convenience, not a safety net.

Updating

update            look for a newer version
update install    fetch it and restart the cloud
update rollback   go back to the previous binary

The servers keep running while the cloud restarts; they are separate processes and reconnect to the bridge afterwards. In a cluster, update the leader first and the workers right after, and do not leave them on different versions for longer than the update takes.

Watching it

GET /api/health answers without a token and is the right target for an uptime check. GET /metrics serves Prometheus text with services, players, memory and node state. Alerts are delivered through the channels in [alerts]; service-crashed, service-hanging, node-lost and backup-failed are the four worth waking up for.

Logs rotate under logs/, crash reports collect under dumps/, and housekeeping removes both after the retention in [logs]. Plan disk space for templates plus backups plus worlds; the cloud itself needs about 30 MB of RAM and no measurable CPU while it idles.

Numbers

Measured with MuteCloud 0.3.7 on a laptop (Apple M4, 24 GB) against a group of trivial custom servers, so the figures show what the cloud costs, not what a Minecraft server costs: