Troubleshooting

Start with service <name> logs for one server and logs for the cloud. Almost every case below announces itself in one of the two.

No node with enough memory

Für Lobby steht keine Node mit genug Speicher bereit

The node's budget is used up. It is a budget, not a measurement of the machine:

[resources]
memory = 4096
reserve = 1024

Servers may use memory minus reserve. Raise memory to what the machine really has minus what the operating system and your databases need, or give the group less per server. Nothing is started until there is room, which is the intended behaviour; the alternative is a machine in swap.

Group files are broken

Gruppendateien sind fehlerhaft, es gilt weiter der letzte gültige Stand

A TOML file under groups/ does not parse. The cloud keeps running on the last valid state and changes nothing. Fix the file, the next pass picks it up; no restart needed.

Java

Java-Pfad existiert nicht: /opt/jdk/bin/java
Nur 'temurin' wird automatisch geladen, 'zulu' braucht einen expliziten java.path

Runtimes are fetched per group and stored under runtimes/. Only Temurin is downloaded automatically. Any other distribution has to be present on the machine and named explicitly:

[java]
path = "/usr/lib/jvm/java-8-openjdk-amd64/bin/java"

A server crashes

Bench-3 ist abgestürzt (Code 137)
Absturzbericht von Bench-3 gesichert unter dumps/Bench-3-1790120833

Code 137 means the process was killed, usually by the kernel's out-of-memory killer. Give the group more memory or the machine fewer servers. The report under dumps/ holds the last output; housekeeping removes old ones after the retention in [logs].

A server hangs

Lobby-1 reagiert nicht mehr: Seit 95s kein Lebenszeichen vom Hauptthread

The process is alive but its main thread has stopped ticking. The watchdog renews the server. If this happens repeatedly, look at the server's own log just before the stall: a plugin in an endless loop, a world generation spike or a garbage collector in trouble all look like this from the outside.

[watchdog]
hang_seconds = 90
restart = true

A server will not stop

Lobby-1 reagiert nicht auf den Stopp-Befehl, wird jetzt beendet

The server was asked to shut down, did not do so within the grace period and was killed. Worlds are flushed before backups, so this rarely costs data, but a server that regularly ignores the stop command is worth a look.

The worker loses the leader

Verbindung zum Leiter beendet, neuer Versuch in 5 Sekunden

The worker keeps its running servers and reconnects on its own. If it does not come back, check that the leader's API is reachable from the worker and that both use the same token. The cluster link is expected to stay inside a private network.

Nothing answers on the API

GET /api/health needs no token and tells you whether the cloud itself is up. A 401 means the token is missing or wrong, an empty answer means you are talking to the wrong address. The default bind is 127.0.0.1:8770, which is only reachable from the machine itself; that is deliberate.

Servers from a previous run

3 Server aus einem früheren Lauf beendet

The cloud was killed without shutting down and found orphaned processes on start. They are stopped so that ports and worlds are free. Nothing to do.