Waking a Minecraft Server on Join

A hosting platform that hibernates idle servers has one hard problem, and it is not the hibernating.

TrueTick, a Minecraft hosting platform I build and run, stops a server when nobody is playing on it and starts it again when somebody joins. The economics are obvious: most servers are empty most of the time, and a machine nobody is using should not be costing anybody anything. The engineering is less obvious, because a Minecraft client will not wait for you. There is no loading state you control and no retry you can schedule. The player clicks an entry in their server list, and either something answers or the connection screen sits there until the client gives up on its own.

So the whole problem reduces to one question: what do you show a player during the seconds a container takes to boot? Everything below is what falls out of answering that honestly.

The First Packet Decides Everything

A Minecraft session opens with a handshake, and that packet carries more than it looks like it should.

// Handshake holds the parsed fields of a Minecraft handshake packet.
type Handshake struct {
    ProtocolVersion int
    ServerAddress   string
    ServerPort      uint16
    NextState       int // 1 = status, 2 = login
    // AddressExtra is the NUL-delimited tail some non-vanilla clients
    // (Forge/FML, legacy forwarding) append; "" = vanilla-shaped. Limbo
    // requires "".
    AddressExtra string
}

Two fields do all the work. ServerAddress is the hostname the client dialled, which means a proxy can figure out which server this connection is for without asking the server anything — and that is the only reason waking on demand is possible at all. If routing required the backend to be up, there would be nothing to route to.

NextState is the client telling you what it intends to do next. 1 means it wants a status response for the server list. 2 means it wants to log in. That single integer is the difference between a request that must cost nothing and a request worth starting a machine for.

A Ping Is Not a Join

Open the multiplayer screen in Minecraft and the client pings every server on the list, unprompted, and pings them again whenever the list refreshes — whether or not you intend to play. If a status ping woke servers, a single player idling on that screen would be starting machines they never join — and paying for none of them, since nobody is playing.

So the status path never wakes anything:

if hs.NextState == 1 { // status ping: never wakes a sleeping server
    // If the server is running, forward the ping to the live backend so the
    // list shows the real motd/version/online.
    if addr, running := p.d.StatusBackend(t.ID); running && addr != "" {
        // ...dial, replay the handshake, splice, return
    }
    // Sleeping: answer synthetically, and say so in the MOTD.
    _, _, _ = protocol.ReadPacket(conn) // Status Request (empty packet)
    _ = protocol.WriteStatusResponse(conn, desc, hs.ProtocolVersion, t.MaxPlayers)
    if ping, _, err := protocol.ReadPacket(conn); err == nil {
        _, _ = conn.Write(ping) // Pong echoes the Ping packet
    }
    return
}

That block is condensed — the real one carries the fallbacks and the MOTD composition — but the shape is exact. A running server gets its ping forwarded to the real backend so the list shows the true MOTD, version, and player count. A sleeping one gets a synthetic response that says, in the MOTD line the player actually reads, that the server is asleep and will start when they join.

The Pong is worth a note. The status exchange ends with the client sending a Ping packet carrying a timestamp, and the server is expected to send back the identical payload; the client subtracts and calls the difference latency. You do not compute anything. You echo the bytes. Get that wrong and the server list shows a broken connection indicator for a server that is perfectly fine.

The Hostname Lies

Routing on ServerAddress works right up until a client is not vanilla.

Forge appends a NUL-delimited marker to the address field. BungeeCord and Velocity's legacy forwarding append their own payload the same way — the client's real IP and UUID, packed into the field that is supposed to hold a hostname. So the string arriving in ServerAddress may be play.example.com\x00FML\x00, and a naive lookup of that whole string matches nothing.

The routable name is everything before the first \x00. The tail is not garbage, though, and throwing it away would be a mistake: its presence is how you know the client is not vanilla, and that turns out to decide whether the trick in the next section is available at all. So the parser splits and keeps both halves.

Nothing to Hold the Player With, So Give Them a World

Now the actual problem. A container is booting. It will be ready in seconds. The player is looking at a connection screen that gives you no way to say anything to them, and their client is running down a timeout you do not control.

The usual answer is to hold the TCP connection open and hope the boot finishes first. It works often enough to ship, and it fails in the worst possible way: an opaque timeout, with the player concluding the host is broken.

The better answer is that you do not have to make them wait outside. You can let them in.

TrueTick runs a limbo server — a minimal Minecraft server implemented in-process, in Go, that speaks just enough of the protocol to be real. It accepts the login, walks the client through the configuration phase, sends the play-state packets that constitute joining a world, and drops the player into an empty one with a message explaining that their server is starting. They are in the game. The client's connection timeout is no longer running, because the connection succeeded. When the real server reports ready, limbo sends a Transfer packet, and the client reconnects and lands on the running server on its own.

The eligibility check is the honest part:

switch {
case running:
    // nothing to wait for — straight to the splice
case !t.LimboOK:
    // backend is not limbo-capable: not paper-family/vanilla >= 1.20.5
case hs.AddressExtra != "" && !p.d.LimboAllowModded:
    // non-vanilla handshake, and modded clients are not enabled
default:
    if p.d.StartLimbo(conn, hs, t, resCh, ip) {
        return // limbo owned the session
    }
    // The engine declined (disabled / unsupported protocol / per-IP cap);
    // fall through to the hold path.
}

Condensed again — the real switch logs each refusal — but every branch is there, and the branches are the point. Limbo is not something the platform can always do. It needs a backend from the Paper family or vanilla at 1.20.5 or newer, because Transfer does not exist below that. It needs a vanilla-shaped handshake unless modded clients have been explicitly enabled. The engine itself can still decline on the client's protocol version, on a per-IP session cap, or because the feature is switched off.

And there is one case where limbo runs but Transfer cannot: a session arriving through Velocity. A proxy relays a backend's Transfer through to its own client, and in a forwarded session the address limbo would be handing over is the node-local one — which would send the player out from under the proxy, at an address that was never meant to be dialled directly. So instead of transferring, limbo disconnects with a message asking the player to rejoin, which by then lands on a server that is up.

Every one of the three refusals in that switch falls back to holding the connection, and the hold has its own limit. When it expires, the player is released with an explanation rather than left to time out silently. The wake keeps going in the background — it is not cancelled just because the person who triggered it stopped waiting — so when they rejoin, the server is usually up and the join goes straight through.

None of this is glamorous. It is several fallbacks deep, and the deepest one is "tell the truth and let them come back." That is the part I would defend hardest.

The Race You Only See Once

There is a bug in the obvious version of this, and it only appears under load.

A server is asleep. A player joins. The proxy starts the container, waits for readiness, and then splices the two connections together. Meanwhile a separate idle monitor is looking for servers with nobody on them and hibernating them.

Consider the window between the container reaching a running state and the splice completing. The idle monitor sees a running server with zero players and zero connections, because the joining player is not connected yet — they are still being connected. The monitor does exactly what it was built to do and puts the server back to sleep, underneath the player it was woken for.

The fix is placement, not logic:

// login: reserve a connection slot BEFORE waking so the idle monitor cannot
// hibernate the server in the window between it reaching RUNNING and the
// splice. Track(+1) resets the idle timer and holds conns>0 across the wake.
p.d.Track(t.ID, +1)
defer p.d.Track(t.ID, -1)
 
resCh := make(chan EnsureResult, 1) // buffered: late result must not leak the goroutine
go func() {
    addr, err := p.d.Ensure(t.ID)
    resCh <- EnsureResult{addr, err}
}()

The slot is claimed before the wake is even requested, and released by a deferred call whenever the session ends however it ends. The reservation resets the idle countdown and holds the connection count above zero for the whole duration of the wake, so the window the monitor could have acted in never exists.

The buffered channel is the same category of care. If the player gives up before the wake finishes, the result arrives with nobody blocked on it. A buffer of one means the send completes anyway and the goroutine exits; the paths that outlive the player drain the channel as well, but the buffer is what makes the leak impossible rather than merely unlikely.

Wake Is an Attack Surface

Anyone who knows a hostname can cause a machine to start. Authentication happens well after the point where the wake has already been triggered, so the wake is, by construction, an unauthenticated write of the most expensive kind available: it allocates CPU and memory.

The mitigation is a rate limit, and the two design decisions in it are both about not punishing the wrong person.

It applies only to cold wakes. Joining a server that is already running costs nothing worth throttling, and throttling it would break legitimate reconnects and every group of players sharing an address behind one NAT.

It is keyed by the source IP rather than by the target server. Key it by server and an attacker can hammer somebody else's hostname until the rate limit is exhausted, and then the owner cannot wake their own server — you would have built the denial of service into the defence against it. Keyed by source, an abusive address only ever spends its own budget.

I am not going to publish the thresholds.

Going Back to Sleep

Hibernating is the easy direction, with one decision in it that took a false start to get right: who decides the server is empty.

The proxy knows how many connections it is holding. That is the number sitting right there, and it is the wrong one. Connections drop without sessions ending, sessions end without connections dropping, and a player whose client is mid-reconnect exists in one view and not the other. The number that decides whether somebody is playing should come from the thing that knows whether somebody is playing, which is the game.

So the idle monitor asks the server for its player count, and falls back to its own connection count only when the game did not answer. Falling back matters: an unreachable server is not an empty one, and a monitor that treats "I could not tell" as "nobody is there" will eventually hibernate a populated server. Not knowing has to fail toward leaving it alone.

Testing a Protocol You Did Not Design

Everything above is an implementation of a specification I do not own, against clients I cannot patch. If my encoding of a packet is wrong, the failure mode is a player's client silently disconnecting.

The obvious way to test it is to write a test client out of the server's own session helpers, drive a session through them, and assert on what comes back. That test passes. It also cannot fail in the way that matters: when the code that composes a Login Success packet and the code that reads one are the same code, a test built from both agrees with itself perfectly while every real client disagrees.

So the end-to-end test for limbo re-implements the client side from scratch at the packet level. Its own handshake body, its own Login Start, its own decoding of Login Success and Transfer and Disconnect — assembled from the generated packet-id table and the wire primitives, never from the session code's own send and receive helpers. Besides the generated id table, the shared layer stops at the wire primitives — VarInts, strings, and the length-prefixed frame; every packet built on top of those, the test builds itself, and so it disagrees loudly the moment the server's idea of a packet's shape drifts from the protocol's. It also runs over a real loopback TCP listener rather than an in-memory pipe, because framing bugs hide behind a pipe that never splits a write. And it lives in a separate test package, so importing an internal helper is not merely discouraged but impossible.

That is more work than a test has any right to be. It is also the only version that can tell me I am wrong.