@SebAaltonen, I’m the only one that I know of that’s made an MMO server with support for 100k+ CCU on a single node. It costs me only $2k a year to host.
If you parallelize things and have O(N/P) worst-case algorithms in place, compute is not really a major bottleneck to worry about. Network egress bandwidth is.
You must design netcode that is guaranteed to never have network congestion regardless of whatever users do. Otherwise server lag and disconnects will affect all users, causing ragequits.
This means TCP is out of the question. The simple act of users disconnecting leads to head-of-line blocking and surges in network egress bandwidth, which can trigger a positive feedback loop of even more users disconnecting if you’re already at max load. Targeting a web browser thus makes it tricky as you really want to build on top of raw UDP but they don’t support that; you must waste bits on QUIC/WebTransport.
You must also identify the maximum amount of dynamic packed data you can send per tick to each client per tick, as well as what your target tick rate must be.
The formula is simple:
Bandwidth = Packet data per player per tick * ticks per seconds * number of concurrent players supported.
To affordably achieve support for a very large numbers of players, especially on a single node, you must optimize the netcode design heavily by having a slow tick rate and limited packet size. RuneScape was designed with 600ms ticks and WoW designed with 400ms ticks for this very reason.
Once you settle on a target bandwidth (I chose 1 Gbps) for your machine that you’re willing to pay for and can prove you can handle, and a tick rate (I chose 600ms/tick) and CCU target (I chose 131,072), then by necessity you end up with a packet size (572 bytes for me) through simple division. This must include all header bytes from each protocol layer (except Ethernet).
Your job from then on is to design the data to packet into each fixed-sized buffer per tick per player and to have a fully playable game within that. The protocol I use is a simple unreliable streaming of the current localized dynamic state of the world around the player. It is purely a snapshot without any temporal deltas in place; no dependencies upon other packets required, therefore packet drops are fine. Bitpack as tightly as you can; only use integers, not floats.
You also need to identify a strategy for interest management to network cull players when too many are in one location. Inevitably this means that some players will be invisible to others.
Lastly, I highly recommend getting a dedicated bare metal server from OVH. They are the only host provider that won’t charge you an arm and a leg for network egress bandwidth. You get guaranteed bandwidth and it’s unmetered, meaning you don’t pay a cent extra even if you’re using full bandwidth capacity all month. I pay $172 a month; the same server at full load hosted on AWS would cost me ~$18k a month.
What do you guys do when you get a sudden urge to build a 100,000 player MMO with fully destructible world? You do the math that it could run on a single 128 core Threadripper server (everybody in the same world) with your crazy netcode ideas and super well optimized C code...