| name | securing-a-low-footprint-mmo-server |
| description | Hard-won, project-specific doctrine for keeping a small, fast, long-lived networked game server safe against a hostile internet — remote crash/DoS, protocol abuse, player-on-player flooding, credential brute-force, and information leaks — WITHOUT a big security team and WITHOUT bloating the binary or the running cost. The premise is the opposite of "add a WAF in front": security is written INTO the engine, in-tree, in the hot path, and is kept correct by the test suite as the code evolves. Distilled from CatchChallenger, an MMO engine that must stay safe while running on hardware as weak as a 66 MHz i486 and over links as bad as I2P. Use when adding a remote packet handler, touching the protocol parser, validating client input, designing a rate limit, handling passwords/tokens, or judging whether a finding is a real remote vulnerability.
NOTE: the embedded-silicon reverse-engineering & bring-up playbook that used to live here has MOVED to reverse-engineering/SKILL.md. This file is now purely about the SECURITY of the project. See also the sibling playbooks: test/SKILL.md (how we prove robustness) and benchmark/SKILL.md (the performance design that IS our DoS defence).
|
Securing a low-footprint, long-lived MMO server
Hard-won doctrine, not a generic security checklist. The threat model is
concrete: the only attacker that matters is bytes on a TCP socket from
a connected or connecting game client. Everything below exists to make
those bytes unable to crash, hang, corrupt, drain, or out-trick the
server — on weak hardware, over bad links, for years, with a tiny team.
The invariants the whole design serves:
- No remote packet can crash or hang the server. Not a malformed
one, not a hostile one, not a 100×-rate flood. A crash is the worst
bug class here — one packet kills the process for every player on it.
- Security lives in-tree and in the hot path, versioned with the
code, exercised by the test suite — never a bolted-on external box
that drifts out of sync with the protocol.
- Defence costs near-zero at runtime. The cheapest DoS defence is a
server that stays cheap under load; performance is a security
property here (see benchmark/SKILL.md).
- Every byte from the client is untrusted and re-derived server-side.
The client is an attacker's puppet; the server is the sole authority on
what is true.
- The hardest surface — the binary/protocol layer — gets the most
scrutiny. That is where memory corruption hides and where a textbook
"missing check" is usually a false positive (or a real RCE).
1. The attack surface is the parser, and only the parser
1.1 Name the one entry point. Remote untrusted input enters as raw
socket bytes and is turned into actions in exactly one place: the
protocol parser (general/base/ProtocolParsing*, dispatched into
server/base/ClientNetworkRead*.cpp). The socket plumbing around it
(EventLoop, io_uring, UnixSocket, SocketUtil) is not the attack
surface — it moves bytes, it does not interpret them. When you triage a
finding, the first question is "is this reachable from the byte stream a
client controls?" If it sits in operator-config handling, in the admin
stdin console, or in code the binary under attack never instantiates,
it is a hardening note, not a remote vulnerability. This distinction is
recorded per-file in security/SECURITY-IA.md — read it before
re-litigating a "bug" someone already triaged as a false positive.
1.2 The parser invariant kills most "missing size check" findings.
ProtocolParsingBase::parseData() reassembles a packet until all
declared bytes have arrived before it dispatches to a handler. So a
per-packet handler is guaranteed a buffer of at least size valid
bytes. A handler that reads a FIXED number of bytes (memcmp 5,
memcpy 64) ≤ its declared packetFixedSize is safe even if it
appears to "ignore size" — verified empirically with short pre-auth
packets under valgrind: 0 errors. A REAL out-of-bounds read needs a
dynamic-size handler that trusts an inner length/count field and
reads past size. Look there, not at the fixed-size handlers. This
single invariant retires the bulk of automated "CWE-125" noise.
1.3 Fixed-size-before-login is itself a defence. ~26 packet types use
the 0xFE marker for variable size; everything reachable before the
client authenticates is fixed-size on purpose — so an unauthenticated
peer "can't send a 4 GB size packet and saturate the bandwidth"
(comment in ProtocolParsingGeneral.cpp). Dynamic packets are
hard-capped in parseDataSize() to CATCHCHALLENGER_BIGBUFFERSIZE-8,
with tighter per-mode ceilings (master 4 KB, mirror 64 KB). Never add
a variable-length pre-auth packet. If you must, cap the inner length
against the remaining buffer before using it, and add a hostile-input
test for it.
1.4 Strict-alignment safety is a correctness AND a portability bug
class. The parser reads multi-byte fields with memcpy/loadLe32()
wrappers, never a direct cast-and-dereference — a misaligned header on a
strict-alignment arch (MIPS, some ARM) would SIGBUS, which is a remote
crash. Same for endianness: store/load little-endian explicitly. A
packet handler that compiles on x86 but faults on a 66 MHz target is
still a shipped remote-DoS.
2. Layered rate limiting, integrated into the read path
DoS resilience is not a front-end appliance; it is a few bytes of state
per client, checked inside the same function that reads the packet.
2.1 The sliding-window counter (server/base/DdosBuffer.hpp). A tiny
fixed-array circular buffer (DdosBuffer<uint8_t,3>) holds the last N
time-windows of a counter; total() is the windowed sum,
incrementLastValue() counts an event, flush() (driven by a timer)
rolls the window. Three live per client, by category:
movePacketKick, chatPacketKick, otherPacketKick (Client.hpp).
Each category has its own ceiling
(serverSettings.ddos.kickLimit{Move,Chat,Other}); exceed it and the
client is disconnected with a logged reason (ClientNetworkReadMessage.cpp).
Separate categories matter: a player legitimately sends many moves but
few clan-creates, so one budget would either be too loose for moves or
too tight for the rest.
2.2 Saturating, branch-light, allocation-free. The counter saturates
at the type max (uint8_t→255) instead of overflowing, never allocates,
and is ~7 bytes of state. That is the whole point: the defence must be
cheaper than the attack, or you have built an amplifier. Do not "improve"
it into something with a heap allocation or a per-event syscall.
2.3 Drop-don't-kick for chat; kick for protocol abuse. Flooding the
chat channels is throttled by dropping messages once a windowed
counter passes ddos.dropGlobalChatMessage{General,LocalClan,Private}
(ClientBroadCast.cpp, ClientLocalBroadcast.cpp) — a chatty player is
silenced, not disconnected, because chat flood is annoyance, not
exploitation. Protocol-rate abuse (too many packets/moves) kicks.
Match the response to the harm: silently absorb annoyance, disconnect
abuse, never crash on either (see test/ doctrine on "safe-absorb").
2.4 Don't rate-limit the server's own replies. There is a deliberately
disabled block in ClientNetworkRead.cpp: "you are into the server, if
server emits too many queries, fix the emit, not the reply." Rate limits
are for untrusted input only; throttling trusted internal traffic just
hides a real bug.
3. Validate every action server-side — except movement speed
The server re-derives the truth; the client is never believed. For each
client action ask the four questions and answer them from server state:
correct position? on the correct map? allowed to do this action? owns
the item/monster?
3.1 Movement. Direction is range-checked; each single step is
validated against the collision map (flat_simplified_map, value <200 =
walkable, 250–253 = one-way ledges checked per direction, 254 =
blocked); map borders are enforced; "move vs look" coherence with the
previous action is enforced; teleporter use re-checks the gating
condition (clan ownership, quest done, item held, bot beaten). See
MoveOnTheMap.{hpp,cpp}, MapBasicMove.cpp, LocalClientHandlerMove.cpp.
3.2 The deliberate non-check: movement SPEED / step count. The move
packet's stepCount (uint8) has no upper rate cap and grouped moves
are accepted. This is intentional, not an oversight: over I2P or a bad
link the server legitimately receives many moves bunched into one batch,
so rejecting "too fast" movement would punish honest high-latency
players. The positions are all still validated step-by-step — a client
can move many tiles in one packet, but every tile must be individually
legal. If you are tempted to add a speed check, don't — re-read this
paragraph; the security comes from per-tile legality, not from timing.
3.3 Actions and ownership. Inventory/monster/trade handlers all
bounds-check the position into the player's own vectors and verify
possession before acting: monster position < monsters.size(), item
present in inventory and quantity sufficient, can't trade the last
monster or trade while in fight, fight actions check the monster is
alive and on the active team (LocalClientHandler*Trade/Object/FightManage.cpp,
CommonFightEngine.cpp). The pattern is always: validate index → verify
ownership/state → act. Never act on a client-supplied index without both.
3.4 Invalid input: absorb safely or disconnect — never corrupt, never
crash. A handler may safely drop nonsense without kicking; the
kick-contract is "no crash, no state corruption," not "punish every bad
byte." Reserve disconnect for malformed framing / protocol-rate abuse.
4. Coherency at load time, where speed does not matter
There are two regimes. The hot path (per-tick movement, visibility)
is cheap and trusts already-validated in-RAM state. The load path
(character select, datapack, map load) is rare, so it pays for full
coherency checking — this is where you spend cycles freely.
- Every index from the DB is bounds-checked on the way in. A map id
from the database is looked up through
DictionaryServer::dictionary_map_database_to_internal, checked for
the not-found sentinel, and validated against
flat_map_list.size() before use; rescue x/y are clamped to the
loaded map's real dimensions; orientation is range-checked with a safe
default (ClientHeavyLoadSelectChar.cpp). The principle: a stored
value is as untrusted as a wire value — DB corruption or an older
schema must not produce an out-of-bounds access at runtime.
- The map index is never allowed out of bounds. This is the
load-time invariant that makes the hot path able to skip the check.
5. HARDENED mode: prove the invariants, then ship lean
CATCHCHALLENGER_HARDENED (CMake option, default OFF in production) turns
the engine's implicit invariants into explicit abort() checks:
null-param guards, packet-type tags, vector-size ceilings (e.g. buffs
≤255), post-move bounds asserts, and — in DdosBuffer — clock-drift and
"forgot-to-flush >300 s" detection that returns 0 instead of a stale
count. The doctrine around it:
- Hardened is ALWAYS ON in the test suite (test/SKILL.md). Any abort
under test is a real invariant violation that must be fixed, not
silenced — it means the production build, which would not abort, was
about to do something undefined.
- Production ships hardened OFF for speed, relying on the test suite
having already driven every abort to never-fires. So the contract is:
the same malformed packet must (a)
abort() loudly under hardened/test
and (b) be gracefully disconnected, never crash, under production. Both
flavours are tested — see test/testingclustersecurity.py, which fires
a battery of malformed/boundary/hostile packets at every client-facing
port with hardened ON and OFF, respawning any server that dies so the
whole battery runs, then re-drives a real client to prove survival.
- Hardened also enables stronger internal data-coherency and the
protocol "mirror" re-parse — it is the mode that distrusts our own
state, not just the client's.
6. Memory discipline is the core security work
On a C/C++ server, the security bug that matters most is memory
corruption from a remote packet. The test/ doctrine (see test/SKILL.md)
is therefore a security tool, not just a QA tool:
- Run under strict memory control whenever you touch C/C++. Every
testing*.py and all.sh wires in clang sanitizers
(asan/lsan/msan, -fno-sanitize-recover=all, halt_on_error=1)
and gcc+valgrind (memcheck). A handler that passes functionally but
trips ASAN/valgrind on a crafted packet is a security failure.
- Hunt infinite loops and hangs as hard as crashes. A packet that
wedges the single-threaded event loop is a full DoS for every player.
(Real example fixed in-tree:
UnixSocketClientFinal OOB read + infinite
loop.) Fuzz toward non-termination, not just toward segfaults.
- Prefer RAII /
std::vector over raw new[]/malloc. Mismatched
free and leaks are the recurring memory-bug shapes here; the project
has repeatedly converted raw buffers to std::vector<char> (.data()).
- Don't instrument a binary mid-test. Adding
[TRACE] prints to a
binary under a running testing*.py changes timing/stdout/back-pressure
and invalidates the verdict. Kill the test → add traces → rebuild →
re-run from scratch → read → strip → rebuild → confirm. (gdb for naked
reproducible crashes; never gdb with ASAN/valgrind.)
7. Performance is a DoS defence (keep the cost floor low)
The cheapest way to survive a flood is to make each packet cheap, so the
server saturates into a flat plateau instead of a death spiral
(benchmark/SKILL.md). Security-relevant consequences:
- Decode in-band, zero-copy. Packets are parsed straight out of the
recv buffer when no header continuation is pending ("parse directly out
of
buf — zero copy"), into a single static read buffer with no
per-connection allocation. io_uring recv_multishot + provided buffer
rings push this further (kernel→userspace zero-copy); datapack delivery
uses sendfile(2)/splice chains so big transfers cost almost no CPU.
Single-threaded epoll/io_uring means no locks and no cross-connection
aliasing — and therefore the concurrency-bug class (TOCTOU, data
races between socket callbacks) simply does not exist here; do not flag
it.
- Low cost floor = headroom against floods, and it is why we can run
exposed on tiny/cheap hardware that is trivial to sandbox tightly (§9).
8. Credentials, hashing, and not leaking internals
- Two hash families, by purpose.
xxhash (XXH32) for fast,
non-security integrity (datapack file checks, chunking). blake3
(general/base/CatchChallenger_Hash.*, 256-bit) for every final /
security hash — login verification, server-to-server cluster auth,
client→game-server tokens. blake3 is the safe, quantum-compute-proof
choice; xxhash must never guard a security boundary. If you add a
hash, ask "can an attacker benefit from forging this?" — yes ⇒ blake3.
- Passwords: salted, shared-secret, never the raw secret on the wire.
The stored secret is combined with a per-session random nonce (16-byte
token from
/dev/urandom) and blake3'd; the client proves knowledge
via a memcmp of digests, so the DB secret itself never crosses the
network and a captured login is not replayable.
- Long-session premise ⇒ strict login-try limits are affordable.
Players stay connected for hours, so logins are rare per user; the
server caps simultaneous un-authenticated connection slots
(
CATCHCHALLENGER_SERVER_MAXNOTLOGGEDCONNECTION, default 50) and
evicts the oldest pending auth when full. Tight pre-auth budgets cost
honest users nothing and starve brute-force/connection-flood attempts.
- Never leak internal DB ids into the protocol. Runtime uses
internal indices; the DB id is mapped through
DictionaryServer
only at serialization boundaries (Client.cpp save/load). The client
sees internal indices, never raw row ids — so it cannot enumerate or
probe the database layout from the wire. When you add a field to a
packet, check it is not an internal/DB id in disguise.
9. The server is only a server — keep it that way
- No useless surface. The server does one thing; it has no admin web
panel, no scripting engine, no plugin loader bolted into the hot binary.
Every feature is attack surface — the smallest server is the safest.
- Minimal dependencies, on purpose. A dependency is not added for a
1% gain: fewer libs = smaller supply-chain surface and less to audit
and maintain. Where a lib is used (blake3, xxhash, tinyxml2, zstd,
zlib) it is system-lib-first with a vendored fallback — the cross/
fleet sysroot is a copy of the real target, so its
.so is what the
device runs and is preferred; the vendored copy compiles in only when
find_library() fails. Keep vendored copies at the latest upstream and
fix bugs at our call site, not in the vendor fork (a patched fork
breaks the next re-import).
vector features are compile-time-removable. Datapack-send extras
(compression, mirror streaming) are exactly the heavy, less-trodden
paths most likely to hide a bug, so they are switchable off:
CATCHCHALLENGER_SERVER_NO_COMPRESSION removes the zstd path entirely
(and the master binary builds with it off); compression=none disables
it at runtime. Compression is applied only to the rare character/
datapack-load packets, never the hot per-tick packets. A removed
feature is an un-attackable feature — if a deployment doesn't need it,
compile it out.- Minimal needs ⇒ strict sandbox. Because the server wants almost
nothing from the OS (a port, a DB socket, a datapack dir,
/dev/urandom),
it is trivial to lock down hard — drop privileges, restrict the
filesystem, seccomp the syscall set. Good sysadmin hygiene is part of
the security posture, not an afterthought; the lean runtime makes it
cheap. (Note the local-only hardening items still tracked in
SECURITY-IA.md, e.g. the operator-config UnixSocketServer paths —
out of the remote threat model, but worth fixing for defence-in-depth.)
10. How to work on this safely (the loop for a future IA/human)
- Read
security/SECURITY-IA.md first. It is the running audit
notebook of trust boundaries and already-triaged false positives.
Don't re-flag what is recorded there; do add new findings to it.
- Locate the byte path. Is your change reachable from a client's
TCP bytes? If not, it is hardening, not a remote vuln — say so.
- Re-derive, don't trust. Any value from the client (or the DB) is
untrusted: bounds-check the index, verify ownership/state, then act.
- Add the hostile test in the same change. A new packet handler
needs a malformed/boundary/flood case in the relevant
testing*.py;
run it under sanitizer and valgrind, hardened ON. The test count
never decreases (no commented-out run_test lines).
- Only keep changes that move a real [FAIL]→[PASS]. No
"maybe-this-helps" security patches; revert anything that fixes
nothing.
- Match the response to the harm. Crash → never acceptable.
Protocol-rate abuse → kick. Annoyance flood → drop. Bad single value
→ absorb or kick, never corrupt.
- The bounty docs (
security/EXAMPLE-BOUNTY-BUG-SERVER*.md) define
the external scope: client-facing standalone game server, login,
gateway, game-server-alone are in scope; master and inter-node links
are internal-only and out of scope. security/server.py is the scan
tooling; its output is git-ignored.
Cross-references
- test/SKILL.md — how we manufacture every hostile condition and
prove survival (sanitizers, valgrind, fuzzing, cluster security test).
- benchmark/SKILL.md — the low-RAM, zero-copy, plateau-not-spiral
design that makes flood resilience cheap.
- reverse-engineering/SKILL.md — (moved from here) the unrelated
embedded-silicon RE & bring-up playbook.
- security/SECURITY-IA.md — the live, per-file audit ledger.
- security/EXAMPLE-BOUNTY-BUG-SERVER.md* — external scope & rules.
