Bots for load-testing

A list of MMOs, MOBAs, and FPS games that have been load-testing using bots.

TERA

Koo, 2010, How to support an action-heavy MMORPG, slides

Their system to load-test a shard is called Sisyphus, and runs 1500 clients per machine. The behavior of the bots is based on the behaviors from real players (probably from alpha). A WAN simulator is used between bots and realm servers; the average latency is set to 200ms. A high-performance machine and a "dedicated line" were needed.

EVE

Press, 2011, Orchestrator: A post-mortem on an automated MMO testing framework, slides

EVE started to make a thin client from its game client in 2010. Orchestrator is the tool they use to load-test and integration-test their architecture and code. Orchestrator does not send one script to each client, proxy, and server. Instead, it runs a single master script, and tells them, as they progress in the test, what the next operation is. The test can be stopped right when a client reports a bug, not until everything scheduled has been sent.

OpenSim

Lake, 2010, Distributed scene graph to enable thousands of interacting users in a virtual environment, paper

The limitations we have encountered with avatar scaling during these experiments have been in getting enough hardware to generate the load of over 1000 clients and the limited physics simulation capabilities of a single thread on the scene server.

League of Legends (allocating games to servers)

Delap, 2010, League of Legends: Scaling to millions of ninjas, yordles, and wizards, video + slides

• Load-testing in a realistic setup: more than 50 machines with the same spec as those in production
• EC2 is a good tool, but the network is not reliable, so careful not trying to fix problems that only happen in the test setting.
• With thousands of clients, logs may not be the best way to gather test results.

League of Legends (chat)

McArthur, 2011, Building the chat service for League of Legends, slides

Dozen of EC2 machines, each running 5-9k bots. Each bot is an XMPP chat client (they used the Smack API). Load-testing is useless without proper modeling.

Crysis 2

Hall, 2011, A Programmer's Post-mortem Crysis 2 Multiplayer, slides

They wanted to check the frame rate with lots of moving entities, and detect bugs or gameplay issues. They used automatic testing. "Lots" of bots are run for 10 minutes per level to stress-test the builds. The bots do random actions like walking, jumping, or shooting.

Gears of War 3

Weilbacher, 2012, Dedicated Servers in Gears of War 3 Scaling to Millions of Players, slides

Their bots are clients without renderer and user input. They run automated bot matches to check the performance of their server platform. For Gears 2, they used to run 2.5 games per core in 2009. For Gears 3, they run 7 games per core in 2011.

Guild Wars 2

Patrick Wyatt, a lead programmer on Guild Wars and Guild Wars 2, discourages using bots: bot's behavior differs too much from actual users. Instead, he recommends recording live play data, and replay it on the server to fix bugs or check the load.

FPS, RTS, and fighting playstyles

Summaries of player types from gamerdna.com. Even though there is no scientific basis in those types, they show various types of gameplay and players to keep in mind when designing a game.

Multiplayer FPS playstyles

Support	sniper protecting critical places or healer/supplier.
Commander	strategist, guide.
Recon	strike when least expected. Crafty, fast, sneaky, behind enemy lines.
Shock Trooper	frontal attack

RTS playstyles

Rusher	fast, micromanagement, keyboard shortcuts, predict and counter-attack opponent
Strategist	thinker more than a doer, big picture, game knowledge
Turtle	build a fortress/maze. Max out the tech tree and prepare a big explosion to win.
Commander	playful curiosity over optimality.

Fighting playstyles

Tactician	thinking, mastering characters, out-maneuvering the opponent
Speed Demon	quick, keep out of harm's way
Brute	massive force
Loose Cannon	unpredictable, go with the flow, take advantage of openings

[Literature] Optimizing Online FPS Game Server Discovery through Clustering Servers by Origin Autonomous System

by Grenville Armitage, 2008 located here.

Armitage is Australian, and Australians seem to suffer from latency problems. This might explain why he was looking for an efficient way to solve a latency problem: server discovery in CS:S. According to the author, the (client-side) server discovery process algorithm is as follow:

1. Issue a getservers query to UDP port 27011 on the Steam master server
2. Receive a packet containing a list of 231 active servers <IP address:port>
3. Probe each game server in the previously received list
4. Repeat until the Steam master server has no more game servers to return

The speed of server discovery is actually configured by the user: it is the "What is your connection capacity" which possible parameters are "Modem 56k", "DSL", "DSL+", etc. This parameter changes the speed of game server probation (phase 3 in the algorithm above). There are approximately 30K game servers and with a Steam client configured with "DSL > 256K", 140 servers can be probed per second. Without optimization, it takes 230s to complete the discovery process with a Steam client configured with "DSL > 256K".

In general, players want the game server with the smallest ping or RTT. A first solution could be grouping servers based on their location, but it is not always true that the closer geographically the server is, the smaller the ping will be. The paper proposes clustering game servers by the Autonomous System to which each game server's IP address belongs (its origin AS), ie grouping servers based on the Internet topology. An AS relies on IP routing prefixes to group IP together. The author also added an automatic termination of server discovery when servers RTT are above a specified threshold such as 200ms. Here is the algorithm:

1. Retrieve all game servers
2. Group servers in clusters based on their AS number
3. Calibration process: this eventually splits clusters into smaller ones. In the end, the RTT of a randomly-picked server inside a cluster should be within a 40ms range of the cluster median RTT.
4. Rank clusters in ascending order of their median RTT
5. Probe the servers which have not been probed during the calibration process until a certain RTT threshold (200ms for instance)

Approximately 3k servers (10% of all the servers) are probed during the calibration phase. Around 1k clusters are created during the calibration process. In locations such as Australia, Taiwan or Japan, the algorithm stops within a minute and gives nearly all the available servers which RTT is below the specified threshold. Countries like Germany, the USA or the UK are closer to game servers and their RTT is smaller in average. Therefore, reaching the 200ms RTT threshold takes nearly the same time for players in these countries. However, if the RTT threshold is lowered to 100ms (for instance), the algorithm may stop much faster. If the player could specify the RTT threshold he/she wants the algorithm to stop, the proposed solution could be an efficient improvement to the current discovery process of Valve Counter-Strike game servers.