28 August 2012

Bots for load-testing

A list of MMOs, MOBAs, and FPS games that have been load-testing using bots.

TERA

Koo, 2010, How to support an action-heavy MMORPG, slides

Their system to load-test a shard is called Sisyphus, and runs 1500 clients per machine. The behavior of the bots is based on the behaviors from real players (probably from alpha). A WAN simulator is used between bots and realm servers; the average latency is set to 200ms. A high-performance machine and a "dedicated line" were needed.

EVE

Press, 2011, Orchestrator: A post-mortem on an automated MMO testing framework, slides

EVE started to make a thin client from its game client in 2010. Orchestrator is the tool they use to load-test and integration-test their architecture and code. Orchestrator does not send one script to each client, proxy, and server. Instead, it runs a single master script, and tells them, as they progress in the test, what the next operation is. The test can be stopped right when a client reports a bug, not until everything scheduled has been sent.

OpenSim

Lake, 2010, Distributed scene graph to enable thousands of interacting users in a virtual environment, paper

The limitations we have encountered with avatar scaling during these experiments have been in getting enough hardware to generate the load of over 1000 clients and the limited physics simulation capabilities of a single thread on the scene server.

League of Legends (allocating games to servers)

Delap, 2010, League of Legends: Scaling to millions of ninjas, yordles, and wizards, video + slides

  • Load-testing in a realistic setup: more than 50 machines with the same spec as those in production
  • EC2 is a good tool, but the network is not reliable, so careful not trying to fix problems that only happen in the test setting.
  • With thousands of clients, logs may not be the best way to gather test results.

League of Legends (chat)

McArthur, 2011, Building the chat service for League of Legends, slides

Dozen of EC2 machines, each running 5-9k bots. Each bot is an XMPP chat client (they used the Smack API). Load-testing is useless without proper modeling.

Crysis 2

Hall, 2011, A Programmer's Post-mortem Crysis 2 Multiplayer, slides

They wanted to check the frame rate with lots of moving entities, and detect bugs or gameplay issues. They used automatic testing. "Lots" of bots are run for 10 minutes per level to stress-test the builds. The bots do random actions like walking, jumping, or shooting.

Gears of War 3

Weilbacher, 2012, Dedicated Servers in Gears of War 3 Scaling to Millions of Players, slides

Their bots are clients without renderer and user input. They run automated bot matches to check the performance of their server platform. For Gears 2, they used to run 2.5 games per core in 2009. For Gears 3, they run 7 games per core in 2011.

Guild Wars 2

Patrick Wyatt, a lead programmer on Guild Wars and Guild Wars 2, discourages using bots: bot's behavior differs too much from actual users. Instead, he recommends recording live play data, and replay it on the server to fix bugs or check the load.

26 August 2012

Dead MMOs closing their eyes on private servers?

There are many MMO server emulators out there. Some private servers are run for profit, and when it is the case, the official and legal operator of the game can sue for disgorged profit, as was the case in Blizzard vs Scapegaming in 2009. Interestingly, out of the $88M Scapegaming had to pay, $3M came from disgorged profit (the private server was running micro-transactions), and $85k came from statutory damages for copyright infringement. Some private servers are run for fun and do not even accept donations. In the case of Ragnarok Online, Gravity used to send cease-and-desist letters to the private servers with the largest number of players. Servers with a dozen players did not seem worth the hassle.

Both WoW and RO are still running. However, the policy towards private servers seems to change when the MMO is discontinued by its legal operator. For instance, SWG closed in December 2011. SWG had an emulator called SWGEmu. In August 2012, the president of SOE unofficially supported SWGEmu: well officially I have to say it's bad. but that's officially. SOE or LucasArts could have been copyright trolls, but instead, they have decided to close their eyes on private servers. Maybe we can expect the same from Blizzard when WoW dies?