Bots for load-testing

A list of MMOs, MOBAs, and FPS games that have been load-testing using bots.

TERA

Koo, 2010, How to support an action-heavy MMORPG, slides

Their system to load-test a shard is called Sisyphus, and runs 1500 clients per machine. The behavior of the bots is based on the behaviors from real players (probably from alpha). A WAN simulator is used between bots and realm servers; the average latency is set to 200ms. A high-performance machine and a "dedicated line" were needed.

EVE

Press, 2011, Orchestrator: A post-mortem on an automated MMO testing framework, slides

EVE started to make a thin client from its game client in 2010. Orchestrator is the tool they use to load-test and integration-test their architecture and code. Orchestrator does not send one script to each client, proxy, and server. Instead, it runs a single master script, and tells them, as they progress in the test, what the next operation is. The test can be stopped right when a client reports a bug, not until everything scheduled has been sent.

OpenSim

Lake, 2010, Distributed scene graph to enable thousands of interacting users in a virtual environment, paper

The limitations we have encountered with avatar scaling during these experiments have been in getting enough hardware to generate the load of over 1000 clients and the limited physics simulation capabilities of a single thread on the scene server.

League of Legends (allocating games to servers)

Delap, 2010, League of Legends: Scaling to millions of ninjas, yordles, and wizards, video + slides

• Load-testing in a realistic setup: more than 50 machines with the same spec as those in production
• EC2 is a good tool, but the network is not reliable, so careful not trying to fix problems that only happen in the test setting.
• With thousands of clients, logs may not be the best way to gather test results.

League of Legends (chat)

McArthur, 2011, Building the chat service for League of Legends, slides

Dozen of EC2 machines, each running 5-9k bots. Each bot is an XMPP chat client (they used the Smack API). Load-testing is useless without proper modeling.

Crysis 2

Hall, 2011, A Programmer's Post-mortem Crysis 2 Multiplayer, slides

They wanted to check the frame rate with lots of moving entities, and detect bugs or gameplay issues. They used automatic testing. "Lots" of bots are run for 10 minutes per level to stress-test the builds. The bots do random actions like walking, jumping, or shooting.

Gears of War 3

Weilbacher, 2012, Dedicated Servers in Gears of War 3 Scaling to Millions of Players, slides

Their bots are clients without renderer and user input. They run automated bot matches to check the performance of their server platform. For Gears 2, they used to run 2.5 games per core in 2009. For Gears 3, they run 7 games per core in 2011.

Guild Wars 2

Patrick Wyatt, a lead programmer on Guild Wars and Guild Wars 2, discourages using bots: bot's behavior differs too much from actual users. Instead, he recommends recording live play data, and replay it on the server to fix bugs or check the load.

Player-developer communication in EVE Online

CCP has an interesting take on community management. The developer's blog of EVE contains interesting behind-the-scene technical information. What is even more interesting is that these articles do not seem to be filtered or censored by any community manager. Articles are direct from the development team. I guess that the management team originally thought this communication policy was a good idea, and the developers were more than happy to post about their work. In any case, this communication policy has advantages and drawbacks.

Drawbacks

On the one hand, revealing to your players that a major database update requires a 14-hour downtime and may result in more bugs in the game (October 2010) is kind of risky. Another developer article (February 2011) started with:

Testing may show that the changes made are not safe and we won't be able to use them. I thought this would be interesting enough to share, but please do keep this disclaimer in mind as you read on.

Players start complaining on the forum boards, and it gets worse when the down time actually takes longer than the announced 14 hours. Moreover, when developers and players get close together, developers may get involved in the game and start giving unfair advantages to the players they like. This kind of cheating can get really messy (also look at the Wikipedia summary of the events).

Advantages, and why it works for EVE

On the other hand, EVE benefits from a huge support of its player base. A bunch of volunteer programs, composed of hardcore and loyal players, add content into the game, administrate a wiki, and provide player support at no cost for CCP. EVE designers targeted the game's steep learning curve for hardcore players. It's not because you are a hardcore player that you never need information or help. In fact, I think it's the opposite: hardcore players bookmark any possible source of information for their game. Therefore, EVE needs its active player community, and needs to keep a tight relationship with its player base. To receive fast and broad feedback, CCP opened nine player positions in the Council of Stellar Management in March 2008. Members of this council were elected for six months by all EVE players and invited to the development studio. They gave feedback to the developers in person. This council may have also been a marketing coup from CCP, as the studio also invited various journalists to cover the event.

So far, tight player-developer communication has been a winning strategy on many fronts for CCP. Yet, EVE does not put forward any charismatic developer. Maybe is it better like this?