07 September 2012

Event managers and MVC - part 2

I wrote about MVC and event managers in January. Here's more!

Ordering of tick events in the game loop

A traditional game loop follows roughly this pattern:

  • process inputs on model (keyboard, mouse, network rcv)
  • process tick on model (update each entity and the world)
  • tick the renderers (graphics, sound, network send)
  • eventually wait for next frame, and repeat

Problem: Let's start with a simple event manager that does not make any difference between the inputs, models, or renderers. In other words, the event manager has no clue about MVC. Then, two annoying things happen. First, and most importantly, the event manager could publish a tick event generated by the clock to the model first, to the controllers second, and to the views last. It's as if we were running a different loop: tick the model, process inputs, and render. Since inputs need a whole frame to be processed on the model, the controls may feel laggy to the player (33ms lag at 30 FPS). This is terrible in twitch games like FPS or fighting games. Second, the dispatching of events could be non-deterministic: in one frame, controllers are ticked first, while in another, models are ticked first. The lag to process player commands would be random, and they may not be able to adapt to it.

Solution: We need an event manager that follows the game loop described above. Poll the controllers first, and send the events they generate right away on the model, views, or other controllers that subscribed to those events. Then tick the model and publish any resulting events right away. And finally, send the tick event to the views.

Implementation: the original simple event manager had a dictionary mapping each event type to a list of callbacks. The simple event manager was not differentiating between tick events and other events. Now, we need to distinguish between tick and non-tick events. And for tick events, we must distinguish between controllers, models, and views. This results in a set of controller callbacks, another set of model callbacks, and a third set of view callbacks for the tick events, and the usual dictionary mapping each non-tick event type to a list of callbacks.

Events that generate new subscribers

Problem: We're sending a tick event to the model. The model happens to create a new creature in the game. Let's say that the data of this creature needs to be updated every simulation step, so it should subscribe to the tick event as a model. However, the event manager is currently iterating on the set of model entities currently subscribed to be ticked. And in Python, when a set is iterated on, it can not be modified. I'm also guessing that lockless sets do not belong to the standard library of most programming languages, so this problem may not be unique to Python.

Solution 1: Before starting to process the tick event on the models, make an empty set. This set will store the callbacks of each model entity that started to subscribe to the tick event during this current frame. So we call this set new_callbacks. Our newly-created creature will be put in that set by the event manager when it subscribes to the tick event. When the event manager has finished iterating on the usual set of model entities, it makes a copy of new_callbacks, and replaces new_callbacks by an empty set. Then, the event manager iterates on the copy set, ie our new creature. If the creature spawns 3 minions at the first tick of its existence, the event manager will store the 3 callbacks from the minions in new_callbacks. Then, when the event manager is done with processing the copy, it adds the copy's callbacks to the usual set. Then, it replaces the copy by new_callbacks, and ticks the 3 minions. This can go on until both new_callbacks and the copy set are empty. Then, all old and new models have been ticked. Below is the output I'm expecting:

Created controller
Created model
Created view
---- Clock tick 0
Ticked controller
Ticked model
Created boss
Ticked boss
Created minion
Ticked view

The code does not exactly output this: the view is ticked before the boss is created. Oh well ... :-)


Solution 2: Solution 1 involves 3 sets and is quite convoluted, but there is simpler (and more dangerous!): using a data structure that is not locked when iterated on. In Python, this means using a list instead of a set. However, it's not the perfect data structure for the job: if an entity happens to subscribe twice to an event, the entity will receive the event twice. Most likely, your entities only need to be ticked once per frame, so if a bug causes an entity to act twice faster than expected, you'll know where it might come from. That's why modifying a list while iterating over it is not recommended.

28 August 2012

Bots for load-testing

A list of MMOs, MOBAs, and FPS games that have been load-testing using bots.

TERA

Koo, 2010, How to support an action-heavy MMORPG, slides

Their system to load-test a shard is called Sisyphus, and runs 1500 clients per machine. The behavior of the bots is based on the behaviors from real players (probably from alpha). A WAN simulator is used between bots and realm servers; the average latency is set to 200ms. A high-performance machine and a "dedicated line" were needed.

EVE

Press, 2011, Orchestrator: A post-mortem on an automated MMO testing framework, slides

EVE started to make a thin client from its game client in 2010. Orchestrator is the tool they use to load-test and integration-test their architecture and code. Orchestrator does not send one script to each client, proxy, and server. Instead, it runs a single master script, and tells them, as they progress in the test, what the next operation is. The test can be stopped right when a client reports a bug, not until everything scheduled has been sent.

OpenSim

Lake, 2010, Distributed scene graph to enable thousands of interacting users in a virtual environment, paper

The limitations we have encountered with avatar scaling during these experiments have been in getting enough hardware to generate the load of over 1000 clients and the limited physics simulation capabilities of a single thread on the scene server.

League of Legends (allocating games to servers)

Delap, 2010, League of Legends: Scaling to millions of ninjas, yordles, and wizards, video + slides

  • Load-testing in a realistic setup: more than 50 machines with the same spec as those in production
  • EC2 is a good tool, but the network is not reliable, so careful not trying to fix problems that only happen in the test setting.
  • With thousands of clients, logs may not be the best way to gather test results.

League of Legends (chat)

McArthur, 2011, Building the chat service for League of Legends, slides

Dozen of EC2 machines, each running 5-9k bots. Each bot is an XMPP chat client (they used the Smack API). Load-testing is useless without proper modeling.

Crysis 2

Hall, 2011, A Programmer's Post-mortem Crysis 2 Multiplayer, slides

They wanted to check the frame rate with lots of moving entities, and detect bugs or gameplay issues. They used automatic testing. "Lots" of bots are run for 10 minutes per level to stress-test the builds. The bots do random actions like walking, jumping, or shooting.

Gears of War 3

Weilbacher, 2012, Dedicated Servers in Gears of War 3 Scaling to Millions of Players, slides

Their bots are clients without renderer and user input. They run automated bot matches to check the performance of their server platform. For Gears 2, they used to run 2.5 games per core in 2009. For Gears 3, they run 7 games per core in 2011.

Guild Wars 2

Patrick Wyatt, a lead programmer on Guild Wars and Guild Wars 2, discourages using bots: bot's behavior differs too much from actual users. Instead, he recommends recording live play data, and replay it on the server to fix bugs or check the load.