Bots for load-testing

A list of MMOs, MOBAs, and FPS games that have been load-testing using bots.

TERA

Koo, 2010, How to support an action-heavy MMORPG, slides

Their system to load-test a shard is called Sisyphus, and runs 1500 clients per machine. The behavior of the bots is based on the behaviors from real players (probably from alpha). A WAN simulator is used between bots and realm servers; the average latency is set to 200ms. A high-performance machine and a "dedicated line" were needed.

EVE

Press, 2011, Orchestrator: A post-mortem on an automated MMO testing framework, slides

EVE started to make a thin client from its game client in 2010. Orchestrator is the tool they use to load-test and integration-test their architecture and code. Orchestrator does not send one script to each client, proxy, and server. Instead, it runs a single master script, and tells them, as they progress in the test, what the next operation is. The test can be stopped right when a client reports a bug, not until everything scheduled has been sent.

OpenSim

Lake, 2010, Distributed scene graph to enable thousands of interacting users in a virtual environment, paper

The limitations we have encountered with avatar scaling during these experiments have been in getting enough hardware to generate the load of over 1000 clients and the limited physics simulation capabilities of a single thread on the scene server.

League of Legends (allocating games to servers)

Delap, 2010, League of Legends: Scaling to millions of ninjas, yordles, and wizards, video + slides

• Load-testing in a realistic setup: more than 50 machines with the same spec as those in production
• EC2 is a good tool, but the network is not reliable, so careful not trying to fix problems that only happen in the test setting.
• With thousands of clients, logs may not be the best way to gather test results.

League of Legends (chat)

McArthur, 2011, Building the chat service for League of Legends, slides

Dozen of EC2 machines, each running 5-9k bots. Each bot is an XMPP chat client (they used the Smack API). Load-testing is useless without proper modeling.

Crysis 2

Hall, 2011, A Programmer's Post-mortem Crysis 2 Multiplayer, slides

They wanted to check the frame rate with lots of moving entities, and detect bugs or gameplay issues. They used automatic testing. "Lots" of bots are run for 10 minutes per level to stress-test the builds. The bots do random actions like walking, jumping, or shooting.

Gears of War 3

Weilbacher, 2012, Dedicated Servers in Gears of War 3 Scaling to Millions of Players, slides

Their bots are clients without renderer and user input. They run automated bot matches to check the performance of their server platform. For Gears 2, they used to run 2.5 games per core in 2009. For Gears 3, they run 7 games per core in 2011.

Guild Wars 2

Patrick Wyatt, a lead programmer on Guild Wars and Guild Wars 2, discourages using bots: bot's behavior differs too much from actual users. Instead, he recommends recording live play data, and replay it on the server to fix bugs or check the load.

TERA's free-targetting combat: server-side

In the May 2012 issue of GD mag, Seungmo Koo, the server architect for TERA, wrote about how their studio implemented the server back-end for the game's free-targeting combat system. Some parts of the technical solution described in the article were unclear, or the problems were not obvious, so I filled the gaps by guessing their approach.

Basics

During fights, the player uses various kinds of attacks and skills. For the player, free-targeting means paying attention to the avatar's orientation and position. There is no monster selection like in WoW. Looking at actual gameplay footage, the combat feels more lively, but there is a lot of hit and run.

From a system perspective, free-targeting means the server does not know which enemy the avatar is targeting. Hence, the server has to detect if a skill's volume of effect collides with any of the monsters around the avatar. To ensure quick computation of the collision, the server runs at 60 FPS. These features are similar to traditional multiplayer FPS requirements, except that for security and simplicity, there is no lag compensation running on the client-side. All the computations happen on the server-side. However, for an MMO to reach a gameplay as responsive as an FPS, the client should be able to execute the player's actions immediately, account for lag, and eventually run some of the NPCs. TERA accounts for lag like an MMO: in PvE, NPCs are slow to give players time to react. In PvP, however, players with a lower ping to the server have an advantage.

Game design

Every creature in the game world is made of cylinders. A skill reaches its target when the skill's volume of effect collides with one of the target's cylinders. A skill's volume of effect can be a cylinder (e.g. usual AoE around a target or self), a portion of a cylinder (e.g. rotating attack using an axe), a portion of a cone (e.g. a frontal attack using a spear), and so on.

Since skills may last for some time, they are discretized in two to more than a dozen targeting times, and each targeting time is associated with a particular volume of effect. Each targeting time consists of two phases: a search (read) phase, and an update (write) phase. For instance, using a rotating attack, the avatar spins for 360 degrees in 300ms, inflicting damage during its turn. The first targeting time fires at 50ms: in the search phase, the server searches for a target standing in a particular volume in front of the avatar, and, in the update phase, inflicts that target 200 damage. The second targeting time fires at 100ms: the server searches for a target standing in another particular volume on the left of the avatar, and inflicts that target 100 damage. And so on for the remaining targeting times.

Multi-threading and performance

Task	Example
Timer	Damage over time or NPC AI movement scripts
Region update	Add/remove/update the position of a creature in a region
Creature update	Update HP, status, position of a creature

Each TERA server runs the whole world for approximately 6k concurrent players. The server has a few things to do in the main loop. First, it processes client packets, and generates timer, region, and/or creature tasks from them. That is when the search-for-target phase happens. Then, it executes scheduled timer tasks, which will likely themselves generate region and creature tasks. And finally, it executes region tasks. The update phase happens in the creature tasks, and also in the region task in case of creature movement, spawn, or death events.

To come up with their final design, the TERA engineers may have followed this train of thoughts:

• Problem: an 8-core server running Windows reaches 100% CPU with 3k players. How can this be improved?
• Solution: use asynchronous IO such as Window's IOCP. It automatically creates a task queue for client packets and a worker thread pool to process that packet queue in parallel.
• New problem: The threads spend too much time waiting for locks when trying to read or update region data (e.g. list of creatures in the region, or creature position).
• Solution: Avoid contention by replicating the world in each thread. No lock: the thread searches in its own version of the world.
• New search-phase problem: When executing a region task, the region data of a particular thread becomes inconsistent with other threads.
• Solution: The thread pushes the region tasks it generates into a queue shared between all threads, so that each thread processes all the region tasks. As an example, when an 8-core server receives a client movement packet, one thread processes it and generates (for instance) one region task from that packet. The thread pushes this region task to the end of the queue with an execution counter of 8. Each of the 8 threads processes the task queue at its own pace. When a thread processes a task, it decrements the task's counter. When the counter reaches 0, the thread deletes the task from the queue. Hence, the task mentioned above will be removed from the queue when all the threads have processed it.
• New update-phase problem: frequently updating a particular creature causes lock contention.
• Solution: The world replica of each thread stores creature pointers. When a thread processes a packet or a timer task that results in a creature update, it pushes the creature's function to run (e.g. Creature.receiveDamage), with particular arguments (e.g. 200) to that creature's task queue. If the creature task queue is currently being iterated over by another thread, then the current thread returns to what it was doing before (e.g. main loop). Otherwise, the current thread iterates over the creature's task queue. The queue starts with the task the thread just pushed, but other threads may push tasks to the queue while the current thread is processing its task. The current thread will process those extra tasks too, until the queue is empty. Then, the thread will go back to its main loop. Hence, creature tasks are executed asynchronously; getting a return value for the task execution requires the active object pattern.

The resulting high-level architecture is something like the following:

Questions

Seugmo replied to my original questions (see the comment below); here are some more!

• How much CPU is caused by packet processing? How much does the CPU increase when 1000 players gather together versus when players are homogeneously scattered?
• When lots of creatures gather in the same region, the search phase (happening on only one thread) may consume a lot of that thread's CPU. Could the search phase happen faster if the regions were smaller? Could it happen faster if the regions were using Binary Space Partitioning?

Networking and online games - Armitage 2006

Notes from "Networking and Online Games - Understanding and Engineering Multiplayer Online Games", by Armitage, Pool and Branch, 2006

Latency, jitter, and packet loss

Latency sources:

• minimum delay: distance of link / speed of light (3ms for every 1000km)
• serialization delay (transmitting a 1500 byte IP packet takes 0.12ms on a 100Mbps link, 20ms through 512kbps ADSL, and 350ms through 33kbps dial-up modem)
• queuing delay: packets from different sources sharing the same link have to wait to be transmitted (add 50-100ms to the RTT if on wifi)

Jitter: main source is burstiness of queuing delay.

Packet loss: main sources are rare data corruption, temporary dynamic routing changes, and proactive packet dropping (the bigger the router's message queue, the less likely to accept new messages; this slows down TCP and completely drops UDP).

Possible solutions:

• Preferential queuing of packets based on their source, destination, and protocol.
• Increase the size of router's active queue management
• Interleave multiple IP packets at the same time through multiple virtual channels, thanks to ATM and the 2 channels of ADSL

Latency compensation

"dumb client" = waits for server approval to render user inputs. no latency compensation. Types of latency compensation techniques:

• prediction = playing on consistency-responsiveness trade-off .
  • Player prediction = client only predicts local player's units and fixes discrepancies when receiving server response.
  • Opponent prediction = predict other players' units based on their velocity and position; other players only send updates on their units when their actual position differs from the predicted position from more than a given threshold. Ex: sudden right turn). Trade-off between fidelity (the notification threshold) and number of messages sent. Unfairness problem: players with higher latency receive updates later. Solution: Time Delay (see bellow)
• time manipulation:
  • Time Delay: server delays processing and sending of user commands to equalize latencies (or up to a maximum buffer size). Pro: fairness. Con: as responsive as the slowest client (or adds latency).
  • Time Warp: server knows current lag to each client. When receiving a client input, server uses current state minus lag to compute new state, and rolls back everyone's state if computed state differs from current state. Used in Valve's engine. Problem: client can cheat by pretending it's lagging a lot while it's not, and send actions that will roll back the state to his advantage.
• bandwidth reduction (hence reduction of serialization delay): LZW compression, sending state deltas instead of the whole new state, interest management, using P2P for voice, and bucketing/update aggregation
• visual tricks: show an animation to cover part of the lag

Playability vs network conditions and cheats

2 ways to discover player tolerance to network: 1) lab experiment, or 2) monitoring public servers.

Network simulations: Using custom routers to simulate latency and packet loss: NISTnet for Linux or dummynet for FreeBSD. ns-2 is a traffic generator. Problem: OS scheduler = 10ms buckets. Solution: change the kernel tick rate to 1ms. Other problem: simulation parameters are game-specific, change with the number of players, etc. (although server-to-client packet simulations seem to estimate reality quite correctly).
Use a normal (not uniform) distribution for latency for more realistic jitter. But careful: packet re-ordering may happen if modifying Nistnet's packet delay value on the fly.

Types of cheats

Type	Description	Solution(s)
client-side	graphics (e.g. wallhack), input (e.g. aimbots runs as proxy and rewrites 'shoot' command packets, 24/7 farm bot)	only send to clients the state they need, and check that client is running normally on the client-side (e.g. PunkBuster
server-side	escaping a game you're losing to not be ranked, brute-forcing an opponent's password to steal his account, or to prevent him from logging in	limit msg rate
network	DDoS to increase victim's latency	hide players IP

Traffic patterns

Sniffing packets on the server side: use a hub, or a switch with port mirroring. Tools: tcpdump (command line) and ethereal (GUI) to capture ethernet frames both in and out. Problem: The accuracy of packet timestamping of the sniffer depends on CPU clock precision, IO throughput, CPU load, ... Solution: calibrate sniffer from a known source emitting regular packets.

Player information: Reverse-DNS-lookup of player IP to know where he comes from. Careful: 2 players behind the same NAT receive the same IP, but they use a different port, so 1 player = (ip, port), not just ip.

Tick and bandwidth: In Wolfenstein Enemy Territory and Quake 3, the server ticks every 50ms (20 hz), while for Half Life 2, it ticks at 15ms (66 Hz). However, for both, the client requests updates every 50ms. The client can chose to request packets more often (e.g. 100 packets/sec), but the server will still send at most at its tick rate (20 hz or 66 hz). Clients can also specify a max downstream bitrate (e.g. 50kbps); this is useful when update packets are too frequent or too big.

Packet size and inter-arrival time: Server-to-client packet size depends on the map design, number of players, ... Packet inter-arrival times: if one packet per player, and each of the n players receives updates about the other n-1 players every server tick, then each client receives a burst of n-1 packets per server tick. In other words, n-1/n of the inter-arrival times should be in the "sub-millisecond region". For example, every 50 ms, a Quake 3 server with 15 players will send to each client 14 packets at a time.
Client-to-server packet size and inter-arrival times depend on client hardware, gameplay behavior, ... but each client has a distribution of packet size and inter-arrival very distinct from other clients. [It could be possible to identify clients that way, and eventually stolen accounts or power-levellers].

Netgames 2003

Modeling player session times of online games, by Chang and Feng

• network traces of a popular CS server for a week in April 2002
• 16k user sessions recorded
• 99% of players play less than 2 hours
• play session follows a Weibull distribution with k = 0.5 and λ = 20 (shape similar to 1/x exp(-x))
• For play sessions from 10 to 100 minutes, the chance of disconnecting (ie failure rate) remains constant at 2.5%.
• For play sessions shorter than 10 minutes, 10% chance of disconnecting. Possible reasons: connection problems, kicked out or leave because of server rules (such as friendly fire allowed, but kicked out if you kill your team-mates too often)

A Fair Message Exchange Framework for Distributed Multi-Player Games, by Guo et al.

• Assumptions: independent clocks with no synchronization mechanism, players react to server updates, updates only consist of creation and/or removal of object(s) (and NOT object position updates)
• Users have reaction time to act in response to server update messages. Ignore latency induced by network and only compare user reaction times to determine which update to actually run on the world state.
• the Fair-Ordering Service [...] dynamically enforces a sufficient waiting period on each action message to guarantee the fair processing of all action messages. But practically, the waiting period is bounded to ensure a relative level of interactivity.
• Proxies are game-agnostic and located near players (ie low latency between a player and her proxy). Proxy receives action message from user, then forwards that action message with a message identification number (to deliver messages in order) and the reaction time to the game server.

Causality and media synchronization control for networked multimedia games: centralized versus distributed, by Ishibashi et al.

• Causality control preserves the order of events of game data (keyboard inputs). No need for causality in voice or video
• Media synchronization control = intra-stream (temporal relation between MU such as voice or video packets) + inter-stream (timing among multiple streams) + group (timing among multiple end-points to ensure fairness) synchronization controls
• Compare C-S to P2P architectures in terms of success of the 4 previously mentioned control schemes. Voice and video don't need to go through the server (they're sent in P2P mode in both scenarios).
• Adaptive Δ-causality control used on game data in both scenarios: the recipient considers a packet still valid Δ = 50 ms after its generation timestamp. [That means the latency automatically increases by Δ ms for all packets]. Adaptive means that the value of Δ changes based on the network load. Smaller Δ = game more interactive, large Δ = less packets are discarded for being late/misordered. Unfairness appears when terminals have different Δ, hence need group sync control.
• Piggy-back an MU on the succeeding k=4 MUs to recover from lost UDP packets
• Experiment: two terminals in both C-S and P2P scenarios [only two?!]. Terminal 1 is connected to an overloaded hub with delay jitter, Terminal 2 is connected to its own hub. Connections are 10 Mbps ethernet. Server connected to T2's hub. Additional delay of 100 ms introduced between the two terminals by a data link simulator between T1's hub and T2's hub. Game MUs = 20 Bytes, sent 10 times per second, while voice MUs = 400 Bytes, sent 20 times per sec, and video MUs = 5kB, sent 20 times per sec [hence most of the load on the network comes from voice and audio, not game data]. Experiment ran for 2 minutes.
• For heavy loads (8Mbps), C-S is better for causality, but worse for consistency, fairness, and interactivity.

Bandwidth requirement and state consistency in three multiplayer game architectures, by Pellegrino and Dovrolis

• Compare C-S, P2P and PP-CA (= P2P with central authority/arbiter receiving moves from all players and notifying them when it detects inconsistencies)
• Tu = Duration of client loop, Lu = size of update messages
• CS: client upstream = Lu/Tu, client downstream = N.Lu/Tu, server downstream = N.Lu/Tu, server upstream = N(N.Lu)/Tu
• P2P: client upstream = client downstream = (N-1)Lu/Tu
• PP-CA: client upstream = N.Lu/Tu, client downstream = (N-1).Lu/Tu + f.N.Lu/Tu with f = ratio of inconsistencies to be corrected, arbiter downstream = N.Lu/Tu, arbiter upstream = f.N(N.Lu)/Tu

Access network delay in networked games, by Jehaes et al.

• Look at delays introduced from access networks (aka last mile links), not from back-bone. Goal: how to dimension the network to reach minimum delay possible.
• Network delay can be caused by propagation (mostly only in the case of back-bones though: 5µs/km), serialization (putting all the bits of a packet on the link), packet processing (route and DNS lookups, error correction), and queuing (other packets have to be treated before; differs from packet to packet, hence jitter, defined as 95% percentile RTT - 5% percentile RTT). AND = minimal RTT (packet processing delay) + S (packet size) / Reff (effective link rate) + Tque (total queuing delay in up- and downstream, results in jitter)
• Experiment: for 5 different values of S, throw 100 pings. Get RTT and jitter (= Tque) from 100 pings. Obtain Reff from taking the inverse of the best-fitting trend-line through the 5 points (S, average RTT). Obtain min RTT from the intercept of the trend-line through the 5 points (S, top-1% RTT).
• QoS improves RTT by separating game traffic from other traffics

A Zone-based Gaming Architecture for Ad-Hoc Networks, by Riera et al.

• Assumption: ad-hoc networks are going to multiply, but C-S and P2P architectures are not well-suited for them. Most interesting part of the job is to determine which device can/should be Zone server.
• Even the nodes that do not play the game assist the other nodes in delivering data
• Nodes are mobile: they do not always stay within reach of the same other nodes. Discovery of Zone servers is done through the SLP v2. When latency to a zone server gets too high, a client can pick another zone server, which in turn notifies all the other zone servers of its new connection. When a client does not reply for a while, a zone server can drop it.

A service platform for on-line games

• Middleware as transparent as possible to the game developer. This middleware sits on top of an existing grid infrastructure from IBM called Globus. Globus decides when to spawn a new game server instance based on current resources and demands.
• Player services are in charge of authentication, account handling, chat rooms, locating games/selecting a server taking into account player preferences (e.g. team or region), and actually playing the game.
• Publisher services deal with software deployment and updates, billing, monitoring server performance, service level agreement (e.g. no more than 5% of players suffer from more than 100ms delay)
• System services include resource management and directory services. These services are accessed by the grid provider.
• Clients submit jobs using a format containing the executable, its arguments, and resource requirements. Jobs can require spawning instances at different grid locations (e.g. different regions).
• Various services such as resource informations and information providers (CPU, OS, RAM, connectivity, ...) are indexed in an LDAP. Game-specific services (tracking player stats, server load, ...) could also be added on top of existing services.

Scalability definitions and milestones

Definitions

Scalability is a major technical focus for MMOG, but there are actually many different meanings and assumptions hidden behind the word. In general, Scalability is the capability of a software system to be adapted to meet new requirements of size and scope (Taylor et al. p467).

For system and network engineers, scalability can be achieved in two ways: horizontally (load balancing and clusters of multiple computing resources) or vertically (adding more power to a current computing resource). In both cases, if any of the layers of the stack (hardware, database, application and so on) does not scale, then the whole system won't.

For software engineers, a system scales well if its rate of growth is not greater than the corresponding rate of complexity increase (Taylor et al. p468-474). This definition looks a lot like positive scalability in system and network. However, software architecture scalability also deals with adding new components and takes into account the dependencies between components and connector logic.

Milestones

To my mind, there are so many definitions of scalability that it makes more sense to talk about domain-specific metrics meaningful to the end-users and stakeholders. The number of users is a perfect example. So here is a list of situations illustrating different meanings of scalability with different numbers of users.

Configuration	System/Example	Scalability meaning
Scalability milestones reached so far
2 or more players sharing the same game world state	Game development platforms such as Unity have quite hard times providing a transparent interface to sharing game world state. Each object in the world has to explicitly contain a Network View for it to be sent to other players.	Being able to send all relevant game objects to other players without the hassle of having to add a networking layer on top of them.
From 40 to 100+ users in the same world	Throughout 2009, Intel has worked on scaling OpenSim. They noticed that the number of users (as well as the number of object prims and scripts) were independent of the number of hardware threads available. This meant OpenSim could host no more than 100 clients, even if more resources were added.	Scalability was achieved through an impressive cleaning and optimization work from both software and netsys engineering perspectives. Adding a server to the back-end means being able to handle more users (thanks to an inter-shard daemon mechanism of some sort). There is still a problem, though: if everyone goes on the same shard, the shard crashes. Sharding originally comes from the database community. It has been applied to the software architecture of military simulations in 1995: 1000 users can be supported simultaneously. Current MMOG architectures still cut their world in maps (shards) to be able to use this principle.
Inter-server events	Cross-realm pick-up groups in WoW. Opensim's hypergrid foreign user information stored locally and non-persistently on remote servers.	In this case, scalability sounds more like portability and data compatibility (as the number of users involved stays relatively small). Also, an extra layer of logic can be added: in WoW, experienced players can be grouped with novices so that cross-realm groups are more balanced. But ... on which server shard(s) are these instances stored and computed?
Cloud gaming (Gaikai or OnLive)	Streaming video to computers that do not have enough graphics power to run the game themselves	A pure example of horizontal scalability: more computers in the cloud means more users.
Scalability milestones not reached yet
1000 players seeing and interacting with each others	Giant boss battles, live events, voting in Parliament	1) Clients must still be able to compute the graphics and 2) the server has to be able to cope with the sudden and unusual overload (dynamic resource re-allocation? Hybrid architectures mixing P2P and client-server?)
Cloud gaming for MMOG	Cloud gaming NOT as a service on top of existing gaming infrastructures, but as a graphics help for weak clients of an MMOG	Some MMOGs have high-end graphics. And there is always a server doing various computations. In the end, the overall architecture would approach something like: weak client <-> "middlebox" shared with a few other weak clients for the graphics <-> game logic server. But this naive method consumes a lot of resources and can certainly be optimized. It sounds a little bit like booster boxes.
10.000+ players seeing each other and interacting with some	Stadiums, French demonstrations	For the server, scalability means 1) being able to distinguish who can interact with who, 2) allocate resources dynamically (as these events are temporary and can be spontaneous) and 3) approximate distant actions and determine which clients share the same content locally so that some computations can be factorized. For the client, graphics might get demanding.

Vivox in SL: client, server and protocols

Server-side

As I wrote before and based on Vivox' white paper, the main point is the Server-side mixing all voices in real time and delivering the audio in a single stream. I could not find much more information about the SL-specific Vivox server system (Linden Lab will not reveal their server-side architecture that easily), but I guess the Vivox server-side does not differ a lot between MMOG/VW. unused but required parameters can be found in the SLVoice documentation. This suggests either Vivox cared for a retrocompatibility with the SLVoice Application 1.0 or Vivox did not tailor their client API to SL needs. I am for the later, even though I could not find any documentation for the SLVoice Application 1.0 infirming or confirming that.

Joe Miller explained how the Vivox server sends the audio stream to users and how the system can scale:

According to Miller, the VoIP product is unique because of the ability to project the sound in three dimensional space, as a function of distance and direction from one avatar to another. It takes a 32khz signal at 32kbps from clients, sends it to an Intel based audio server where the input signals are mixed and properly positioned, acoustically, in three dimensions, and a stereo stream is sent back to the client at 64kbps. Even with 100 people speaking at once, the bandwidth requirements are the same for each individual because the servers (dual quad-core Xeons) mix the voices together into a single data stream.
The codec used is Siren 14/G.722.1 Annex C, developed by Polycom but now an international standard. It was chosen because it uses relatively low bandwidth but can carry a wide and dynamic range of audio – not just human voices – making it an ideal codec to broadcast, say, a musical event.

The range at which other resident can hear each other are explained in the SL wiki article "How far does my voice carry". Similarly to text-chat, the server computes the distances between people to determine who hears who, and sends appropriate messages after this computation. Hence (and hopefully) it's impossible to use a modified Second Life Viewer to remove the hearing range limits.

The OpenSim server architecture might not differ a lot from the SL one regarding to voice support. However, I could not find it reading the OpenSim wiki.

Client-side

On the client-side, Linden Lab have chosen to keep the voice features outside of the Viewer: The Second Life Viewer handles configuration, control, and display functions, but the voice streams (from the microphone and from the Vivox voice server) do not enter the Viewer. In other words, These [voice] technologies are contained in external daemon software that is started and stopped by the Second Life client.

The requestId can/should be a GUID so that each response matches a unique request. Each gateway response also contains the request it received. This enables the XML-based protocol to be stateless. TCP provides a reliable transmission that prevents packet loss (important to update the UI reliably and in a timely manner).

voipforvw is a GPL alternative for SLVoice on OpenSim. One of its developer wrote it is a snap-in replacement for this executable [SLVoice.exe] that communicates with the viewer and as you’d guess, does the heavy lifting and coding/decoding. But the project started in February 2008 and has not received any commit since May 2009.

More about the client components in an incoming article ...

Voice protocols (in a nutshell)

The following protocols or techniques are used by some components in the SL client.

SIP is an application-layer protocol and incorporates many elements of HTTP such as headers, encoding rules and status codes. As indicated by its name, SIP is only used to initiate communications between clients. Clients start communicating in peer-to-peer after they have been paired by a SIP server. The SL Viewer uses ports 5060 (non-encrypted) and 5062 (TLS-encrypted) for SIP with UDP. Once clients are paired, they can start exchanging data.

ICE is not a protocol but rather an initialization technique that facilitates peer-to-peer communications in reducing the NAT-traversal delay. It uses a STUN client-server strategy to pair agents. When paired, agents do not rely on the server anymore.

RTP is an application-layer protocol that defines a packet format for delivering audio and video. RTP Use Scenarios in the RFC contain multicast, Mixers and Translators. The use of UDP for the transport layer is obvious in this real-time "send-at-most-once" media-streaming context. The SL Viewer uses the 12000 to 15000 (or 13000?) port range for RTP.

[Literature] Optimizing Online FPS Game Server Discovery through Clustering Servers by Origin Autonomous System

by Grenville Armitage, 2008 located here.

Armitage is Australian, and Australians seem to suffer from latency problems. This might explain why he was looking for an efficient way to solve a latency problem: server discovery in CS:S. According to the author, the (client-side) server discovery process algorithm is as follow:

1. Issue a getservers query to UDP port 27011 on the Steam master server
2. Receive a packet containing a list of 231 active servers <IP address:port>
3. Probe each game server in the previously received list
4. Repeat until the Steam master server has no more game servers to return

The speed of server discovery is actually configured by the user: it is the "What is your connection capacity" which possible parameters are "Modem 56k", "DSL", "DSL+", etc. This parameter changes the speed of game server probation (phase 3 in the algorithm above). There are approximately 30K game servers and with a Steam client configured with "DSL > 256K", 140 servers can be probed per second. Without optimization, it takes 230s to complete the discovery process with a Steam client configured with "DSL > 256K".

In general, players want the game server with the smallest ping or RTT. A first solution could be grouping servers based on their location, but it is not always true that the closer geographically the server is, the smaller the ping will be. The paper proposes clustering game servers by the Autonomous System to which each game server's IP address belongs (its origin AS), ie grouping servers based on the Internet topology. An AS relies on IP routing prefixes to group IP together. The author also added an automatic termination of server discovery when servers RTT are above a specified threshold such as 200ms. Here is the algorithm:

1. Retrieve all game servers
2. Group servers in clusters based on their AS number
3. Calibration process: this eventually splits clusters into smaller ones. In the end, the RTT of a randomly-picked server inside a cluster should be within a 40ms range of the cluster median RTT.
4. Rank clusters in ascending order of their median RTT
5. Probe the servers which have not been probed during the calibration process until a certain RTT threshold (200ms for instance)

Approximately 3k servers (10% of all the servers) are probed during the calibration phase. Around 1k clusters are created during the calibration process. In locations such as Australia, Taiwan or Japan, the algorithm stops within a minute and gives nearly all the available servers which RTT is below the specified threshold. Countries like Germany, the USA or the UK are closer to game servers and their RTT is smaller in average. Therefore, reaching the 200ms RTT threshold takes nearly the same time for players in these countries. However, if the RTT threshold is lowered to 100ms (for instance), the algorithm may stop much faster. If the player could specify the RTT threshold he/she wants the algorithm to stop, the proposed solution could be an efficient improvement to the current discovery process of Valve Counter-Strike game servers.