Panel on MMOs from Netgames 2011

Panel at NetGames 2011 about the future of MMO research.

Cheating (no control of the client-side) and performance (weakest node performance conditions the other nodes' performance) are issues in p2p architectures. I could p2p when I play with friends, but not competitively. Even with friends, the developer has no control over the network, NAT traversal problem. Potential solutions: match-making on server, then one peer hosts the game. If host disconnects, the game migrates to another peer. Problem: how to partition and replicate the game world between peers?

How to integrate scaling into game mechanics?

Researchers can get data from companies if they show that it is worth the company's time (taken to fetch and anonymize data).

Problems shared across p2p, client-server and cloud architectures: scalability, cheating, latency, concurrency, synchronization, and replication.

Using the cloud is cheap for small user bases, but very expensive for very large user bases.

Casual Connect and GDC Online are industry conferences that also deal with network and system support for MMOs. But they're expensive.

MMO player research methods

Types of data of interest in MMO player studies:

• demographic data: country, gender, age, job, psychological traits, tech-saviness, happiness, revenue
• marketing data: how much spent on games per month, how many games bought,
• play data (applicable to all games and game genres): weekly play time, average play session duration, game and genre literacy,
• genre-specific and game-specific data: for MMOs and WoW in particular: who you play with, guild position, achievement/immersion/social motivation scores

Challenges of player studies: using tools and methods to convert data into useful information, and avoiding erroneous conclusions by crossing results obtained from various methods.

List of qualitative tools.

Methods	Qualitative	Quantitative
Data collection tools	Note taking or recording during open-ended interviews, lab studies, think aloud, or participant observation/ethnographic play. All methods gather all types of data - you just need to ask. Snowball sampling is useful to collect more people concerned by the same phenomenon (eg people from the same guild).	Questionnaires can gather multiple types of data together, but beware: what people say they do differs from what they actually do. Non-obtrusive logging of game data has the advantage of being objective, and can be done using programmable game add-ons that players need to install. It's even possible to cross game and marketing data together from the developer/publisher side (cf the EverQuest dataset from SOE).
Data analysis tools	Note tagging and affinity diagrams are methods used during the note parsing phase. Stop interviewing when respondents do not say/show anything new anymore. Preferably, do not wait to start parsing the data; parsing as notes are taken gives an immediate feedback loop useful to detect useless interview questions, and to know when to stop accumulating data.	The most popular tools to use for stat analysis are SPSS, R, and Excel. Obtaining causal relationships is usually quite hard. On the other hand, simple comparisons and correlations are often successful. Regressions can work too. Data mining/machine learning using Weka or Matlab. Clustering (PCA, LDA, and even KPCA if small dataset) can identify patterns during exploratory phases. Feature selection or decision trees to identify the most important features. Poisson process or Markov chains for temporal evolutions. Bayes, neural network, SVM, random forests, and others for classification.
Pros and cons	Deep, and useful to hint at the reasons explaining a particular phenomena, especially in MMOs, where the metagame (forums, blogs, chats) has a huge influence on the actual game. Snowball sampling in WoW brings lots of players from the same guild(s), or with similar opinions. Therefore (and also because of small sample size): poor ability to generalize. Qualitative methods can also be used as exploratory studies to help build a quantitative questionnaire with relevant questions.	Broad and useful to detect surface trends. Sample bias/representativity: it's very hard to select a representative sample of the player population. For example, selecting players from wow.com brings a lot of hardcore players (since hardcore players read forums while the most casual players don't). Very hard to explain completely a phenomenon because there's always important features missing from the dataset. Machine learning is also difficult for that same reason. It's very easy to get lost in post-hoc effects, or to simply not be able to explain a particular number because you've never played the game. Generally, demographic data can not be caused by game-specific data: it's not because people are hardcore that they're young, but rather the opposite.
Researchers	Bardzell, Bartle, Kow, Nardi, Pace, Pearce, TL Taylor	Andreasen (= quantitative Bartle test), Bateman, Ducheneaut, Seay (not working on that domain anymore?), Williams, Yee,