[Literature] The mapping principle and a research framework for virtual worlds

This paper was written by Dmitri Williams in 2008 (no date is written in the white paper, but this post at Terra Nova is dated from November 2008. Quotes and [comments].

Academic research has taken two distinct approaches to virtual worlds. [Academic research in sociology only! CS/SWeng, humanities, law or psychology academics follow other approaches] The first is understanding the virtual worlds populations and behaviors, and confronting them with traditional computer-mediated communiation. Examples are determining who play and why, how people perceive each other, how they collaborate, etc. The second consists of using situations that happen in VW to understand RL behaviors. For instance, virtual economies or the spreadth of a virtual epidemy are of particular interest. In this second approach, researchers can use VW as a petri-dish to conduct controlled experiments. Mapping is the extent to which human behaviors occur in virtual spaces in the same way they occur in real spaces. But no one knows whether these behaviors map or not. One of the reasons to be suspicious about mapping is that game risks and rewards (eg pain and death) really often do not map to reality.

Validity and generalizability are two key necessary conditions to establish mapping. Validity is the extent ot which an instrument measures what it is intended to measure, the instrument being here VW. Face validity is whether the measure appears to measure the phenomenon in question For instance, a violent MMO [such as 2Moons which is presumably violent] is better than Club Penguin to study violence in video games. Concurrent validity is whether the current measures are coherent with other measures of the same phenomenon. In the virtual case, this presents a new challenge: the virtual wold is often self-contained and therefore using a measure of the VW GDP to detect inflation might not apply very well. [Whether the GDP of Everquest can be compared to Russia's is a different question...] Predictive validity tests whether a measure relates to other measures. Its meaning is close to external validity: a result found within a virtual world that does not exist outside of it is nonsense. For instance, virtual inflation may not have the same consequence for players than RL inflation to consumers, and players' behaviors can not be mapped to consumers'. The key to external validity would be whether the people involved perceived the risks and costs to be as powerful as those experienced offline.. The biggest challenge for generalizability is Contextual and social architecture factors (see table below). [Nate Combs wrote in a terranova article, Virus!, that player behaviors in VW can be totally different (sometimes even the opposite of what is expected IRL) than their RL behaviors: when can you trust the players in a game? After all, to some of those spreading the virus the plague turned out to be much-about-fun and without real consequence those on the receiving end could shrug if off]

The framework, resting within the tradition of computer mediated communication (CMC) research, aims at answering the mapping issues. The framework relies on the four tables below.

Group size
Individual	Dyads	Small groups	Large groups	Communities	Societies

Traditional controls and independent variables
Psychological profile	Motivations	Demographics	Communications medium	Network-level variables

Contextual and social architecture factors
World size	Persistence	Competitive vs. Collaborative	Role play	Sandbox vs. linear	Representation	Interaction affordances	Costs of a behavior

Directionality
Online to offline	Offline to online	Endogenous

A case study for the framework is provided as an example: The Proteus Effect series of studies conducted by Yee et al. Shortly, the studies report that some RL behaviors such as social distance, eye contact or the fact that the respect you give to your interlocutor is linked to his/her height are imported inside VW. Because the mapping of the results is not automatically applicable to any population in any context, this study is considered as an important baseline, or starting point.

• The studies focused on dyads, but it is not sure that the results can apply to larger groups. Also, the studies involved both human- and computer-based agents, which is different than a total human-to-human environment.
• As for controls and independent variables, the tests were not focused on the profiles of the users because the intent of the experiments was to establish the presence of the phenomenon, not to explore the nuances right away. Only gender was examined, but the authors could have looked at personality-based differences, whether the use of voice would change the outcomes or the position of avatars within some social hierarchy.
• The Proteus studies were conducted between human-looking avatars. The results might have been different between penguins avatars in Club Penguin, between Orcs and Taurens [, between two gunmen in a MMOFPS such as Global Agenda, or even between two spaceships in EVE Online!]. The presence of game-based tasks (hunting a dragon or chating in a virtual bar) performed during the study may influence the results as well.
• The directionality was only real to virtual.

Other considerations
Studies of different scale rely on different methods and suffer typical methodological issues.

	Large-scale studies	Small-scale studies
Rely on	estimates, surveys or sampling	controlled experiment, participant observation or ethnography
Typically lack	internal validity - the possibility to determine relationships	external validity - the possibility to generalize results

In many MMOG, players choose a realm to play in - most of the time, there are a few thousands of players per realm. Williams remarked that there may be no totally independent draws from server to server [even if these servers are both PvP or PvE or PvPvE or ...].
Non-obstrusive logging methods [like the ones used by Ducheneaut] avoid any Hawthorne effects (subjects being aware of the researcher). However, the researcher has little chance to have an opportunity of control because VW are controlled by companies. As written in page 8, researchers can address validity issues if they can make the virtual world as similar to the real one as possible with regards to the phenomenon in question. [But are developers going to accept that? No, the game should be fun, not realistic!]
Even if the target populations happen to be virtual, researchers should keep the same level of ethics in their VW studies as in RL studies.
Finally, it is the norm that results from new methodologies will be ignored or attacked, especially if these results challenge some existing theory. Researchers have the responsibility to tackle flaws in methodology (with this framework, for instance) and be conservative with their results, otherwise journalists or novice researchers may make large and irresponsible claims to a public which may not know better.