Academic research has taken two distinct approaches to virtual worlds. [Academic research in sociology only! CS/SWeng, humanities, law or psychology academics follow other approaches] The first is understanding the virtual worlds populations and behaviors, and confronting them with traditional computer-mediated communiation. Examples are determining who play and why, how people perceive each other, how they collaborate, etc. The second consists of using situations that happen in VW to understand RL behaviors. For instance, virtual economies or the spreadth of a virtual epidemy are of particular interest. In this second approach, researchers can use VW as a
petri-dish to conduct controlled experiments.
Mapping is the extent to which human behaviors occur in virtual spaces in the same way they occur in real spaces. But
no one knows whether these behaviors map or not. One of the
reasons to be suspicious about mapping is that game
risks and rewards (eg pain and death) really often do not map to reality.
Validity and generalizability are
two key necessary conditions to establish mapping. Validity is
the extent ot which an instrument measures what it is intended to measure, the instrument being here VW. Face validity is
whether the measure appears to measure the phenomenon in question For instance, a violent MMO [such as 2Moons which is presumably violent] is better than Club Penguin to study violence in video games. Concurrent validity is whether the current measures are coherent with other measures of the same phenomenon.
In the virtual case, this presents a new challenge: the virtual wold is
often self-contained and therefore using a measure of the VW GDP to detect inflation might not apply very well. [Whether the GDP of Everquest can be compared to Russia's is a different question...] Predictive validity
tests whether a measure relates to other measures. Its meaning is close to external validity:
a result found within a virtual world that does not exist outside of it is nonsense. For instance, virtual inflation may not have the same consequence for players than RL inflation to consumers, and players' behaviors can not be mapped to consumers'.
The key to external validity would be whether the people involved perceived the risks and costs to be as powerful as those experienced offline.. The
biggest challenge for generalizability is
Contextual and social architecture factors (see table below). [Nate Combs wrote in a terranova article, Virus!, that player behaviors in VW can be totally different (sometimes even the opposite of what is expected IRL) than their RL behaviors:
when can you trust the players in a game? After all, to some of those spreading the virus the plague turned out to be much-about-fun and without real consequence those on the receiving end could shrug if off]
resting within the tradition of computer mediated communication (CMC) research, aims at answering the mapping issues. The framework relies on the four tables below.
|Individual||Dyads||Small groups||Large groups||Communities||Societies|
|Traditional controls and independent variables|
|Psychological profile||Motivations||Demographics||Communications medium||Network-level variables|
|Contextual and social architecture factors|
|World size||Persistence||Competitive vs. Collaborative||Role play||Sandbox vs. linear||Representation||Interaction affordances||Costs of a behavior|
|Online to offline||Offline to online||Endogenous|
A case study for the framework is provided as an example: The Proteus Effect series of studies conducted by Yee et al. Shortly, the studies report that some RL behaviors such as social distance, eye contact or the fact that the respect you give to your interlocutor is linked to his/her height are imported inside VW. Because the mapping of the results is not automatically applicable to any population in any context, this study is considered as
an important baseline, or starting point.
- The studies focused on dyads, but it is not sure that the results can apply to larger groups. Also, the studies involved both human- and computer-based agents, which is different than a total human-to-human environment.
- As for controls and independent variables,
the tests were not focused on the profiles of the usersbecause
the intent of the experiments was to establish the presence of the phenomenon, not to explore the nuances right away. Only gender was examined, but the authors could have looked at
personality-based differences, whether the
use of voice would change the outcomesor the
position of avatars within some social hierarchy.
- The Proteus studies were conducted between human-looking avatars. The results might have been different between penguins avatars in Club Penguin, between Orcs and Taurens [, between two gunmen in a MMOFPS such as Global Agenda, or even between two spaceships in EVE Online!]. The presence of game-based tasks (hunting a dragon or chating in a virtual bar) performed during the study may influence the results as well.
- The directionality was only real to virtual.
Studies of different scale rely on different methods and suffer typical methodological issues.
|Large-scale studies||Small-scale studies|
|Rely on||estimates, surveys or sampling||controlled experiment, participant observation or ethnography|
|Typically lack||internal validity - the possibility to determine relationships||external validity - the possibility to generalize results|
In many MMOG, players choose a realm to play in - most of the time, there are a few thousands of players per realm. Williams remarked that
there may be no totally independent draws from server to server [even if these servers are both PvP or PvE or PvPvE or ...].
Non-obstrusive logging methods [like the ones used by Ducheneaut] avoid any Hawthorne effects (subjects being aware of the researcher). However, the researcher has little chance to have an
opportunity of control because VW are controlled by companies. As written in page 8, researchers can address validity issues if they can
make the virtual world as similar to the real one as possible with regards to the phenomenon in question. [But are developers going to accept that? No, the game should be fun, not realistic!]
Even if the target populations happen to be virtual, researchers should keep the same level of ethics in their VW studies as in RL studies.
Finally, it is the
norm that results from new methodologies
will be ignored or attacked, especially if these results
challenge some existing theory. Researchers have the responsibility to tackle flaws in methodology (with this framework, for instance) and be conservative with their results, otherwise
journalists or novice researchers may
make large and irresponsible claims to a public which may not know better.