Researchers have added puzzles to the Minecraft game to study behavior. Source: Matthew Tostevin/Reuters
When the American cognitive scientist Joshua Hartshorne researched how people around the world learn English, he had to have tens of thousands of people take a language test. He designed “What English?”, a grammar game that presented a series of difficult word problems and then guessed where in the world the player learned the language. Participants shared their results – accurate or inaccurate – on social media, creating a snowball effect in recruitment. The findings, based on data from nearly 670,000 people, revealed that there is a ‘critical period’ for second language learning that extends into adolescence1.
This kind of “gamification” is becoming a powerful research tool in fields that people study, including psychology, neuroscience, economics and behavioral economics. By making research fun, the approach can help experiments reach thousands or millions of participants. For example, experiments in a video game have shown that the layout of the city where a child lives determines their future navigation ability2. Data from a digital word search showed that people skilled in the game do not necessarily give better advice to those trying to learn3. And a dilemma game involving millions of people revealed that most people have reliable moral intuition4.
Gamification can help avoid the pitfalls of conventional lab experiments by enabling researchers to study diverse populations, conduct more sophisticated experiments, and observe human behavior in naturalistic environments. It can improve statistical power and reproducibility, making research more robust. Technical advances are making gamification cheaper and easier, and the COVID-19 pandemic has forced many laboratories to move their human experiments online. But despite these changes, most have not yet embraced the opportunities gamification offers.
To exploit the full potential of this approach, researchers must dispel misconceptions, develop new gamification technologies, improve access to existing technologies, and apply the methods to productive research questions. We are researchers in psychology, linguistics, developmental science, data science and music who have conducted our own gamified experiments. We think it’s time for science to take games seriously.
Games theory
Gamification motivates people to participate in experiments through scoring, competition, performance feedback and the opportunity to learn about themselves through play (see ‘Participate in gamified research’). In early forays in 2005 and 2008, cognitive neuropsychologist Laura Germine, now at Harvard Medical School in Boston, Massachusetts, adapted and posted psychological assessments to her citizen science website, TestMyBrain.org. Her work showed that self-selected samples can produce high-quality data even when the participants are not supervised or paid5. As gamification has evolved, it has united developmental psychology with computer science, web development and user experience research to create exciting, immersive encounters for participants.
Take part in gamified research
Gamification motivates people to participate in experiments through competition, fun and the opportunity to learn about ourselves. This may interest you : UConn’s new multidisciplinary master’s degree in computer science starts this fall, with a focus on ethics today. Here are some games to try:
Which English? A viral grammar test that tries to determine what ‘world English’ someone speaks, as part of language research.
Are you a super listener? A citizen science experiment in which participants try to discover musical harmonies in distorted tones.
Glyph: An online gaming applet to explore the shape of letters in the world’s writing systems.
Moral Machine: A platform for gathering a human perspective on moral decisions made by machine intelligence.
Visual Vocabulary: An Online Assessment of Vocabulary Knowledge Across the Lifespan.
Gamification can include turning experiments into custom games, embedding experiments into existing games, and extracting data from ongoing games. The popular game Wordle – in a way the world’s largest psycholinguistic experiment – has already led to research on topics such as optimization problems in active learning6 and the context in which people cheat7.
Reproducibility: expect less from the scientific article
Reproducibility: expect less from the scientific article
Participants in conventional laboratory studies of human behavior are often small in number and WEIRD (that is, from Western, educated, industrialized, wealthy, and democratic societies). This leads to results that are statistically inaccurate, irreproducible, or cannot be generalized to other groups.
The huge datasets made possible by gamified science can help answer questions about reproducibility and generalizability. For example, small studies had shown that a person’s experience of speaking a tonal language—one that uses pitch or tones to distinguish words, as in Mandarin—changes their ability to perceive musical pitch. However, such work was largely done in Mandarin or Cantonese. To explore tonal languages that are less commonly studied, Jingxuan Liu, now a graduate student at Columbia University in New York City, and one of us (S.A.M.) used data from a popular web-based quiz, “Test your Musical IQ,” to replicate these findings in half a million speakers of such languages, including Ewe, spoken in West Africa, and Burmese, used in Myanmar8.
Issues of reproducibility and generalizability are particularly acute for scientists working with hard-to-reach research populations, such as children. Gamified experiments have the potential to encourage children’s participation in settings such as schools or museums, rather than requiring a special trip to a lab, which only a few families have the time or interest to do. For example, one of us (B.L.) installed a ‘Let’s Draw!’ newsstand at the Children’s Discovery Museum of San Jose in California. Children visiting the museum were asked to draw different things, such as a watch or a tiger, and then play games to recognize each other’s drawings. Over the next 18 months, the kiosk collected more than 37,000 drawings from some 8,000 children between the ages of 2 and 10, creating the world’s largest corpus of children’s drawings and showing how object recognition changes with age9.
A newsstand at the Children’s Discovery Museum in San Jose, California, collected drawings from thousands of two- to ten-year-olds to show how drawing systematically changes with age. Source: Brian Long
Games played in the lab can also improve data robustness. Two of us (A.B.-L. and D.G.W.), working with psycholinguist Joe Toscano at Villanova University in Pennsylvania, wanted to study how humans communicate through prosody—the patterns of intonation, pauses, and rhythm that convey what information is important, as well as sarcasm and humour. The stilted conversations participants usually had in the lab limited these acoustic signals, and previous results on prosody have been inconsistent. We designed puzzles in the Minecraft game that players had to solve by interacting while immersed in the game. This produced more natural dialogue and higher quality data, showing that participants used speech intonation and duration to convey subtle information, such as whether a word was new in the conversation10.
Invest 5% of the research funds in making data reusable
Invest 5% of the research funds in making data reusable
Compared to laboratory research, online gamified experiments make it easier and cheaper to recruit diverse participants, including from groups that are underrepresented in scientific research. In a study published last year, a gamified experiment using infant-directed speech and song recruited hundreds of self-identified individuals across three gender categories and a wide range of household incomes and ethnicities across the United States11. More generally, translating a gamified experiment and disseminating it worldwide on the Internet can help to reduce the English-focused nature of behavioral science research12.
The thresholds for gamification are decreasing. Coding an experiment to run online used to require a lab to maintain and develop custom code. But the past few years have seen an explosion of free, open-source tools in mainstream programming languages, such as JavaScript and Python. One of the most widely used tools for basic human experimentation on the web, the jsPsych13 JavaScript library was launched in 2012 by Josh de Leeuw, a cognitive scientist at Vassar College in Poughkeepsie, New York. It has been used in nearly 1,000 published articles, according to Google Scholar.
The jsPsych tool provides researchers with a means to show their participants stimuli (text, images, audio, video); to ask questions about it; collect responses in various ways (multiple choice, free text, clicks or taps on an image); and to output data in a structured format. Most importantly, de Leeuw maintains an active user forum and GitHub repository where researchers can discuss questions and collaborate on the codebase. These and similar tools, such as lab.js, Open Sesame and psychTestR, make it very easy for researchers to create online behavioral experiments (see ‘Go into gamification’).
Get into gamification
Using simple tools and minor tweaks to the design of a gamified experiment can greatly increase engagement and increase the number of participants. This may interest you : Executive Order to Enhance Safeguards for United States Signal Intelligence Activities.
• Take advantage of front-end tools. Software, including jsPsych, psychTestR, lab.js, and OpenSesame, provides an entry point to web development and gamification. Common data collection frameworks, such as displaying a sequence of images and measuring the time it takes to press a button, are available by default through plugins. A researcher gamifying a lab experiment needs only this: a computer and an idea.
• Think about back-end tools. Larger-scale experiments require a cloud infrastructure that can distribute them, handle data collection, and scale dynamically so it doesn’t all fall apart when your experiment goes viral. Customizable options like Pushkin are available, although they currently require more technical expertise than front-end tools.
• Provide an incentive. These can be points, scores, or percentile ranks that allow participants to see how well they did, or to learn something about themselves. Pay participants a fair wage for long or strenuous tasks.
• Keep it short. Try to limit online games to less than 10 minutes. Some gamified experiments are even faster.
• Help it snowball on social media. Make it easy for participants to share their results on social media, especially as an image or graph. This attracts more participants.
• Use a hook. Try adding a storyline or other form of visual progression to keep participants engaged. In one experiment on auditory perception, participants gain points while performing listening tasks to help a bird avatar stay aloft. Nearly 200,000 people have participated so far.
• Image is important. Putting a little effort into choosing free images or paying to design images can help engage attendees.
• Run a pilot. Before making your experiment public, try it out on friends and collaborators. They can tell you what works and what could be improved.
• Plan and test. Make sure you have a data management plan for handling large amounts of data (such as MongoDB or Google Cloud Firestore) if your game goes viral.
• Collaborate and share. Developing a project with a gamification lab can benefit everyone. As you develop new functionality, add it to collaborative codebases, as is done with jsPsych plugins.
Ups and downs
Gamified experiments have obvious weaknesses. Many scientists are used to having complete control over their laboratory environment: they can directly observe the behavior of participants during experiments and check whether people are who they say they are. On the same subject : Where the 2022 news was (mostly) good: The best science papers of the year. Critics may fear losing this control, or fear that people won’t fully participate in the tests or distort the results by faking their identities, completing games multiple times, or participating maliciously using Internet bots.
This criticism can be partly dispelled. Gamified studies have the potential to engage participants better than lab-based studies because they are intrinsically motivating. Indeed, many psychologists have observed all kinds of detachment in lab experiments, such as participants watching social media or even taking a nap, because tasks are often boring and poorly paid. The fact that many gamified experiments have gone viral on YouTube, Reddit, and other social media platforms is a proof of concept that participants can become deeply involved in research — perhaps more so than conventional approaches, although we need more data to be sure. are.
Creative approaches to experimental design and data analysis can also give researchers more control over gamified science than expected. For example, in a music study aimed at children, cognitive scientist Courtney Hilton of the University of Auckland, New Zealand, and colleagues (including S.A.M.) studied when participants played a game and compared the distribution of local times with the times adults played. The participants were less inclined to play at night, implying that children were indeed more likely to sleep at those times14. Concerns that participants may be bots can be allayed by comparing the timing of recruitment spikes to known events, such as a social media post promoting a game, and by using manipulation checks, which are designed to participant is easy to answer, but difficult for a scammer or bot. In the end, however, some research questions are better suited to the lab. The less controlled environments used in web-based gamified science can and should complement that of laboratory work.
The Wordle game has enabled research into the contexts in which people cheat. Source: Claire Welsh/Nature
Another valid criticism is that gamified science, when delivered online, excludes people who don’t have smartphones, computers and internet access. But it still offers an improvement over the status quo. Under-resourced communities are already at risk of being excluded from laboratory experiments because laboratories are concentrated in a small number of places and participants must spend time and money to visit them. Smartphones, on the other hand, are becoming ubiquitous worldwide. According to a 2021 study from the Pew Research Center in Washington DC, three quarters of adults in the United States with a household income of less than $30,000 a year own a smartphone. The fact that people from lower-income communities are more likely to connect to the internet through smartphones than desktop computers means it’s a high priority for researchers to develop mobile-friendly experiments that work when internet connections are slow or patchy.
Gamified experiments, especially large online experiments, must also address copyright, privacy, data storage, and data accessibility issues. Data protection and privacy laws vary from country to country, and some studies collect information (such as that related to gender or income) that falls under protected categories. To preserve privacy and avoid legal issues, researchers may avoid collecting personally identifiable information, such as IP addresses or other identifying information.
If data is collected by a third party in an existing game, the information may be proprietary. A data use agreement is necessary for research and to avoid conflicts of interest. Many universities have legal and ethics teams that can advise on gamified experiments, just as they do for lab experiments. For example, while gamifying an experiment with published musical recordings, one of us (S.A.M.) received helpful guidance on US copyright law from Harvard lawyers15.
Four future steps
For gamification to reach its full potential, four steps must be completed. First, researchers already in the community need to coordinate their development activities to broaden access to technology and share technical skills. Labs should make their software publicly available so novice scientists don’t have to reinvent the wheel. While open science practices are becoming more widespread, thanks to platforms such as GitHub, Zenodo, and the Open Science Framework, they are not yet ubiquitous. The community would benefit from pooling resources to create centralized technology development tools and best practices, as well as holding conferences and workshops aimed at increasing access to gamification and citizen science expertise.
Everyone should decide how their digital data is used, not just technology companies
Everyone should decide how their digital data is used, not just technology companies
Second, existing tools need to be improved. Many researchers use commercial software such as Qualtrics or SurveyMonkey to conduct online surveys, but no analog software is available for free for gamified experiments. Front-end tools like jsPsych are generally not yet optimized for mobile-friendly experiments. Back-end tools accessible to scientists without substantial technical expertise are uncommon, but these would jump-start scientists to produce large-scale gamified experiments, which are still relatively rare. The scientific community should work to develop these resources, and funders around the world should make it a priority to support these efforts. The US National Science Foundation has already called for digital infrastructure projects as part of efforts to promote reproducibility (see go.nature.com/3xi6bae).
Third, scientists should seek occupations outside of research to broaden their technical toolkits. Gamified experiments that generate a lot of data could benefit from using software and practices that are standard in the technology industry but are still underused in science. These include version control using Git (a way to systematically track different versions of code) and storing data in relational databases (such as those using SQL, a database management language ubiquitous in tech companies). Research labs should consider consulting or recruiting artists, graphic designers, educators, and others from a variety of fields who can increase the robustness of experiments by improving game design. And hiring or contracting dedicated software developers ensures that technical skills aren’t lost when graduate students and postdocs leave a lab group. Financiers should support these essential positions.
Finally, the community needs to improve the way it dispels inaccurate impressions about gamification, which naturally risk sounding like a distraction rather than a serious investigative tool. We think the best way to do this is through science itself. If gamified research continues to enable substantial scientific discoveries, the results will speak for themselves. Scientists should try gamification on their most creative, pressing and exciting research questions.
It is up to the community to ensure that these experiments are credible and valuable. Researchers can do this by designing studies that attract deep public engagement, using robust data management practices, applying analytical strategies that demonstrate the validity of gamified data, and sharing data for peers to examine. Editors and reviewers can support these efforts by encouraging the publication of high-quality experiments.
In 1986, the sociologist Lee Sproull suggested that researchers consider “a new data collection tool — electronic mail.” Decades after her work on what is arguably the first web-based research16, data collection on the internet is powerful and routine. Gamified science has not yet reached such widespread acceptance, but we think it can, should and will.
References
Hartshorne, J. K., Tenenbaum, J. B. & Pinker, S. Cognition 177, 263-277 (2018).
Article
PubMed
Google scholar
Coutrot, A. et al. Nature 604, 104-110 (2022).
Article
PubMed
Google scholar
Levari, D.E., Gilbert, D.T. & Wilson, T. D. Psychol. Science. 33, 685-698 (2022).
Article
PubMed
Google scholar
Awad, E. et al. Nature 563, 59-64 (2018).
Article
PubMed
Google scholar
Germine, L. et al. psychon. Taurus. 19, 847-857 (2012).
Article
PubMed
Google scholar
Brown, K. A. Npj Comput. matter. 8, 97 (2022).
Article
Google scholar
Wormley, A. S. & Cohen, A. B. Perspect. Psychol. Science. https://doi.org/10.1177/1745691622111375 (2022).
What makes gamification successful?
Article
What makes gamification effective?
Google scholar
What are three elements of gamification?
Liu, J., Hilton, C.B., Bergelson, E. & Mehr, S.A. Preprint on bioRxiv https://doi.org/10.1101/2021.10.18.464888 (2021).
How successful is gamification?
Long, B., Fan, J. E., Huey, H. Chai, Z. & Frank, MC Preprint on PsyArXiv https://doi.org/10.31234/osf.io/5yv7x (2021).
What are the benefits of game based learning?
Buxó-Lugo, A., Toscano, JC & Watson, D.G. Trading process. 55, 305-323 (2018).
- Article
- PubMed
- Google scholar
- Hilton, C.B. et al. Nature buzz. Behavior 6, 1545-1556 (2022).
- Article
PubMed
Is game-based learning more effective?
How does game-based learning help students learn?
Google scholar
What is one benefit of using games in the classroom?
Blasi, D.E., Henrich, J., Adamou, E., Kemmerer, D. & Majid, A. Trends Cogn. Science. 26, 1153-1170 (2022).
Why does gamification matter?
Article
PubMed
What is the impact of gamification?
Why is gamification so important?
Google scholar
How important is gamification in today’s world?
de Leeuw, JR Behav. Res. Methods 47, 1–12 (2015).
How many components of behavioral science are there?
Article
What are the 4 components of behavior?
PubMed
What are the three branches of behavioral science?
