Our research on citizen science has been funded by three earlier NSF awards. You can see the descriptions linked below.
This completed project was a two-phase theory-based study of virtual organizations that enable massive virtual collaboration in scientific research. The virtual organizations studied have a core of scientists and project leaders coordinating the work of a larger number of volunteer contributors, a format called citizen science. The project was directed at advancing the understanding of what constitutes effective citizen science virtual organizations and under what conditions citizen science virtual organizations can enable and enhance scientific and education production and innovation. The study was theoretically grounded in small group theory and rooted empirically in a survey of and case studies in citizen science projects. A survey was used to develop a typology of citizen science projects, illuminating the important dimensions of this form. The case studies identified key lever points in work design for enabling citizen science virtual organizations to involve distributed, diverse volunteers in producing large-scale, high quality, valued scientific research in an organizationally sustainable fashion.
Publications from the grant are listed below.
<p>Crowdsourcing work with high levels of coupling between tasks poses challenges for coordination. This paper presents a study of two online citizen science projects that involved volunteers in such tasks: not just analyzing bulk data but also interpreting data and writing a paper for publication in one project and identifying new classes of data in the other. However, extending the reach of citizen science adds tasks with more dependencies, which calls for more elaborate coordination mechanisms but the relationship between the project and volunteers limits how work can be coordinated. Contrariwise, a mismatch between dependencies and available coordination mechanisms can be expected to lead to performance problems. The results of the study offer recommendations for design of citizen science projects for advanced tasks.</p>
<p>This paper develops an organization design-oriented conceptual model of scientific knowledge production through citizen science virtual organizations. Citizen science is a form of organization design for collaborative scientific research involving scientists and volunteers, for which Internet-based modes of participation enable massive virtual collaboration by thousands of members of the public. The conceptual model provides an example of a theory development process and discusses its application to an exploratory study. The paper contributes a multi-level process model for organizing investigation into the impact of design on this form of scientific knowledge production.</p>
<p>Citizen science is a form of research collaboration involving members of the public in scientific research projects to address real-world problems. Often organized as a virtual collaboration, these projects are a type of open movement, with collective goals addressed through open participation in research tasks. Existing typologies of citizen science projects focus primarily on the structure of participation, paying little attention to the organizational and macrostructural properties that are important to designing and managing effective projects and technologies. By examining a variety of project characteristics, we identified five types—Action, Conservation, Investigation, Virtual, and Education—that differ in primary project goals and the importance of physical environment to participation.</p>
<p>Citizen science is a form of research collaboration involving members of the public in scientific research projects to address real-world problems. Often organized as a virtual collaboration, these projects are a type of open movement, with collective goals addressed through open participation in research tasks. We conducted a survey of citizen science projects to elicit multiple aspects of project design and operation. We then clustered projects based on the tasks performed by participants and on the project’s stated goals. The clustering results group projects that show similarities along other dimensions, suggesting useful divisions of the projects.</p>
<p>This poster presents initial findings from a dissertation pilot study on a citizen science project involving the public with scientists in collaborative research. The goal for the pilot study was familiarity with the contextual factors that influence citizen science project design, and in turn, observing how the design choices contribute to the project's knowledge creation and participation outcomes. The initial results highlight an unexpected form of `middle-out' organizing that challenges assumptions about top-down and bottom-up organizing, as the location of the top and bottom are clearly a matter of perspective in inter-organizational partnerships.</p>
<p>Citizen science has seen enormous growth in recent years, in part due to the influence of the Internet, and a corresponding growth in interest. However, the few stand-out examples that have received attention from media and researchers are not representative of the diversity of the field as a whole, and therefore may not be the best models for those seeking to study or start a citizen science project. In this work, we present the results of a survey of citizen science project leaders, identifying sub-groups of project types according to a variety of features related to project design and management, including funding sources, goals, participant activities, data quality processes, and social interaction. These combined features highlight the diversity of citizen science, providing an overview of the breadth of the phenomenon and laying a foundation for comparison between citizen science projects and to other online communities.</p>
<p>Citizen science creates a nexus between science and education that, when coupled with emerging technologies, expands the frontiers of ecological research and public engagement. Using representative technologies and other examples, we examine the future of citizen science in terms of its research processes, program and participant cultures, and scientific communities. Future citizen-science projects will likely be influenced by sociocultural issues related to new technologies and will continue to face practical programmatic challenges. We foresee networked, open science and the use of online computer/video gaming as important tools to engage non-traditional audiences, and offer recommendations to help prepare project managers for impending challenges. A more formalized citizen-science enterprise, complete with networked organizations, associations, journals, and cyberinfrastructure, will advance scientific research, including ecology, and further public education.</p>
The SOCS project (NSF grant 09-68470) investigates the capabilities and potential of social computational systems (SoCS) in the context of citizen science. Citizen science projects are a form of social-computational system. Whether it be volunteers playing a role in massive, distributed sensing networks exploring the migration of birds, or applying their unique human perceptual skills to searching the skies, human motivation and performance is fundamental to system performance. However, undertaking science through a social computational system brings unique challenges. To understand and address these challenges, this proposal presents a three-phase study of SoCS to support scientific research, grounded in group theory and rooted empirically in case studies and action research. More specifically, the proposal includes case studies of several citizen science projects to establish the nature of the SoCS currently in use, development of SoCS to support different kinds of citizen science projects and evaluation of the impacts of these systems on the outputs and processes of the projects. The system development for this project has resulted in a set of serious games to motivate volunteer participants to work on the classification of images of biological species. The systems can be seen and played at http://citizensort.org/. Research Professor Jun Wang acted as replacement PI on this project.
Some key publications from the project are listed below.
<p>Forgotten Island, a citizen science video game, is part of an NSF-funded design science research project, Citizen Sort. It is a mechanism to help life scientists classify photographs of living things and a research tool to help HCI and information science scholars explore storytelling, engagement, and the quality of citizenproduced data in the context of citizen science.</p>
<p>Two key problems for crowd-sourcing systems are motivating contributions from participants and ensuring the quality of these contributions. Games have been suggested as a motivational approach to encourage contribution, but attracting participation through game play rather than intrinsic interest raises concerns about the quality of the contributions provided. These concerns are particularly important in the context of citizen science projects, when the contributions are data to be used for scientific research. To assess the validity of concerns about the effects of gaming on data quality, we compare the quality of data obtained from two citizen science games, one a “gamified” version of a species classification task and one a fantasy game that used the classification task only as a way to advance in the game play. Surprisingly, though we did observe cheating in the fantasy game, data quality (i.e., classification accuracy) from participants in the two games was not significantly different. As well, data from short-time contributors was also at a usable level of accuracy. Finally, learning did not seem to affect data quality in our context. These findings suggest that various approaches to gamification can be useful for motivating contributions to citizen science projects.</p>
This SOCS project (NSF 12-11071) examined strategies for dealing with the flood of digital data that confronts researchers. New techniques, tools and strategies for dealing with massive data sets, whether they consist of vast numbers of base-pair sequences or terabytes of data from all-sky astronomical surveys, present an opportunity to establish a 'fourth paradigm' of scientific discovery, but the task is not easy. In many areas of research, the relentless growth of data sets has led to the adoption of increasingly automated and unsupervised methods of classification. In many cases, this has led to degradation in classification quality, with machine learning and computer vision unable to replicate the successes of human pattern recognition. The growth of citizen science on the web has provided a temporary solution to this problem; in particular, the highly successful Galaxy Zoo (Lintott et al. 2008, 2011) and the Zooniverse projects (Smith et al. 2011, Fischer et al. 2011, Davis et al. 2011), which have grown from it and which this proposal takes as its starting point, have demonstrated that it is possible to recruit hundreds of thousands of volunteers to make an authentic contribution to results, boosting human analysis through the collective wisdom of a crowd of classifiers. However, human classifiers alone will not be able to cope with expected flood of data from future scientific instruments. The project was to develop a next-generation socio-computational citizen science platform that combines the efforts of human classifiers with those of computational systems to maximize the efficiency with which human attention can be used. We recognize that to do so requires a thorough understanding of human motivation and learning in this context, and knowledge of how the proposed system will affect these. The project was a partnership between computer and social scientists, addressing research problems both in automated data analysis and social science through systems implementation, alongside field research and experiments with project participants. This project was conducted in collaboration with the Adler Planetarium (Arfon Smith, PI). Carsten Østerlund served as replacement PI at Syracuse University.
Some key publications from the project are listed below.
<p>This study examines the relative efficacy of citizen science recruitment messages appealing to four motivations that were derived from previous research on motives for participation in citizen-science projects. We report on an experiment (N=36,513) that compared the response to email messages designed to appeal to these four motives for participation. We found that the messages appealing to the possibility of contributing to science and learning about science attracted more attention than did one about helping scientists but that one about helping scientists generated more initial contributions. Overall, the message about contributing to science resulted in the largest volume of contributions and joining a community, the lowest. The results should be informative to those managing citizen-science projects.</p>
<p>How online community members learn to become valuable contributors constitutes a long-standing concern of Community & Technology researchers. The literature tends to highlight participants’ access to practice, feedback from experienced members, and relationship building. However, not all crowdsourcing environments offer participants opportunities for access, feedback, and relationship building (e.g., Citizen Science). We study how volunteers learn to participate in a citizen science project, Planet Hunters, through participant observation, interviews, and trace ethnography. Drawing on Sørensen’s sociomaterial theories of presence, we extend the notion of situated learning to include several modes of learning. The empirical findings suggest that volunteers in citizen science engage more than one form of access to practice, feedback, and relationship building. Communal relations characterize only one form of learning. Equally important to their learning are authority–subject and agent-centered forms of access, feedback, and relationship building.</p>
<p>We propose a set of methodological principles and strategies for the use of trace data, i.e., data capturing performances carried out on or via information systems, often at a fine level of detail. Trace data comes with a number of methodological and theoretical challenges associated with the inseparable nature of the social and material. Drawing on Haraway and Barad’s distinctions among refraction, reflection and diffraction, we compare three approaches to trace data analysis. We argue that a diffractive methodology allows us to explore how trace data are not given but created though construction of a research apparatus to study trace data. By focusing on the diffractive ways in which traces ripple through an apparatus, it is possible to explore some of the taken-for-granted, invisible dynamics of sociomateriality. Equally, important this approach allows us to describe what and when distinctions within entwined phenomena emerge in the research process. Empirically, we illustrate the guiding principles and strategies by analyzing trace data from Gravity Spy, a crowdsourced citizen science project on Zooniverse. We conclude by suggesting that a diffractive methodology may help us draw together quantitative and qualitative research practices in new and productive ways that also raises interesting design questions.</p>
<p>Citizen science projects rely on the voluntary contribution of nonscientists to take part in scientific research projects. Projects taking place exclusively over the Internet face significant challenges, chief among them is the attracting and keeping the critical mass of volunteers needed to conduct the work outlined by the science team. The extent to which platforms can design experiences that positively influence volunteers’ motivation can help address the contribution challenges. Consequently, project organizers need to develop strategies to attract new participants and keep existing ones. One strategy to encourage participation is implementing features, which re-enforce motives known to change people’s attitudes towards contributing positively. The literature in psychology noted that novelty is an attribute of objects and environments that occasion curiosity in humans leading to exploratory behaviors, e.g., prolonged engagement with the object or environment. This dissertation described the design, implementation, and evaluation of an experiment conducted in three online citizen science projects. Volunteers received novelty cues when they classified data objects that no other volunteer had previously seen. The hypothesis was that exposure to novelty cues while classifying data positively influences motivational attitudes leading to increased engagement in the classification task and increased retention. The experiments resulted in mixed results. In some projects, novelty cues were universally salient, and in other projects, novelty cues had no significant impact on volunteers’ contribution behaviors. The results, while mixed, are promising since differences in the observed behaviors arise because of individual personality differences and the unique attributes found in each project setting. This research contributes to empirically grounded studies on motivation in citizen science with analyses that produce new insights and questions into the functioning of novelty and its impact on volunteers’ behaviors.</p>
<p>Researchers studying user behaviors in online communities often conduct analyses of events collected in system logs, e.g., a system’s record of a comment post or of a contribution. However, analysis of user behaviors is more difficult if users make contributions without being logged-in (i.e., anonymously). Since a user’s account will not be associated with contributions that user makes anonymously, conclusions about user behaviors that look only at attributed actions might not account for a user’s full experience. To understand the impacts of anonymous contributions on research, we conducted an analysis of system logs containing anonymous activities in two online citizen science projects. By linking anonymous events with user IDs we found that (1) many users contribute anonymously, though with varied patterns of contribution; and (2) including anonymous activities alter conclusions made about users’ experience with the project. These results suggest that researchers of human behaviors in online communities should consider the possible impacts of anonymous interaction on their ability to draw conclusions about user behaviors in these settings.</p>
<p>The explosion of data available from online systems such as social media is creating a wealth of trace data, that is, data that record evidence of human activity. The volume of data available offers great potential to advance social and behavioural science research. However, the data are of a very different kind than more conventional social and behavioural science data, posing challenges to use. This paper adopts a data framework from Earth Observation science and applies it to trace data to identify possible issues in analyzing trace data. Application of the framework also reveals issues for sharing and reusing data.</p>
<p>The paper explores the motivations of volunteers in a large crowd sourcing project and contributes to our understanding of the motivational factors that lead to deeper engagement beyond initial participation. Drawing on the theory of legitimate peripheral participation (LPP) and the literature on motivation in crowd sourcing, we analyze interview and trace data from a large citizen science project. The analyses identify ways in which the technical features of the projects may serve as motivational factors leading participants towards sustained participation. The results suggest volunteers first engage in activities to support knowledge acquisition and later share knowledge with other volunteers and finally increase participation in Talk through a punctuated process of role discovery.</p>
<p>Making the traces of user participation in primary activities visible in online crowdsourced initiatives has been shown to help new users understand the norms of participation but participants do not always have access to others’ work. Through a combination of virtual and trace ethnography we explore how new users in two online citizen science projects engage other traces of activity as a way of compensating. Merging the theory of legitimate peripheral participation with Erickson and Kellogg’s theory of social translucence we introduce the concept of practice proxies; traces of user activities in online environment that act as resources to orient newcomers towards the norms of practice. Our findings suggest that newcomers seek out practice proxies in the social features of the projects that highlight contextualized and specific characteristics of primary work practice.</p>
<p>User-generated content (UGC) projects involve large numbers of mostly unpaid contributors collaborating to create content. Motivation for such contributions has been an active area of research. In prior research, motivation for contribution to UGC has been considered a single, static and individual phenomenon. In this paper, we argue that it is instead three separate but interrelated phenomena. Using the theory of helping behaviour as a framework and integrating social movement theory, we propose a stage theory that distinguishes three separate sets (initial, sustained and meta) of motivations for participation in UGC. We test this theory using a data set from a Wikimedia Editor Survey (Wikimedia Foundation, 2011). The results suggest several opportunities for further refinement of the theory but provide support for the main hypothesis, that different stages of contribution have distinct motives. The theory has implications for both researchers and practitioners who manage UGC projects.</p>
<p>Research on newcomer roles in peer production sites (e.g., Wikipedia) is characterized by a broad and relatively well-articulated set of functionally and culturally recognizable roles. But not all communities come with well-defined roles that newcomers can aspire to occupy. The present study explores activity clusters newcomers create when faced with few recognizable roles to fill and limited access to other participants’ work that serves as an exemplar. Drawing on a mixed method research design, we present findings from an analysis of 1,687 newcomers’ sessions in a citizen science project. Combining session- and individual-level analysis produced three findings (1) newcomers activities manifest a diverse range of session types; (2) Newcomers toggle between light work sessions and more involved types of production or community engagement; (3) an interesting relationship between high-level contributors who do a lot of work but little talk and a small group that does a lot of talk but less work. The former group draws heavily on posts contributed by the latter group. Identifying shifts and regularities in contribution facilitate improved mechanisms for engaging participants and the design of online citizen science communities.</p>
<p>In this paper, we describe the results of an online field experiment examining the impacts of messaging about task novelty on the volume of volunteers’ contributions to an online citizen science project. Encouraging volunteers to provide a little more content as they work is an attractive strategy to increase the community’s output. Prior research found that an important motivation for participation in online citizen science is the wonder of being the first person to observe a particular image. To appeal to this motivation, a pop-up message was added to an online citizen science project that alerted volunteers when they were the first to annotate a particular image. Our analysis reveals that new volunteers who saw these messages increased the volume of annotations they contributed. The results of our study suggest an additional strategy to increase the amount of work volunteers contribute to online communities and citizen science projects specifically.</p>