%0 Conference Proceedings %B Information for a Better World: Normality, Virtuality, Physicality, Inclusivity: 18th International Conference, iConference %D 2023 %T Design principles for background knowledge to enhance learning in citizen science %A Kevin Crowston %A Corey Brian Jackson %A Isabella Corieri %A Carsten Østerlund %X

Citizen scientists make valuable contributions to science but need to learn about the data they are working with to be able to perform more advanced tasks. We present a set of design principles for identifying the kinds of background knowledge that are important to support learning at different stages of engagement, drawn from a study of how free/libre open source software developers are guided to create and use documents. Specifically, we suggest that newcomers require help understanding the purpose, form and content of the documents they engage with, while more advanced developers add understanding of information provenance and the boundaries, relevant participants and work processes. We apply those principles in two separate but related studies. In study 1, we analyze the background knowledge presented to volunteers in the Gravity Spy citizen-science project, mapping the resources to the framework and identifying kinds of knowledge that were not initially provided. In study 2, we use the principles proactively to develop design suggestions for Gravity Spy 2.0, which will involve volunteers in analyzing more diverse sources of data. This new project extends the application of the principles by seeking to use them to support understanding of the relationships between documents, not just the documents individually. We conclude by discussing future work, including a planned evaluation of Gravity Spy 2.0 that will provide a further test of the design principles.

%B Information for a Better World: Normality, Virtuality, Physicality, Inclusivity: 18th International Conference, iConference %C Barcelona, Spain and virtual %P 563–580 %G eng %R 10.1007/978-3-031-28032-0_43 %> https://citsci.syr.edu/sites/crowston.syr.edu/files/Design_Background_iConf.pdf %0 Journal Article %J Journal of the Association for Information Systems %D 2020 %T Building an apparatus: Refractive, reflective and diffractive readings of trace data %A Carsten Østerlund %A Kevin Crowston %A Corey Brian Jackson %X

We propose a set of methodological principles and strategies for the use of trace data, i.e., data capturing performances carried out on or via information systems, often at a fine level of detail. Trace data comes with a number of methodological and theoretical challenges associated with the inseparable nature of the social and material. Drawing on Haraway and Barad’s distinctions among refraction, reflection and diffraction, we compare three approaches to trace data analysis. We argue that a diffractive methodology allows us to explore how trace data are not given but created though construction of a research apparatus to study trace data. By focusing on the diffractive ways in which traces ripple through an apparatus, it is possible to explore some of the taken-for-granted, invisible dynamics of sociomateriality. Equally, important this approach allows us to describe what and when distinctions within entwined phenomena emerge in the research process. Empirically, we illustrate the guiding principles and strategies by analyzing trace data from Gravity Spy, a crowdsourced citizen science project on Zooniverse. We conclude by suggesting that a diffractive methodology may help us draw together quantitative and qualitative research practices in new and productive ways that also raises interesting design questions.

%B Journal of the Association for Information Systems %V 21 %P Article 10 %G eng %N 1 %R 10.17705/1jais.00590 %> https://citsci.syr.edu/sites/crowston.syr.edu/files/RA-JAIS-17-0130.R3.1_FIN%20to%20share.pdf %0 Journal Article %J IEEE Transactions on Learning Technologies %D 2020 %T Knowledge Tracing to Model Learning in Online Citizen Science Projects %A Kevin Crowston %A Carsten Østerlund %A Tae Kyoung Lee %A Corey Brian Jackson %A Mahboobeh Harandi %A Sarah Allen %A Sara Bahaadini %A Scott Coughlin %A Aggelos Katsaggelos %A Shane Larson %A Neda Rohani %A Joshua Smith %A Laura Trouille %A Michael Zevin %X

We present the design of a citizen science system that uses machine learning to guide the presentation of image classification tasks to newcomers to help them more quickly learn how to do the task while still contributing to the work of the project. A Bayesian model for tracking volunteer learning for training with tasks with uncertain outcomes is presented and fit to data from 12,986 volunteer contributors. The model can be used both to estimate the ability of volunteers and to decide the classification of an image. A simulation of the model applied to volunteer promotion and image retirement suggests that the model requires fewer classifications than the current system.

%B IEEE Transactions on Learning Technologies %V 13 %P 123-134 %G eng %6 1 %R 10.1109/TLT.2019.2936480 %> https://citsci.syr.edu/sites/crowston.syr.edu/files/transaction%20paper%20final%20figures%20in%20text.pdf %0 Unpublished Work %D 2020 %T Motivating Participation through Novelty Cues %A Corey Brian Jackson %X

The voluntary nature of participation in many open collaboration platforms means platform managers and system designers need to develop strategies to attract new participants and retain existing ones. Implementing features that positively impact users’ motivational states will help address challenges surrounding user recruitment and attrition. This paper reports on the design, implementation, and evaluation of one such feature – novelty cues. Prior studies have shown that novel stimuli attract and increase attention towards objects. Citizen science projects regularly ask volunteers to classify data that no other human has seen previously. An experiment was designed where volunteers were shown novelty cues. The goal was to evaluate the saliency of novelty in motivating participation among citizen science volunteers. The results showed that under most circumstances, novelty cues are effective mechanisms to increase user motivation; however, its effectiveness may be mediated by other factors such as existing system design and individual preferences for novelty.

%G eng %9 Unpublished working paper %> https://citsci.syr.edu/sites/crowston.syr.edu/files/Novelty_Motivator_2020.pdf %0 Journal Article %J Proceedings of the ACM on Human-Computer Interaction %D 2020 %T Shifting forms of Engagement: Volunteer Learning in Online Citizen Science %A Corey Brian Jackson %A Carsten Østerlund %A Mahboobeh Harandi %A Kevin Crowston %A Laura Trouille %X

Open collaboration platforms involve people in many tasks, from editing articles to analyzing datasets. To facilitate mastery of these practices, communities offer a number of learning resources, ranging from project-defined FAQs to individually-oriented search tools and communal discussion boards. However, it is not clear which project resources best support participant learning, overall and at different stages of engagement with the project. We draw on Sørensen’s framework of forms of presence to distinguish three forms of engagement with learning resources: authoritative, agent-centered and communal. We analyzed trace data from the GravitySpy citizen-science project using a mixed-effects logistic regression with volunteer performance as an outcome variable. The findings suggest that engagement with authoritative resources (e.g., those constructed by project organizers) facilitates performance initially. However, as tasks become more difficult, volunteers seek and benefit from engagement with their own agent-centered resources and community generated resources. These findings suggest a broader scope for the design of learning resources for online communities.

%B Proceedings of the ACM on Human-Computer Interaction %P 36 %G eng %N CSCW %R 10.1145/3392841 %> https://citsci.syr.edu/sites/crowston.syr.edu/files/3392841.pdf %0 Thesis %B School of Information Studies %D 2019 %T Characterizing Novelty as a Motivator in Online Citizen Science %A Corey Brian Jackson %X

Citizen science projects rely on the voluntary contribution of nonscientists to take part in scientific research projects. Projects taking place exclusively over the Internet face significant challenges, chief among them is the attracting and keeping the critical mass of volunteers needed to conduct the work outlined by the science team. The extent to which platforms can design experiences that positively influence volunteers’ motivation can help address the contribution challenges. Consequently, project organizers need to develop strategies to attract new participants and keep existing ones. One strategy to encourage participation is implementing features, which re-enforce motives known to change people’s attitudes towards contributing positively. The literature in psychology noted that novelty is an attribute of objects and environments that occasion curiosity in humans leading to exploratory behaviors, e.g., prolonged engagement with the object or environment. This dissertation described the design, implementation, and evaluation of an experiment conducted in three online citizen science projects. Volunteers received novelty cues when they classified data objects that no other volunteer had previously seen. The hypothesis was that exposure to novelty cues while classifying data positively influences motivational attitudes leading to increased engagement in the classification task and increased retention. The experiments resulted in mixed results. In some projects, novelty cues were universally salient, and in other projects, novelty cues had no significant impact on volunteers’ contribution behaviors. The results, while mixed, are promising since differences in the observed behaviors arise because of individual personality differences and the unique attributes found in each project setting. This research contributes to empirically grounded studies on motivation in citizen science with analyses that produce new insights and questions into the functioning of novelty and its impact on volunteers’ behaviors.

%B School of Information Studies %I Syracuse University %C Syracuse, NY, USA %G eng %U https://surface.syr.edu/etd/1046/ %9 PhD dissertation %0 Journal Article %J Physical Review D %D 2019 %T Classifying the unknown: Discovering novel gravitational-wave detector glitches using similarity learning %A Scott Coughlin %A Sara Bahaadini %A Neda Rohani %A Michael Zevin %A Patane, Oli %A Mahboobeh Harandi %A Corey Brian Jackson %A Noroozi, V. %A Sarah Allen %A Areeda, J. %A Coughlin, M. %A Ruiz, P. %A Berry, C. P. L. %A Kevin Crowston %A Aggelos Katsaggelos %A Andrew Lundgren %A Carsten Østerlund %A Joshua Smith %A Laura Trouille %A Vicky Kalogera %X

The observation of gravitational waves from compact binary coalescences by LIGO and Virgo has begun a new era in astronomy. A critical challenge in making detections is determining whether loud transient features in the data are caused by gravitational waves or by instrumental or environmental sources. The citizen-science project Gravity Spy has been demonstrated as an efficient infrastructure for classifying known types of noise transients (glitches) through a combination of data analysis performed by both citizen volunteers and machine learning. We present the next iteration of this project, using similarity indices to empower citizen scientists to create large data sets of unknown transients, which can then be used to facilitate supervised machine-learning characterization. This new evolution aims to alleviate a persistent challenge that plagues both citizen-science and instrumental detector work: the ability to build large samples of relatively rare events. Using two families of transient noise that appeared unexpectedly during LIGO's second observing run, we demonstrate the impact that the similarity indices could have had on finding these new glitch types in the Gravity Spy program.

%B Physical Review D %V 99 %P 082002 %G eng %N 8 %R 10.1103/PhysRevD.99.082002 %0 Conference Proceedings %B International Conference on Information Systems %D 2019 %T Linguistic adoption in online citizen science: A structurational perspective %A Corey Brian Jackson %A Carsten Østerlund %A Mahboobeh Harandi %A Dhruv Kharwar %A Kevin Crowston %X

For peer-production projects to be successful, members must develop a specific and common language that enables them to cooperate. We address the question of what factors affect the development of shared language in open peer production communities? Answering this question is important because we want the communities to be productive even when self-managed, which requires understanding how shared language emerges. We examine this question using a structurational lens in the setting of a citizen science project. Examining the use of words in the Gravity Spy citizen science project, we find that many words are reused and that most novel words that are introduced are not picked up, showing reproduction of structure. However, some novel words are used by others, showing an evolution of the structure. Participants with roles closer to the science are more likely to have their words reused, showing the mutually reinforcing nature of structures of signification, legitimation and domination.

%B International Conference on Information Systems %C Munich, Germany %G eng %U https://aisel.aisnet.org /icis2019/crowds_social/crowds_social/28/ %> https://citsci.syr.edu/sites/crowston.syr.edu/files/Linguistic%20Adoption%20%28ICIS%29%20final.pdf %0 Journal Article %J Proceedings of the ACM on Human-Computer Interaction %D 2018 %T Did they login? Patterns of anonymous contributions to online communities %A Corey Brian Jackson %A Kevin Crowston %A Carsten Østerlund %X

Researchers studying user behaviors in online communities often conduct analyses of events collected in system logs, e.g., a system’s record of a comment post or of a contribution. However, analysis of user behaviors is more difficult if users make contributions without being logged-in (i.e., anonymously). Since a user’s account will not be associated with contributions that user makes anonymously, conclusions about user behaviors that look only at attributed actions might not account for a user’s full experience. To understand the impacts of anonymous contributions on research, we conducted an analysis of system logs containing anonymous activities in two online citizen science projects. By linking anonymous events with user IDs we found that (1) many users contribute anonymously, though with varied patterns of contribution; and (2) including anonymous activities alter conclusions made about users’ experience with the project. These results suggest that researchers of human behaviors in online communities should consider the possible impacts of anonymous interaction on their ability to draw conclusions about user behaviors in these settings.

%B Proceedings of the ACM on Human-Computer Interaction %V 2 %P Article 77 %G eng %N CSCW %R 10.1145/3274346 %> https://citsci.syr.edu/sites/crowston.syr.edu/files/anonymous-contributions-cameraready.pdf %0 Journal Article %J Computer Supported Cooperative Work %D 2018 %T Folksonomies to support coordination and coordination of folksonomies %A Corey Brian Jackson %A Kevin Crowston %A Carsten Østerlund %A Mahboobeh Harandi %X

Members of highly-distributed groups in online production communities face challenges in achieving coordinated action. Existing CSCW research highlights the importance of shared language and artifacts when coordinating actions in such settings. To better understand how such shared language and artifacts are, not only a guide for, but also a result of collaborative work we examine the development of folksonomies (i.e., volunteer-generated classification schemes) to support coordinated action. Drawing on structuration theory, we conceptualize a folksonomy as an interpretive schema forming a structure of signification. Our study is set in the context of an online citizen-science project, Gravity Spy, in which volunteers label "glitches" (noise events recorded by a scientific instrument) to identify and name novel classes of glitches. Through a multi-method study combining virtual and trace ethnography, we analyze folksonomies and the work of labelling as mutually constitutive, giving folksonomies a dual role: an emergent folksonomy supports the volunteers in labelling images at the same time that the individual work of labelling images supports the development of a folksonomy. However, our analysis suggests that the lack of supporting norms and authoritative resources (structures of legitimation and domination) undermines the power of the folksonomy and so the ability of volunteers to coordinate their decisions about naming novel glitch classes. These results have implications design. If we hope to support the development of emergent folksonomies online production communities need to facilitate 1) tag gardening, a process of consolidating overlapping terms of artifacts; 2) demarcate a clear home for discourses around folksonomy disagreements; 3) highlight clearly when decisions have been reached; and 4) inform others about those decisions.

%B Computer Supported Cooperative Work %V 27 %P 647–678 %G eng %U https://rdcu.be/NZ7E %N 3–6 %R 10.1007/s10606-018-9327-z %> https://citsci.syr.edu/sites/crowston.syr.edu/files/ECSCW-Paper-Final.pdf %0 Generic %D 2016 %T Encouraging Work in Citizen Science: Experiments in Goal Setting and Anchoring %A Corey Brian Jackson %A Kevin Crowston %A Gabriel Mugar %A Carsten Østerlund %X This paper describes the results of an online field experiment where we designed and analyzed the effects of a goal-setting tracker in an online citizen science project - Floating Forest. The design of our tracker was influenced by psychology theories of anchoring and goal-setting. Our results of our experiment revealed: (1) setting goals increases annotations in a session; (2) numeric anchors influence goals; and (3) participants in the treatment who saw a prompt but did not set a goal, contributed more annotations than the participants in the control group. Our research shows how goal-setting and anchoring combine to increase work in online communities. %B ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW) %R 10.1145/2818052.2869129 %> https://citsci.syr.edu/sites/crowston.syr.edu/files/CSCW-abstract.pdf %0 Conference Proceedings %B ACM Group %D 2016 %T “Guess what! You’re the first to see this event”: Increasing Contribution to Online Production Communities %A Corey Brian Jackson %A Kevin Crowston %A Gabriel Mugar %A Carsten Østerlund %X

In this paper, we describe the results of an online field experiment examining the impacts of messaging about task novelty on the volume of volunteers’ contributions to an online citizen science project. Encouraging volunteers to provide a little more content as they work is an attractive strategy to increase the community’s output. Prior research found that an important motivation for participation in online citizen science is the wonder of being the first person to observe a particular image. To appeal to this motivation, a pop-up message was added to an online citizen science project that alerted volunteers when they were the first to annotate a particular image. Our analysis reveals that new volunteers who saw these messages increased the volume of annotations they contributed. The results of our study suggest an additional strategy to increase the amount of work volunteers contribute to online communities and citizen science projects specifically.

%B ACM Group %R 10.1145/2957276.2957284 %0 Conference Proceedings %B 19th ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW) %D 2016 %T Which Way Did They Go? Newcomer Movement through the Zooniverse %A Corey Brian Jackson %A Carsten Østerlund %A Maidel, Veronica %A Kevin Crowston %A Gabriel Mugar %X

Research on newcomer roles in peer production sites (e.g., Wikipedia) is characterized by a broad and relatively well-articulated set of functionally and culturally recognizable roles. But not all communities come with well-defined roles that newcomers can aspire to occupy. The present study explores activity clusters newcomers create when faced with few recognizable roles to fill and limited access to other participants’ work that serves as an exemplar. Drawing on a mixed method research design, we present findings from an analysis of 1,687 newcomers’ sessions in a citizen science project. Combining session- and individual-level analysis produced three findings (1) newcomers activities manifest a diverse range of session types; (2) Newcomers toggle between light work sessions and more involved types of production or community engagement; (3) an interesting relationship between high-level contributors who do a lot of work but little talk and a small group that does a lot of talk but less work. The former group draws heavily on posts contributed by the latter group. Identifying shifts and regularities in contribution facilitate improved mechanisms for engaging participants and the design of online citizen science communities.

%B 19th ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW) %8 2016 %R 10.1145/2818048.2835197 %> https://citsci.syr.edu/sites/crowston.syr.edu/files/CSCW2016-Roles.pdf %0 Conference Proceedings %B Proceedings of the Forty-eighth Hawai'i International Conference on System Sciences (HICSS-48) %D 2015 %T Motivations for sustained participation in crowdsourcing: The role of talk in a citizen science case study %A Corey Brian Jackson %A Carsten Østerlund %A Gabriel Mugar %A Kevin Crowston %A Katie DeVries Hassman %X

The paper explores the motivations of volunteers in a large crowd sourcing project and contributes to our understanding of the motivational factors that lead to deeper engagement beyond initial participation. Drawing on the theory of legitimate peripheral participation (LPP) and the literature on motivation in crowd sourcing, we analyze interview and trace data from a large citizen science project. The analyses identify ways in which the technical features of the projects may serve as motivational factors leading participants towards sustained participation. The results suggest volunteers first engage in activities to support knowledge acquisition and later share knowledge with other volunteers and finally increase participation in Talk through a punctuated process of role discovery.

%B Proceedings of the Forty-eighth Hawai'i International Conference on System Sciences (HICSS-48) %C Koloa, HI %8 1/2015 %> https://citsci.syr.edu/sites/crowston.syr.edu/files/Motivation%20in%20Talk%20Submitted_FINAL%28Formatted%29.pdf %0 Conference Proceedings %B 17th ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW 2014) %D 2014 %T Planet Hunters and Seafloor Explorers: Legitimate Peripheral Participation Through Practice Proxies in Online Citizen Science %A Gabriel Mugar %A Carsten Østerlund %A Katie DeVries Hassman %A Kevin Crowston %A Corey Brian Jackson %X

Making the traces of user participation in primary activities visible in online crowdsourced initiatives has been shown to help new users understand the norms of participation but participants do not always have access to others’ work. Through a combination of virtual and trace ethnography we explore how new users in two online citizen science projects engage other traces of activity as a way of compensating. Merging the theory of legitimate peripheral participation with Erickson and Kellogg’s theory of social translucence we introduce the concept of practice proxies; traces of user activities in online environment that act as resources to orient newcomers towards the norms of practice. Our findings suggest that newcomers seek out practice proxies in the social features of the projects that highlight contextualized and specific characteristics of primary work practice.

%B 17th ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW 2014) %8 2/2014 %R 10.1145/2531602.2531721 %> https://citsci.syr.edu/sites/crowston.syr.edu/files/paper_revised%20copy%20to%20post.pdf