Crowdsourcing has successfully solved a wide variety of real-world problems. By inviting lots of people to participate in the problem solving process, crowdsourcing has tackled problems that neither computers nor humans could solve alone. Common application areas include data collection and labeling, collaborative ideation and problem solving, citizen science, education, and accessibility. In building a successful crowdsourcing system, important challenges arise in recruitment, incentive structure, task design, workflow design, quality control, data processing, and ethics, just to name a few. This course will cover major design issues and computational techniques in building crowdsourcing systems. You will (1) read, present, and discuss important papers in the field, (2) make, run, and analyze crowdsourcing tasks, and (3) design your own crowdsourcing system as a final project.
| Week | Date | Topic | Reading (response indicates a reading response is required for the article.) | Due |
|---|---|---|---|---|
| 1 | 9/1 | Introduction & Course Info (PDF) | ||
| 1 | 9/6 | Introduction to crowdsourcing and human computation (PDF) Discussion by Juho (PDF) |
(1) Howe, Jeff. "The rise of crowdsourcing." Wired magazine 14.6 (2006): 1-4. (2) Quinn, Alexander J., and Bederson, Benjamin B. "Human computation: a survey and taxonomy of a growing field." CHI 2011. |
|
| 2 | 9/8 | Crowdsourcing platforms (PDF) Discussion by Oisin (PDF) |
(1) response Ipeirotis, Panagiotis G. "Analyzing the amazon mechanical turk marketplace." XRDS: Crossroads 17.2 (2010): 16-21. (2) response Geiger, David, et al. "Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes." AMCIS. 2011. (3) Vakharia, Donna, and Matthew Lease. "Beyond AMT: An analysis of crowd work platforms." arXiv preprint arXiv:1310.1672 (2013). |
|
| 2 | 9/13 | Worker Issues in Crowdsourcing (PDF) Discussion by Sang-gyun (PDF) |
(1) response Irani, Lilly C., and M. Silberman. "Turkopticon: interrupting worker invisibility in amazon mechanical turk." CHI 2013. (2) response Martin, David, et al. "Being a turker." CSCW 2014. (3) Bigham, Jeffrey P. "My MTurk (half) Workday." (4) Gray, Mary L., et al. "The crowd is a collaborative network." CSCW 2016. |
Assignment 1: Be a crowd worker |
| 3 | 9/15 | No class (Chuseok) | ||
| 3 | 9/20 | Technique: programming paradigms part 1 (PDF) Discussion by Youngbo (PDF) |
(1) response Little, Greg, et al. "Turkit: human computation algorithms on mechanical turk." UIST 2010. (2) Little, Greg, et al. "Exploring iterative and parallel human computation processes." Proceedings of the ACM SIGKDD workshop on human computation. ACM, 2010. (3) Barowy, Daniel W., et al. "Automan: A platform for integrating human-based and digital computation." ACM SIGPLAN Notices 47.10 (2012): 639-654. |
Assignment 2: Analyze crowdsourcing systems |
| 4 | 9/22 | Technique: programming paradigms part 2 (PDF) Discussion by Juho (slides in the main material) |
(1) response Bernstein, Michael S., et al. "Soylent: a word processor with a crowd inside." UIST 2010. (2) response Kittur, Aniket, et al. "Crowdforge: Crowdsourcing complex work." UIST 2011. (3) Ahmad, Salman, et al. "The jabberwocky programming environment for structured social computing." UIST 2011. |
Project 0: Team Formation |
| 4 | 9/27 | Technique: quality control part 1(PDF) Discussion by Junsoo (PDF) |
(1) response Harris, Mark. "How a lone hacker shredded the myth of crowdsourcing." (2) response Snow, Rion, et al. "Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks." EMNLP 2008. (3) Zaidan, Omar F., and Chris Callison-Burch. "Crowdsourcing translation: Professional quality from non-professionals." ACL 2011. |
|
| 5 | 9/29 8pm 10/5 |
Technique: quality control part 2(PDF) Discussion by Goh (PDF) |
response : choose one from (1)-(3) (1) Ipeirotis, Panos. "Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers?" (2) Gaikwad, Snehalkumar (Neil) S. et al. "Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms." UIST 2016 (in press). (3) Rzeszotarski, Jeffrey, and Aniket Kittur. "CrowdScape: interactively visualizing user behavior and output." UIST 2012. |
|
| 5 | 10/4 8pm 10/7 |
Technique: realtime crowdsourcing (PDF) Discussion by Yekaterina & Nurzhan (PDF) |
(1) response Bigham, Jeffrey P., et al. "VizWiz: nearly real-time answers to visual questions." UIST 2010. (2) response Bernstein, Michael S., et al. "Crowds in two seconds: Enabling realtime crowd-powered interfaces." UIST 2011. |
Project 1: Idea |
| 6 | 10/6 | Technique: crowd agents (PDF) Discussion by Umair (PDF) |
response : choose one from (1)-(3) (1) Lasecki, Walter S., et al. "Real-time captioning by groups of non-experts." UIST 2012. (2) Lasecki, Walter S., et al. "Chorus: a crowd-powered conversational assistant." UIST 2013. (3) Lasecki, Walter S., et al. "Apparition: Crowdsourced user interfaces that come to life as you sketch them." CHI 2015. |
|
| 6 | 10/11 | Design: workflow design & task decomposition part 1(PDF) Discussion by Jean (PDF) |
(1) response Kulkarni, Anand, Matthew Can, and Bjoern Hartmann. "Collaboratively crowdsourcing workflows with turkomatic." CSCW 2012. (2) response Zhang, Haoqi, et al. "Human computation tasks with global constraints." CHI 2012. |
|
| 7 | 10/13 | Design: workflow design & task decomposition part 2(PDF) Discussion by Jiwoo (PDF) |
response : choose one from (1)-(3) (1) Chilton, Lydia B., et al. "Cascade: Crowdsourcing taxonomy creation." CHI 2013. (2) Kim, Juho, et al. "Crowdsourcing step-by-step information extraction to enhance existing how-to videos." CHI 2014. (3) Noronha, Jon, et al. "Platemate: crowdsourcing nutritional analysis from food photographs." UIST 2011. |
Project 2: Story |
| 7 | 10/18 | Project Feedback Session | ||
| 8 | 10/20 | Project Proposal Pitches & Feedback | Project 3: Pitch | |
| 8 | 10/25 | No class (Midterms week) | ||
| 9 | 10/27 | Design: incentives GUEST LECTURE by Krzysztof Gajos (Harvard University) Discussion by Juho Sun (PDF) |
(1) response Reinecke, Katharina, and Krzysztof Z. Gajos. "LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples." CSCW 2015. (2) Mason, Winter, and Duncan J. Watts. "Financial incentives and the performance of crowds." ACM SIGKDD Explorations Newsletter 11.2 (2010): 100-108. (3) Hsieh, Gary, and Rafal Kocielnik. "You Get Who You Pay for: The Impact of Incentives on Participation Bias." CSCW 2016. |
|
| 9 | 11/1 | Design: games and gamification (PDF) Discussion by Hyunsung (PDF) |
response : choose one from (1)-(3) (1) Von Ahn, Luis, and Laura Dabbish. "Labeling images with a computer game." CHI 2004. (2) Cooper, Seth, et al. "Predicting protein structures with a multiplayer online game." Nature 466.7307 (2010): 756-760. (3) Deterding, Sebastian, et al. "From game design elements to gamefulness: defining gamification." MindTrek 2011. |
|
| 10 | 11/3 | No class (students are encouraged to attend Computing in the 21st Century 2016) | ||
| 10 | 11/8 | Design: experimentation & analysis (PDF) Discussion by Hoon and Sunggeun (PDF) |
response : choose TWO from (1)-(3) (1) Kittur, Aniket, Ed H. Chi, and Bongwon Suh. "Crowdsourcing user studies with Mechanical Turk." CHI 2008. (2) Salganik, Matthew J., and Karen EC Levy. "Wiki surveys: Open and quantifiable social data collection." PloS one 10.5 (2015): e0123483. (3) Kohavi, Ron, et al. "Online controlled experiments at large scale." KDD 2013. |
|
| 11 | 11/10 | Application: data labeling, computer vision, NLP (PDF) Discussion by Oscar (PDF) |
response : choose one from (1)-(3) (1) Deng, Jia, et al. "Imagenet: A large-scale hierarchical image database." CVPR 2009. (2) Russell, Bryan C., et al. "LabelMe: a database and web-based tool for image annotation." International journal of computer vision 77.1-3 (2008). (3) Callison-Burch, Chris. "Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk." EMNLP 2009. |
Project 4: Low-Fi Prototype |
| 11 | 11/15 | Application: Machines and Crowds (PDF) Discussion by Youngjae (PDF) |
response : choose TWO from (1)-(3) (1) Laput, Gierad, et al. "Zensors: Adaptive, rapidly deployable, human-intelligent sensor feeds." CHI 2015. (2) Williams, Joseph Jay, et al. "AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning." Learning at Scale 2016. (3) Cheng, Justin, and Michael S. Bernstein. "Flock: Hybrid crowd-machine learning classifiers." CSCW 2015. |
|
| 12 | 11/17 | Application: can the crowd learn? Learnersourcing (PDF) Discussion by Yoo Jin (PDF) |
response : choose one from (1)-(3) (1) Weir, Sarah, et al. "Learnersourcing subgoal labels for how-to videos." CSCW 2015. (2) D. Bragg, K. Rector, R.E. Ladner. "A User-Powered American Sign Language Dictionary." CSCW 2015. (3) Glassman, Elena L., et al. "Mudslide: A spatially anchored census of student confusion for online lecture videos." CHI 2015. |
|
| 12 | 11/22 | Application: feedback & creativity (PDF) Discussion by Yungi (PDF) |
response : choose TWO from (1)-(3) (1) Chinmay E. Kulkarni, Michael S. Bernstein, and Scott R. Klemmer. "PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance." Learning @ Scale 2015. (2) Kurt Luther et al. "Structuring, Aggregating, and Evaluating Crowdsourced Design Critique." CSCW 2015. (3) Pao Siangliulue et al. "IdeaHound: Improving Large-scale Collaborative Ideation with Crowd-Powered Real-time Semantic Modeling." UIST 2016. |
Project 5: Mid-fi Prototype |
| 13 | 11/24 | No class (Undergraduate admission interviews) | Assignment 3: Heuristic Evaluation | |
| 13 | 11/29 | Application: civic engagement (PDF) Discussion by Paul (PDF) |
response : choose TWO from (1)-(3) (1) Haklay, Mordechai, and Patrick Weber. "Openstreetmap: User-generated street maps." IEEE Pervasive Computing 7.4 (2008): 12-18. (2) Kim, Nam Wook, et al. "Budgetmap: Engaging taxpayers in the issue-driven classification of a government budget." CSCW 2016. (3) Heimerl, Kurtis, et al. "CommunitySourcing: engaging local crowds to perform expert work via physical kiosks." CHI 2012. |
|
| 14 | 12/1 | Application: citizen science and participatory sensing (PDF) Discussion by Noé (PDF) |
response : choose one from (1)-(3) (1) Bonney, Rick, et al. "Citizen science: a developing tool for expanding science knowledge and scientific literacy." BioScience 59.11 (2009): 977-984. (2) Sullivan, Brian L., et al. "eBird: A citizen-based bird observation network in the biological sciences." Biological Conservation 142.10 (2009). (3) Goldman, Jeffrey, et al. "Participatory Sensing: A citizen-powered approach to illuminating the patterns that shape our world." Foresight & Governance Project, White Paper (2009). |
|
| 14 | 12/6 | Application: teamwork and expert crowdsourcing (PDF) Discussion by Miro (PDF) |
response : choose one from (1)-(3) (1) Woolley, Anita Williams, et al. "Evidence for a collective intelligence factor in the performance of human groups." Science 330.6004 (2010) (2) Retelny, Daniela, et al. "Expert crowdsourcing with flash teams." UIST 2014. (3) Salehi, Niloufar, et al. "Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability." CSCW 2017. |
Project 6: Hi-Fi Prototype |
| 15 | 12/8 | Application: accessibility (PDF) Discussion by Young-Min (PDF) |
response : choose one from (1)-(2) (1) Guo, Anhong, et al. "VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World." UIST 2016. (2) Hara, Kotaro, et al. "Tohme: detecting curb ramps in google street view using crowdsourcing, computer vision, and machine learning." UIST 2014. |
|
| 15 | 12/13 | The future of crowd work (PDF) Discussion by Sungjae (PDF) |
(1) response Kittur, Aniket, et al. "The future of crowd work." CSCW 2013. (2) Licklider, Joseph CR. "Man-computer symbiosis." IRE transactions on human factors in electronics 1 (1960): 4-11. (3) Humans Need Not Apply (15-min video) |
|
| 16 | 12/15 | Final Project Presentations | Project 7: Final Presentations | |
| 16 | 12/20 | No class (Finals week) |