KAIST Fall 2017

CS492: Crowdsourcing & Social Computing

Human-computer interaction no longer only concerns a single user in front of their computer. An increasing number of modern systems are inherently social, involving a large group of users to collaborate, discuss, ideate, solve problems, and make decisions together via social interaction. This course focuses on crowdsourcing and social computing, two of the most important concepts in the era of interaction at scale. This course will cover major design issues and computational techniques in building crowdsourcing and social computing systems.

Changes from the 2016 course: The course now covers a broader scope by including social computing; The reading load is reduced and includes more accessible articles.

This is a highly interactive class: You’ll be expected to actively participate in activities, projects, assignments, and discussions. There will be no lectures or exams. Major course activities include:

  • Reading Response: You'll read and discuss important papers and articles in the field. Each week, there will be 1-2 reading assignments, for which you'll write a short response to.
  • Topic Presentation: Once a semester, you'll lead the class by summarizing the readings and peer students' critiques, and spurring the in-class discussion. You’ll also get a chance to design your own activity that best captures the topic of a class.
  • Design Project: In a semester-long team project, you'll design, build, and test your own crowdsourcing / social computing system. If you have an ongoing research project that might benefit from having a crowdsourcing / social computing component, connecting to your research is encouraged.
  • In-class Activities: Each class will feature activities that will help you experience and practice the core concepts introduced in the course.

Course Staff

Instructor: Prof. Juho Kim
    Office Hours: 4-5pm Tue/Thu @ N1-605

TA: Seoyoung Kim
    Office Hours: 2:30-3:30pm Tue @ N1-625

Staff Mailing List: cs492@kixlab.org

Time & Location

When: 1:00-2:15pm Tue/Thu
Where: N1-114


Course Website: http://kixlab.org/courses/crowdsourcing/
Submission & Grading: KLMS
Discussion Forum: Piazza



Week Date Topic Reading (response indicates a reading response is required for the article.) Due
1 8/29 Introduction & Course Overview (PDF)
1 8/31 Part 1: Crowdsourcing
Introduction to Crowdsourcing and Human Computation (PDF)
- Discussion by Juho (PDF)
(1) Howe, Jeff. "The rise of crowdsourcing." Wired magazine 14.6 (2006): 1-4.
(2) Quinn, Alexander J., and Bederson, Benjamin B. "Human computation: a survey and taxonomy of a growing field." CHI 2011.
2 9/5 Crowdsourcing Platforms (PDF)
- Discussion by Kamil and Muso (PDF)
(1) response Ipeirotis, Panagiotis G. "Analyzing the amazon mechanical turk marketplace." XRDS: Crossroads 17.2 (2010): 16-21.
(2) Katz, Miranda. "Amazon's Turk Crowd Has Had Enough." Wired. 2017.
(3) Geiger, David, et al. "Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes." AMCIS. 2011.
2 9/7 Crowd Workers (PDF)
- Discussion by Jiyoun (PDF)
(1) response Irani, Lilly C., and M. Silberman. "Turkopticon: interrupting worker invisibility in amazon mechanical turk." CHI 2013.
(2) Martin, David, et al. "Being a turker." CSCW 2014.
(3) Bigham, Jeffrey P. "My MTurk (half) Workday."
Assignment 1: Be a Crowd Worker
Project 0: Team Formation
3 9/12 Programming Paradigms (PDF)
- Discussion by Gautam & Lucas (PDF)
(1) response Little, Greg, et al. "Turkit: human computation algorithms on mechanical turk." UIST 2010.
(2) Little, Greg, et al. "Exploring iterative and parallel human computation processes." Proceedings of the ACM SIGKDD workshop on human computation. 2010.
(3) Kittur, Aniket, et al. "Crowdforge: Crowdsourcing complex work." UIST 2011.
3 9/14 Workflows (PDF)
- Discussion by Sungmin & Woo Jin (PDF)
(1) response Bernstein, Michael S., et al. "Soylent: a word processor with a crowd inside." UIST 2010.
(2) Kulkarni, Anand, Matthew Can, and Bjoern Hartmann. "Collaboratively crowdsourcing workflows with turkomatic." CSCW 2012.
(3) Chilton, Lydia B., et al. "Cascade: Crowdsourcing taxonomy creation." CHI 2013.
4 9/19 Quality Control 1 (PDF)
- Discussion by Chae-Ryn (PDF)
(1)response Harris, Mark. "How a lone hacker shredded the myth of crowdsourcing."
(2) Snow, Rion, et al. "Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks." EMNLP 2008.
Project 1: Ideas
4 9/21 Quality Control 2 (PDF)
- Discussion by Kyung Je (PDF)
(1)response Ipeirotis, Panos. "Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers?"
(2) Rzeszotarski, Jeffrey, and Aniket Kittur. "CrowdScape: interactively visualizing user behavior and output." UIST 2012.
5 9/26 Realtime Crowdsourcing (PDF)
- Discussion by Bolat & Thao (PDF)
(1) response Bigham, Jeffrey P., et al. "VizWiz: nearly real-time answers to visual questions." UIST 2010.
(2) Bernstein, Michael S., et al. "Crowds in two seconds: Enabling realtime crowd-powered interfaces." UIST 2011.
(3) Lasecki, Walter S., et al. "Real-time captioning by groups of non-experts." UIST 2012.
5 9/28 Machines and Crowds (PDF)
- Discussion by Zhantore (PDF)
(1) response Laput, Gierad, et al. "Zensors: Adaptive, rapidly deployable, human-intelligent sensor feeds." CHI 2015.
(2) Cheng, Justin, and Michael S. Bernstein. "Flock: Hybrid crowd-machine learning classifiers." CSCW 2015.
(3) Sankar, Shyam. The Rise of Human Computer Cooperation. TED Talk Video (12 mins).
Project 2: Story
6 10/3 No Class (Chuseok)
6 10/5 No Class (Chuseok)
7 10/10 Project Pitches 1
7 10/12 Project Pitches 2 Project 3: Pitch
8 10/17 No Class (Midterm Exams)
8 10/19 No Class (Midterm Exams) Project 4: Low-fi Prototype
9 10/24
7pm 10/31
Application: Learning & Feedback (PDF)
- Discussion by Hyungyu & Andreas (PDF)
response : choose one from (1)-(3)
(1) Weir, Sarah, et al. "Learnersourcing subgoal labels for how-to videos." CSCW 2015.
(2) Bragg, Rector, & Ladner. "A User-Powered American Sign Language Dictionary." CSCW 2015.
(3) Luther, Kurt, et al. "Structuring, Aggregating, and Evaluating Crowdsourced Design Critique." CSCW 2015.
9 10/26
7pm 10/31
Application: Civic Engagement & Accessibility (PDF)
- Discussion by Dongkwan & Chan Ju (PDF)
response : choose one from (1)-(3)
(1) Haklay, Mordechai, and Patrick Weber. "Openstreetmap: User-generated street maps." IEEE Pervasive Computing 7.4 (2008): 12-18.
(2) Kim, Nam Wook, et al. "Budgetmap: Engaging taxpayers in the issue-driven classification of a government budget." CSCW 2016.
(3) Guo, Anhong, et al. "VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World." UIST 2016.
10 10/31 Part 2: Social Computing
Introduction to Social Computing (PDF)
- Discussion by Sovanny (PDF)
response : choose one from (1)-(2)
(1) Grudin, Jonathan. "Why CSCW applications fail: problems in the design and evaluationof organizational interfaces." CSCW 1988.
(2) Ackerman, Mark. "The Intellectual Challenge of CSCW: The Gap Between Social Requirements and Technical Feasibility". Human-computer interaction 15.2 (2000): 179-203.
10 11/2 Communication Tools (email, teleconferencing) (PDF)
- Discussion by Olzhas (PDF)
(1) response Hollan & Stornetta. "Beyond Being There." CHI 1992.
(2) Viégas, Golder, & Donath. "Visualizing Email Content: Portraying Relationships from Conversational Histories." CHI 2006.
(3) Venolia, Tang, & Inkpen. "SeeSaw: I See You Saw My Video Message." MobileHCI 2015.
11 11/7 Collaboration Tools (peer production, innovation) (PDF)
- Discussion by Hyoungwook & Wonkeun (PDF)
response : choose one from (1)-(2)
(1) Kittur & Kraut. "Harnessing the wisdom of crowds in wikipedia: quality through coordination." CSCW 2008.
(2) Vaish et al. "Crowd Research: Open and Scalable University Laboratories." UIST 2017.
11 11/9 Discussion Tools (forum, Q&A) (PDF)
- Discussion by Hyunwoo & Adi (PDF)
response : choose one from (1)-(2)
(1) Zhang et al. "Wikum: Bridging Discussion Forums and Wikis Using Recursive Summarization." CSCW 2017.
(2) Mamykina et al. "Design Lessons from the Fastest Q&A Site in the West." CHI 2011.
Project 5: Mid-fi Prototype
12 11/14 Social Networks 1 (Facebook) (PDF)
- Discussion by So Hee & Hyeungshik (PDF)
response : choose one from (1)-(2)
(1) Gilbert, Eric, and Karrie Karahalios. "Predicting tie strength with social media." CHI 2009.
(2) Eslami, Motahhare, et al. "I always assumed that I wasn't really that close to [her]: Reasoning about Invisible Algorithms in News Feeds." CHI 2015.
12 11/16 Social Networks 2 (Twitter) (PDF)
- Discussion by Ina (PDF)
response : choose one from (1)-(2)
(1) Marcus, Adam, et al. "TwitInfo: Aggregating and Visualizing Microblogs for Event Exploration." CHI 2011.
(2) Manjoo, Farhad. "How Twitter Is Being Gamed to Feed Misinformation." The New York Times, 2017.05.31.
Assignment 2: Analyze Crowdsourcing Platforms
13 11/21 Incentives & Gamification (PDF)
- Discussion by Jae Hoon & Adil (PDF)
response : choose one from (1)-(2)
(1) Beenan, Gerard, et al. "Using social psychology to motivate contributions to online communities." CSCW 2004.
(2) Cooper, Seth, et al. "Predicting protein structures with a multiplayer online game." Nature 466.7307 (2010): 756-760.
13 11/23 No Class (Project Work Time)
14 11/28 Large-Scale Online Experimentation (PDF)
- Discussion by Namung (PDF)
response : choose one from (1)-(2)
(1) Reinecke, Katharina, and Krzysztof Z. Gajos. "LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples." CSCW 2015.
(2) Kohavi, Ron, et al. "Online controlled experiments at large scale." KDD 2013.
Project 6: High-fi Prototype
14 11/30 The Future of Crowdsourcing & Social Computing (PDF)
- Discussion by Simon (PDF)
(1) response Kittur, Aniket, et al. "The future of crowd work." CSCW 2013.
(2) Woolley, Anita Williams, et al. "Evidence for a collective intelligence factor in the performance of human groups." Science 330.6004 (2010)
15 12/5 Final Presentations 1
14 12/7 No Class (Undergrad Admission Interviews)
15 12/8 (7pm) Final Presentations 2 Project 7: Final Presentation
16 12/12 No Class (Final Exams)
16 12/14 No Class (Final Exams) Project 8: Final Paper & Video

Topics (tentative)

Major topics include:
  • Crowdsourcing and human computation
  • Social computing platforms
  • Social networks and online communities
  • Computer supported collaborative work
  • Privacy and ethics
  • Incentives and gamification
  • Online experimentation and analysis
  • Programming paradigms
  • Quality control and fighting malicious behaviors


  • Design project: 40%
  • Topic Presentation: 20%
  • Reading responses: 20%
  • Assignments: 10%
  • Class participation: 10%
Late policy: Three lowest reading response grades will be removed. No late submissions are allowed for the reading responses. For assignments and project milestones, you'll lose 10% for each late day. Submissions will be accepted until three days after the deadline.


There are no official course prerequisites. But assignments and the final project will require building features of a crowdsourcing / social computing system, so programming skills are needed. Knowledge or research experience in HCI (CS374 or equivalent) or social computing is useful, but not required.