CS691CS: Crowdsourcing - Opportunities & Challenges
Overview Schedule Reading List
Reading List
Format
A paper review should summarize the core ideas in the paper in about 1-2 paragraphs, followed by a list of pros and cons of the approach with justifications. You can then have a more detailed set of comments that list any questions/comments/thoughts about the work. There is no need to comment on the English Grammar/typo issues in the paper. We are interested only in the problem, ideas, and evaluation methodology. Here are some instructions on how to write reviews, and here is an example of a review.

Submission
Please email paper reviews to the instructor by 12 noon on the day of class. Please send reviews in the body of the message. Do not send them as attachments.

Software

  • TurKit: Iterative Tasks on Mechanical Turk
  • Ushahidi: Crowdsourcing Crisis Information
  • get-another-label: A set of tools for managing redundant answers collected from Amazon Mechanical Turk
  • mCrowd: a platform for mobile crowdsourcing

    Reading list (subject to change)

  • CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones (MobiSys 2010)
  • Cooperative Transit Tracking using GPS-enabled Smartphones (SenSys 2010)
  • Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design (CHI 2010)
  • Stress OutSourced: A Haptic Social Network via Crowdsourcing (CHI 2009)
  • Crowdsourcing user studies with Mechanical Turk (CHI 2008)
  • Who are the Crowdworkers? Shifting Demographics in Mechanical Turk (CHI 2010)
  • Community-Guided Learning: Exploiting Mobile Sensor Users to Model Human Behavior (AAAI 2010)
  • Toward community sensing (IPSN 2008)
  • Sound Sense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones (MobiSys 2009)
  • txtEagle: Mobile Crowdsourcing (HCII 2009)
  • Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers (KDD 2008)
  • Decision-Theoretic Control of Crowd-Sourced Workflows (AAAI 2010).
  • TurKit: Tools for Iterative Tasks on Mechanical Turk (HComp 2009)
  • Cheap and fast? but is it good? evaluating non-expert annotations for natural language tasks (EMNLP 2008)
  • Games with a purpose (IEEE Computer 2006)
  • Predicting the Present with Google Trends (Google Tech Report 2009)
  • JabberWocky: Crowd-Sourcing Metadata for Files (SCC 2009)
  • Citizens as Sensors: the world of volunteered georgraphy (GeoJournal 2007)
  • A Taxonomy of Distributed Human Computation (UMD TR 2009)
  • Crowdsourcing Service-Level Network Event Detection (Sigcomm 2010)
  • VTrack: Accurate, Energy-Aware Road Traffic Delay Estimation Using Mobile Phones
  • Recaptcha: Human-based character recognition via web security measures, Science 2008.
  • Crowd-Sourced Sensing and Collaboration Using Twitter (WoWMoM 2010).
  • Secure Collaborative Sensing for Crowdsourcing Spectrum Data in White Space Networks (DySpan 2010)
  • Low Cost and Secure Smart Meter Communications using the TV White Spaces (ISRCS 2010)
  • Did You See Bob? Using Mobile Phones to Locate People (Mobicom 2010)
  • MoVi: Mobile Phone based Video Highlights via Collaborative Sensing (Mobicom 2010)
  • Participatory Sensing in Commerce: Using Mobile Camera Phones to Track Market Price Dispersion (UrbanSense 2008)
  • Design of Experiments for Crowdsourcing Search Evaluation: challenges and opportunities (Alonso, 2010)
  • Financial Incentives and the 'Performance of Crowds (KDD-HCOMP 2009)
  • Crowdsourcing Critical Success Factor Model
  • Crowdsourcing, attention, and productivity
  • Crowdsourcing and All-Pay Auctions (EC 2009)
  • Geo-Wiki.Org: The Use of Crowdsourcing to Improve Global Land Cover
  • PEIR, the Personal Environmental Impact Report, as a Platform for Participatory Sensing Systems Research (MobiSys 2009)
  • GreenGPS: A Participatory Sensing Fuel-Efficient Maps Application (MobiSys 2010)
  • Crowdsourcing and Knowledge Sharing: Strategic User Behavior on Taskcn (EC 2008)
  • Money, Glory and Cheap Talk: Analyzing Strategic Behavior of Contestants in Simultaneous Crowdsourcing Contests on TopCoder.com (WWW 2010).
  • Get another label? improving data quality and data mining using multiple, noisy labelers. (KDD 2008)
  • The Labor Economics of Paid Crowdsourcing (EC 2010)
  • Examining Micro-Payments for Participatory Sensing Data Collections (Ubicomp 2010)
  • Recruitment Framework for Participatory Sensing Data Collections (Pervasive 2010)
  • Crowdsourcing as a Model for Problem Solving: An Introduction and Cases
  • Common Sense Community: Scaffolding Mobile Sensing and Analysis for Novice Users (Pervasive 2010)
  • MyExperience: A System for In situ Tracing and Capturing of User Feedback on Mobile Phones (MobiSys 2007)
  • Generalized Task Markets for Human and Machine Computation (AAAI 2010)
  • Mobile Opportunistic Commerce: Mechanisms, Architecture, and Application (AAMAS 2008)
  • Personalization via Friendsourcing (ACM TOCHI)
  • Toward Automatic Task Design: A Progress Report (KDD-HCOMP 2010)
  • Utility data annotation with Amazon MTurk, CVPR Workshop, 2008)
  • Galaxy Zoo: Reproducing Galaxy Morphologies via Machine Learning (Monthly Notices of the Royal Astronomical Society, 2010)
  • Vision of a Visipedia, Proceedings of IEEE, 2010
  • Online crowdsourcing: rating annotators and obtaining cost-effective labels (CVPR Workshop, 2010)
  • Common Sense Community: Scaffolding Mobile Sensing and Analysis for Novice Users (Pervasive 2010)
  • A Vehicle for Research: Using Street Sweepers to Explore the Landscape of Environmental Community Action (CHI 2009)
  • Recommender Systems, Springer 2010
  • Exploring Iterative and Parallel Human Computation Processes (HCOMP 2010)
  • TurKit: Human Computation Algorithms on Mechanical Turk (UIST, 2010)
  • Motivating Firm-Sponsored E-Collective Work (SSRN)
  • Hapori: Context-based Local Search for Mobile Phones using Community Behavioral Modeling and Similarity (Ubicomp 2010)
  • Bucking the Trend: Large-Scale Cost-Focused Active Learning for Statistical Machine Translation (ACL-2010)
  • Cheap, Fast and Good Enough: Automatic Speech Recognition with Non-Expert Transcription, NAACL-2010.
  • Predicting Human-Targeted Translation Edit Rate via Untrained Human Annotators, NAACL-2010.
  • Nericell: Rich Monitoring of Road and Traffic Conditions using Mobile Smartphones, SenSys 2008.
  • Sell Your Experiences: A Market Mechanism based Incentive for Participatory Sensing, Pervasive 2010
  • Active Learning and Crowdsourcing for Machine Translation, LREC 2010.
  • Far-Sighted Active Learning on a Budget for Image and Video Recognition, CVPR 2010
  • CrowdFlow: Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility, UMD TR
  • Cost-sensitive Active Visual Category Learning, IJCV 2010
  • Global Budgets for Local Recommendations, RecSys 2010
  • Designing Reputation Systems for the Social Web, BU School of Management Research Paper, 2010
  • PoolView: Stream Privacy in Grassroots Participatory Sensing, SenSys 2008.
  • UbiGreen: investigating a mobile tool for tracking and supporting green transportation habits, CHI 2009.
  • SMILE: Encounter-Based Trust for Mobile Social Services, CCS 2009.
  • Automatic Construction of Travel Itineraries using Social Breadcrumbs, HT 2009.
  • Predicting protein structures with a multiplayer online game, Nature 2010.
  • The challenge of designing scientific discovery games, FDG 2010.
  • Reconstructing the World in 3D: Bringing Games with a Purpose Outdoors, FDG 2010.
  • Prediction Without Markets, EC 2010.
  • Predicting Consumer Behavior with Web Search, NAS 2010.
  • Micro-Blog: Sharing and Querying Content Through Mobile Phones and Social Participation, MobiSys 2008.
  • Recognizing Multi-Agent Activities from GPS Data, AAAI 2010.
  • CrowdSourcing Human Problem Solving Strategy, CrowdConf 2010.
  • Crowd Sourcing Local Search Relevance, CrowdConf 2010.
  • Crowdsourcing for Human Subjects Research, CrowdConf 2010.
  • Preventing Satisficing in Online Surveys: A \"Kapcha\" to Ensure Higher Quality Data, CrowdConf 2010.
  • Inferring friendship network structure by using mobile phone data, PNAS 2009.
  • A Guide to Conducting Behavioral Research on Amazon's Mechanical Turk, Working Paper
  • Virtual Trip Lines for Distributed Privacy-Preserving Traffic Monitoring, MobiSys 2008.
  • Crowdsourced translation for emergency response in Haiti: the global collaboration of local knowledge, AMTA Workshop on Collaborative Crowdsourcing for Translation 2010.
  • Designing Incentives for Online Question and Answer Forums, EC 2009
  • Earthquake shakes Twitter users: real-time event detection by social sensors, WWW 2010
  • Preserving privacy in gps traces via uncertainty-aware path cloaking, CCS 2007.
  • Hiding Stars with Fireworks: Location Privacy through Camouflage, Mobicom 2009.
  • TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones, OSDI 2010.
  • Annotating named entites in Twitter data with crowdsourcing, NAACL-HLT 2010.
  • Social Sensing for Epidemiological Behavior Change, Ubicom 2010.
  • Personal Data Vaults: A Locus of Control for Personal Data Streams, CoNext 2010.
  • Time Critical Social Mobilization: The DARPA Network Challenge Winning Strategy, CSS Workshop, NIPS 2010.

    Relevant workshops

  • Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk
  • ACM SIGKDD Workshop on Human Computation 2009
  • ACM SIGKDD Workshop on Human Computation 2010
  • UBICOMP Workshop on Ubiquitous Crowdsourcing 2010
  • SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation