Cost of Quality in Crowdsourcing

Authors

  • Deniz Iren Middle East Technical University
  • Semih Bilgen Middle East Technical University

DOI:

https://doi.org/10.15346/hc.v1i2.14

Keywords:

Crowdsourcing, Human Computation, Cost of Quality, Cost Models, Quality Assurance

Abstract

Crowdsourcing is a business model which allows practitioners to access a rather cheap and scalable workforce. However, due to loose worker-employer relationships, skill diversity of the crowd and anonymity of participants, it tends to result in lower quality compared to traditional ways of doing work. Thus crowdsourcing practitioners have to apply certain techniques to make sure that the end product complies with the quality requirements. Quality assurance techniques impact project costs and schedules. A well-defined methodology is needed to estimate these impacts in order to manage the crowdsourcing process effectively and efficiently. This paper introduces cost models of common quality assurance techniques that may be applied in crowdsourcing and describes a proposed cost of quality approach for analyzing quality related costs.

References

Allahbakhsh, M., Benatallah, B., Ignjatovic, A., Motahari-Nezhad, H. R., Bertino, E., & Dustdar, S. (2013). Quality Control in Crowdsourcing Systems: Issues and Directions. Internet Computing, IEEE, 17(2), 76–81. doi:10.1109/MIC.2013.20

Amazon Mechanical Turk. (n.d.). Retrieved from http://www.mturk.com/

Arlot, S., & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40–79.

Conte, S. D., Dunsmore, H. E., & Shen, V. Y. (1985). Software effort estimation and productivity. Advances in Computers, 24, 1–60.

Crosby, P. B. (1979). Quality is free: The art of making quality certain (Vol. 94). McGraw-Hill New York.

CrossRef. (n.d.). Retrieved from www.crossref.org

Downs, J., & Holbrook, M. (2010). Are your participants gaming the system?: screening mechanical turk workers. In … in Computing Systems (pp. 0–3). Retrieved from http://dl.acm.org/citation.cfm?id=1753688

Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010). Are Your Participants Gaming the System ? Screening Mechanical Turk Workers, 0–3.

Fishman, G. S. (1996). Monte Carlo. Springer.

Geiger, D., & Seedorf, S. (2011). Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes. In … (pp. 1–11). Retrieved from http://schader.bwl.uni-mannheim.de/fileadmin/files/schader/files/publikationen/Geiger_et_al._-_2011_-_Managing_the_Crowd_Towards_a_Taxonomy_of_Crowdsourcing_Processes.pdf

Grier, D. A. (2011). Foundational Issues in Human Computing and Crowdsourcing. In Position Paper for the CHI 2011 Workshop on Crowdsourcing and Human Computation. CHI.

Hirth, M., Hoßfeld, T., & Tran-Gia, P. (2013). Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling, 57(11-12), 2918–2932. doi:10.1016/j.mcm.2012.01.006

Ho, C.-J., & Vaughan, J. W. (2012). Online Task Assignment in Crowdsourcing Markets. In AAAI.

Hossfeld, T., Hirth, M., & Tran-Gia, P. (2011). Modeling of Crowdsourcing Platforms and Granularity of Work Organization in Future Internet. In Proceedings of the 23rd International Teletraffic Congress (pp. 142–149). International Teletraffic Congress. Retrieved from http://dl.acm.org/citation.cfm?id=2043468.2043491

Hsueh, M.-C., Tsai, T. K., & Iyer, R. K. (1997). Fault injection techniques and tools. Computer, 30(4), 75–82.

Huang, E., Zhang, H., & Parkes, D. C. (n.d.). Toward Automatic Task Design : A Progress Report Categories and Subject Descriptors, 77–85.

Ipeirotis, P. G., Provost, F., & Wang, J. (2010). Quality management on Amazon Mechanical Turk. Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP ’10, 64. doi:10.1145/1837885.1837906

Karger, D. R., Oh, S., & Shah, D. (2011). Budget-optimal crowdsourcing using low-rank matrix approximations. 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), 284–291. doi:10.1109/Allerton.2011.6120180

Kazai, G., Kamps, J., & Milic-Frayling, N. (2011). Worker types and personality traits in crowdsourcing relevance labels. Proceedings of the 20th ACM international conference on Information and knowledge management - CIKM ’11, 1941. doi:10.1145/2063576.2063860

Kern, R., Thies, H., Bauer, C., & Satzger, G. (2010). Quality assurance for human-based electronic services: A decision matrix for choosing the right approach. In Current Trends in Web Engineering (pp. 421–424). Retrieved from http://link.springer.com/chapter/10.1007/978-3-642-16985-4_39

Kern, R., Zirpins, C., & Agarwal, S. (2009). Managing quality of human-based eservices. In Service-Oriented Computing--ICSOC 2008 Workshops (pp. 304–309).

Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., … Horton, J. (2013). The future of crowd work. Proceedings of the 2013 conference on Computer supported cooperative work - CSCW ’13, 1301. doi:10.1145/2441776.2441923

Laporte, C. Y., Berrhouma, N., Doucet, M., & Palza-Vargas, E. (2012). Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation.

Law, E., & Ahn, L. von. (2011). Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 5(3), 1–121.

Le, J., & Edmonds, A. (2010). Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In … crowdsourcing for search … (pp. 17–20). Retrieved from http://ir.ischool.utexas.edu/cse2010/materials/leetal.pdf

Literally Canvas. (n.d.). Retrieved from http://literallycanvas.com/

Malcolm, D. G., Roseboom, J. H., Clark, C. E., & Fazar, W. (1959). Application of a technique for research and development program evaluation. Operations research, 7(5), 646–669.

McCann, R., Shen, W., & Doan, A. (2008). Matching Schemas in Online Communities: A Web 2.0 Approach. 2008 IEEE 24th International Conference on Data Engineering, 110–119. doi:10.1109/ICDE.2008.4497419

Okubo, Y., Kitasuka, T., & Aritsugi, M. (2013). A Preliminary Study of the Number of Votes under Majority Rule in Crowdsourcing. Procedia Computer Science, 22, 537–543. doi:10.1016/j.procs.2013.09.133

Oleson, D., Sorokin, A., Laughlin, G., & Hester, V. (2011). Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. Human computation, 43–48. Retrieved from http://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/viewFile/3995/4267

Quinn, A., & Bederson, B. (2011). Human computation: a survey and taxonomy of a growing field. In … Conference on Human Factors in Computing …. Retrieved from http://dl.acm.org/citation.cfm?id=1979148

Quinn, A. J., & Bederson, B. B. (2011). Human Computation: A Survey and Taxonomy of a Growing Field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1403–1412). New York, NY, USA: ACM. doi:10.1145/1978942.1979148

Rao, R. B., Fung, G., & Rosales, R. (2008). On the Dangers of Cross-Validation. An Experimental Evaluation. In SDM (pp. 588–596).

Ribeiro, F., Florencio, D., & Nascimento, V. (2011). Crowdsourcing subjective image quality evaluation. In Image Processing (ICIP), 2011 18th IEEE International Conference on (pp. 3097–3100). doi:10.1109/ICIP.2011.6116320

Ross, J., Irani, L., & Silberman, M. (2010). Who are the crowdworkers?: shifting demographics in mechanical turk. In CHI’10 Extended … (pp. 2863–2872). Retrieved from http://dl.acm.org/citation.cfm?id=1753873

Rouse, A. (2010). A preliminary taxonomy of crowdsourcing. Retrieved from http://aisel.aisnet.org/acis2010/76/

Schenk, E., & Guittard, C. (2011). Towards a characterization of crowdsourcing practices. Journal of Innovation Economics & Management, (1), 93–107.

Schiffauerova, A., & Thomson, V. (2006). A review of research on cost of quality models and best practices. International Journal of Quality & Reliability Management, 23(6), 647–669. doi:10.1108/02656710610672470

Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 614–622). New York, NY, USA: ACM. doi:10.1145/1401890.1401965

Sorokin, A., & Forsyth, D. (2008). Utility data annotation with Amazon Mechanical Turk. In Computer Vision and Pattern Recognition Workshops, 2008. CVPRW ’08. IEEE Computer Society Conference on (pp. 1–8). doi:10.1109/CVPRW.2008.4562953

Surowiecki, J. (2005). The wisdom of crowds. Random House LLC.

Von Ahn, L., & Dabbish, L. (2008). Designing games with a purpose. Communications of the ACM, 51(8), 57. doi:10.1145/1378704.1378719

Voyer, R., Nygaard, V., Fitzgerald, W., Copperman, H., Suite, B. S., & Francisco, S. (2010). A Hybrid Model for Annotating Named Entity Training Corpora, (July), 243–246.

Vukovic, M. (2009). Crowdsourcing for Enterprises. In Congress on Services - I (pp. 686–692). doi:10.1109/SERVICES-I.2009.56

Vukovic, M., & Bartolini, C. (2010). Towards a research agenda for enterprise crowdsourcing. In Leveraging applications of formal methods, verification, and validation (pp. 425–434). Springer.

Welinder, P., & Perona, P. (2010). Online crowdsourcing: Rating annotators and obtaining cost-effective labels. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, 25–32. doi:10.1109/CVPRW.2010.5543189

Wieringa, R., & Morali, A. (2012). Technical action research as a validation method in information systems design science. In Design Science Research in Information Systems. Advances in Theory and Practice (pp. 220–238). Springer.

Wikipedia. (n.d.). Retrieved from www.wikipedia.org

Downloads

Published

2014-12-30

How to Cite

Iren, D., & Bilgen, S. (2014). Cost of Quality in Crowdsourcing. Human Computation, 1(2). https://doi.org/10.15346/hc.v1i2.14

Issue

Section

Research