Cost of Quality in Crowdsourcing
DOI:
https://doi.org/10.15346/hc.v1i2.14Keywords:
Crowdsourcing, Human Computation, Cost of Quality, Cost Models, Quality AssuranceAbstract
Crowdsourcing is a business model which allows practitioners to access a rather cheap and scalable workforce. However, due to loose worker-employer relationships, skill diversity of the crowd and anonymity of participants, it tends to result in lower quality compared to traditional ways of doing work. Thus crowdsourcing practitioners have to apply certain techniques to make sure that the end product complies with the quality requirements. Quality assurance techniques impact project costs and schedules. A well-defined methodology is needed to estimate these impacts in order to manage the crowdsourcing process effectively and efficiently. This paper introduces cost models of common quality assurance techniques that may be applied in crowdsourcing and describes a proposed cost of quality approach for analyzing quality related costs.References
Allahbakhsh, M., Benatallah, B., Ignjatovic, A., Motahari-Nezhad, H. R., Bertino, E., & Dustdar, S. (2013). Quality Control in Crowdsourcing Systems: Issues and Directions. Internet Computing, IEEE, 17(2), 76–81. doi:10.1109/MIC.2013.20
Amazon Mechanical Turk. (n.d.). Retrieved from http://www.mturk.com/
Arlot, S., & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40–79.
Conte, S. D., Dunsmore, H. E., & Shen, V. Y. (1985). Software effort estimation and productivity. Advances in Computers, 24, 1–60.
Crosby, P. B. (1979). Quality is free: The art of making quality certain (Vol. 94). McGraw-Hill New York.
CrossRef. (n.d.). Retrieved from www.crossref.org
Downs, J., & Holbrook, M. (2010). Are your participants gaming the system?: screening mechanical turk workers. In … in Computing Systems (pp. 0–3). Retrieved from http://dl.acm.org/citation.cfm?id=1753688
Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010). Are Your Participants Gaming the System ? Screening Mechanical Turk Workers, 0–3.
Fishman, G. S. (1996). Monte Carlo. Springer.
Geiger, D., & Seedorf, S. (2011). Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes. In … (pp. 1–11). Retrieved from http://schader.bwl.uni-mannheim.de/fileadmin/files/schader/files/publikationen/Geiger_et_al._-_2011_-_Managing_the_Crowd_Towards_a_Taxonomy_of_Crowdsourcing_Processes.pdf
Grier, D. A. (2011). Foundational Issues in Human Computing and Crowdsourcing. In Position Paper for the CHI 2011 Workshop on Crowdsourcing and Human Computation. CHI.
Hirth, M., Hoßfeld, T., & Tran-Gia, P. (2013). Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling, 57(11-12), 2918–2932. doi:10.1016/j.mcm.2012.01.006
Ho, C.-J., & Vaughan, J. W. (2012). Online Task Assignment in Crowdsourcing Markets. In AAAI.
Hossfeld, T., Hirth, M., & Tran-Gia, P. (2011). Modeling of Crowdsourcing Platforms and Granularity of Work Organization in Future Internet. In Proceedings of the 23rd International Teletraffic Congress (pp. 142–149). International Teletraffic Congress. Retrieved from http://dl.acm.org/citation.cfm?id=2043468.2043491
Hsueh, M.-C., Tsai, T. K., & Iyer, R. K. (1997). Fault injection techniques and tools. Computer, 30(4), 75–82.
Huang, E., Zhang, H., & Parkes, D. C. (n.d.). Toward Automatic Task Design : A Progress Report Categories and Subject Descriptors, 77–85.
Ipeirotis, P. G., Provost, F., & Wang, J. (2010). Quality management on Amazon Mechanical Turk. Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP ’10, 64. doi:10.1145/1837885.1837906
Karger, D. R., Oh, S., & Shah, D. (2011). Budget-optimal crowdsourcing using low-rank matrix approximations. 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), 284–291. doi:10.1109/Allerton.2011.6120180
Kazai, G., Kamps, J., & Milic-Frayling, N. (2011). Worker types and personality traits in crowdsourcing relevance labels. Proceedings of the 20th ACM international conference on Information and knowledge management - CIKM ’11, 1941. doi:10.1145/2063576.2063860
Kern, R., Thies, H., Bauer, C., & Satzger, G. (2010). Quality assurance for human-based electronic services: A decision matrix for choosing the right approach. In Current Trends in Web Engineering (pp. 421–424). Retrieved from http://link.springer.com/chapter/10.1007/978-3-642-16985-4_39
Kern, R., Zirpins, C., & Agarwal, S. (2009). Managing quality of human-based eservices. In Service-Oriented Computing--ICSOC 2008 Workshops (pp. 304–309).
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., … Horton, J. (2013). The future of crowd work. Proceedings of the 2013 conference on Computer supported cooperative work - CSCW ’13, 1301. doi:10.1145/2441776.2441923
Laporte, C. Y., Berrhouma, N., Doucet, M., & Palza-Vargas, E. (2012). Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation.
Law, E., & Ahn, L. von. (2011). Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 5(3), 1–121.
Le, J., & Edmonds, A. (2010). Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In … crowdsourcing for search … (pp. 17–20). Retrieved from http://ir.ischool.utexas.edu/cse2010/materials/leetal.pdf
Literally Canvas. (n.d.). Retrieved from http://literallycanvas.com/
Malcolm, D. G., Roseboom, J. H., Clark, C. E., & Fazar, W. (1959). Application of a technique for research and development program evaluation. Operations research, 7(5), 646–669.
McCann, R., Shen, W., & Doan, A. (2008). Matching Schemas in Online Communities: A Web 2.0 Approach. 2008 IEEE 24th International Conference on Data Engineering, 110–119. doi:10.1109/ICDE.2008.4497419
Okubo, Y., Kitasuka, T., & Aritsugi, M. (2013). A Preliminary Study of the Number of Votes under Majority Rule in Crowdsourcing. Procedia Computer Science, 22, 537–543. doi:10.1016/j.procs.2013.09.133
Oleson, D., Sorokin, A., Laughlin, G., & Hester, V. (2011). Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. Human computation, 43–48. Retrieved from http://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/viewFile/3995/4267
Quinn, A., & Bederson, B. (2011). Human computation: a survey and taxonomy of a growing field. In … Conference on Human Factors in Computing …. Retrieved from http://dl.acm.org/citation.cfm?id=1979148
Quinn, A. J., & Bederson, B. B. (2011). Human Computation: A Survey and Taxonomy of a Growing Field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1403–1412). New York, NY, USA: ACM. doi:10.1145/1978942.1979148
Rao, R. B., Fung, G., & Rosales, R. (2008). On the Dangers of Cross-Validation. An Experimental Evaluation. In SDM (pp. 588–596).
Ribeiro, F., Florencio, D., & Nascimento, V. (2011). Crowdsourcing subjective image quality evaluation. In Image Processing (ICIP), 2011 18th IEEE International Conference on (pp. 3097–3100). doi:10.1109/ICIP.2011.6116320
Ross, J., Irani, L., & Silberman, M. (2010). Who are the crowdworkers?: shifting demographics in mechanical turk. In CHI’10 Extended … (pp. 2863–2872). Retrieved from http://dl.acm.org/citation.cfm?id=1753873
Rouse, A. (2010). A preliminary taxonomy of crowdsourcing. Retrieved from http://aisel.aisnet.org/acis2010/76/
Schenk, E., & Guittard, C. (2011). Towards a characterization of crowdsourcing practices. Journal of Innovation Economics & Management, (1), 93–107.
Schiffauerova, A., & Thomson, V. (2006). A review of research on cost of quality models and best practices. International Journal of Quality & Reliability Management, 23(6), 647–669. doi:10.1108/02656710610672470
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 614–622). New York, NY, USA: ACM. doi:10.1145/1401890.1401965
Sorokin, A., & Forsyth, D. (2008). Utility data annotation with Amazon Mechanical Turk. In Computer Vision and Pattern Recognition Workshops, 2008. CVPRW ’08. IEEE Computer Society Conference on (pp. 1–8). doi:10.1109/CVPRW.2008.4562953
Surowiecki, J. (2005). The wisdom of crowds. Random House LLC.
Von Ahn, L., & Dabbish, L. (2008). Designing games with a purpose. Communications of the ACM, 51(8), 57. doi:10.1145/1378704.1378719
Voyer, R., Nygaard, V., Fitzgerald, W., Copperman, H., Suite, B. S., & Francisco, S. (2010). A Hybrid Model for Annotating Named Entity Training Corpora, (July), 243–246.
Vukovic, M. (2009). Crowdsourcing for Enterprises. In Congress on Services - I (pp. 686–692). doi:10.1109/SERVICES-I.2009.56
Vukovic, M., & Bartolini, C. (2010). Towards a research agenda for enterprise crowdsourcing. In Leveraging applications of formal methods, verification, and validation (pp. 425–434). Springer.
Welinder, P., & Perona, P. (2010). Online crowdsourcing: Rating annotators and obtaining cost-effective labels. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, 25–32. doi:10.1109/CVPRW.2010.5543189
Wieringa, R., & Morali, A. (2012). Technical action research as a validation method in information systems design science. In Design Science Research in Information Systems. Advances in Theory and Practice (pp. 220–238). Springer.
Wikipedia. (n.d.). Retrieved from www.wikipedia.org
Downloads
Published
How to Cite
Issue
Section
License
Authors who publish with this journal agree to the following terms:- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
