Human Computation vs. Machine Learning: an Experimental Comparison for Image Classification
DOI:
https://doi.org/10.15346/hc.v5i1.2Keywords:
Image classification, Game with a PurposeAbstract
Image classification is a classical task heavily studied in computer vision and widely required in many concrete scientific and industrial scenarios. Is it better to rely on human eyes, thus asking people to classify pictures, or to train a machine learning system to automatically solve the task? The answer largely depends on the specific case and the required accuracy: humans may be more reliable - especially if they are domain experts - but automatic processing can be cheaper, even if less capable to demonstrate an "intelligent" behaviour.In this paper, we present an experimental comparison of different Human Computation and Machine Learning approaches to solve the same image classification task on a set of pictures used in light pollution research. We illustrate the adopted methods and the obtained results and we compare and contrast them in order to come up with a long term combined strategy to address the specific issue at scale: while it is hard to ensure a long-term engagement of users to exclusively rely on the Human Computation approach, the human classification is indispensable to overcome the "cold start" problem of automated data modelling.References
Breiman, L. (2001). Random forests. Machine learning 45, 1 (2001), 5–32.
Celino, I, Contessa, S, Corubolo, M, Dell'Aglio, D, Della Valle, E, Fumeo, S, and Krüger, T. (2012). Linking Smart Cities Datasets with Human Computation: the case of UrbanMatch. In Proceedings of the 11th International Conference on The Semantic Web. Springer-Verlag, 34–49.
Celino, I, Fiano, A, and Re Calegari, G. (2016). Games Release (initial release). Technical Report. STARS4ALL project deliverable, https://figshare.com/s/5bd9c9f96c8dcee121b5.
Debevec, P. (2008). A median cut algorithm for light probe sampling. In ACM SIGGRAPH 2008 classes. 33.
Donahue, J, Jia, Y, Vinyals, O, Hoffman, J, Zhang, N, Tzeng, E, and Darrell, T. (2014). DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014. 647–655.
Hollan, J. (2009). What is light pollution, and how do we quantify it? Technical Report. N. Copernicus Observatory and Planetarium, Brno.
Howe, J. (2008). Crowdsourcing: How the power of the crowd is driving the future of business. Random House.
Irwin, A. (1995). Citizen science: A study of people, expertise and sustainable development. Psychology Press.
Kotsiantis, S. B, Zaharakis, I, and Pintelas, P. (2007). Supervised machine learning: A review of classification techniques. (2007).
Krizhevsky, A, Sutskever, I, and Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems. 1097–1105.
Law, E and Ahn, L. v. (2011). Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning 5, 3 (2011), 1–121.
Lin, Y, Lv, F, Zhu, S, Yang, M, Cour, T, Yu, K, Cao, L, and Huang, T. (2011). Large-scale image classification: fast feature extraction and svm training. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. 1689–1696.
Nixon, M and Aguado, A. S. (2008). Feature Extraction & Image Processing, Second Edition (2nd ed.). Academic Press.
Rand, W. M. (1971). Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association 66, 336 (1971), 846–850.
Russakovsky, O, Deng, J, Su, H, Krause, J, Satheesh, S, Ma, S, Huang, Z, Karpathy, A, Khosla, A, Bernstein, M, Berg, A. C, and Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115, 3 (2015), 211–252. DOI:http://dx.doi.org/10.1007/s11263-015-0816-y
Sánchez de Miguel, A. (2015). Variación espacial, temporal y espectral de la contaminación lumínica y sus fuentes: Metodología y resultados. Ph.D. Dissertation. Universidad Complutense de Madrid.
Settles, B. (2012). Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning 6, 1 (2012), 1–114.
Smith, J. R and Chang, S.-F. (1995). Single color extraction and image query. In Image processing, 1995. Proceedings., International conference on, Vol. 3. IEEE, 528–531.
Szegedy, C, Liu, W, Jia, Y, Sermanet, P, Reed, S, Anguelov, D, Erhan, D, Vanhoucke, V, and Rabinovich, A. (2015). Going Deeper with Convolutions. In Computer Vision and Pattern Recognition (CVPR).
Von Ahn, L. (2006). Games with a purpose. Computer 39, 6 (2006), 92–94.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2018 Human Computation
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
