Human Computation vs. Machine Learning: an Experimental Comparison for Image Classification




Image classification, Game with a Purpose


Image classification is a classical task heavily studied in computer vision and widely required in many concrete scientific and industrial scenarios. Is it better to rely on human eyes, thus asking people to classify pictures, or to train a machine learning system to automatically solve the task? The answer largely depends on the specific case and the required accuracy: humans may be more reliable - especially if they are domain experts - but automatic processing can be cheaper, even if less capable to demonstrate an "intelligent" behaviour.In this paper, we present an experimental comparison of different Human Computation and Machine Learning approaches to solve the same image classification task on a set of pictures used in light pollution research. We illustrate the adopted methods and the obtained results and we compare and contrast them in order to come up with a long term combined strategy to address the specific issue at scale: while it is hard to ensure a long-term engagement of users to exclusively rely on the Human Computation approach, the human classification is indispensable to overcome the "cold start" problem of automated data modelling.

Author Biography

Irene Celino, Cefriel - Politecnico di Milano

Research Manager in the area of Data and Citizen Science


Breiman, L. (2001). Random forests. Machine learning 45, 1 (2001), 5–32.

Celino, I, Contessa, S, Corubolo, M, Dell'Aglio, D, Della Valle, E, Fumeo, S, and Krüger, T. (2012). Linking Smart Cities Datasets with Human Computation: the case of UrbanMatch. In Proceedings of the 11th International Conference on The Semantic Web. Springer-Verlag, 34–49.

Celino, I, Fiano, A, and Re Calegari, G. (2016). Games Release (initial release). Technical Report. STARS4ALL project deliverable,

Debevec, P. (2008). A median cut algorithm for light probe sampling. In ACM SIGGRAPH 2008 classes. 33.

Donahue, J, Jia, Y, Vinyals, O, Hoffman, J, Zhang, N, Tzeng, E, and Darrell, T. (2014). DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014. 647–655.

Hollan, J. (2009). What is light pollution, and how do we quantify it? Technical Report. N. Copernicus Observatory and Planetarium, Brno.

Howe, J. (2008). Crowdsourcing: How the power of the crowd is driving the future of business. Random House.

Irwin, A. (1995). Citizen science: A study of people, expertise and sustainable development. Psychology Press.

Kotsiantis, S. B, Zaharakis, I, and Pintelas, P. (2007). Supervised machine learning: A review of classification techniques. (2007).

Krizhevsky, A, Sutskever, I, and Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems. 1097–1105.

Law, E and Ahn, L. v. (2011). Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning 5, 3 (2011), 1–121.

Lin, Y, Lv, F, Zhu, S, Yang, M, Cour, T, Yu, K, Cao, L, and Huang, T. (2011). Large-scale image classification: fast feature extraction and svm training. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. 1689–1696.

Nixon, M and Aguado, A. S. (2008). Feature Extraction & Image Processing, Second Edition (2nd ed.). Academic Press.

Rand, W. M. (1971). Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association 66, 336 (1971), 846–850.

Russakovsky, O, Deng, J, Su, H, Krause, J, Satheesh, S, Ma, S, Huang, Z, Karpathy, A, Khosla, A, Bernstein, M, Berg, A. C, and Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115, 3 (2015), 211–252. DOI:

Sánchez de Miguel, A. (2015). Variación espacial, temporal y espectral de la contaminación lumínica y sus fuentes: Metodología y resultados. Ph.D. Dissertation. Universidad Complutense de Madrid.

Settles, B. (2012). Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning 6, 1 (2012), 1–114.

Smith, J. R and Chang, S.-F. (1995). Single color extraction and image query. In Image processing, 1995. Proceedings., International conference on, Vol. 3. IEEE, 528–531.

Szegedy, C, Liu, W, Jia, Y, Sermanet, P, Reed, S, Anguelov, D, Erhan, D, Vanhoucke, V, and Rabinovich, A. (2015). Going Deeper with Convolutions. In Computer Vision and Pattern Recognition (CVPR).

Von Ahn, L. (2006). Games with a purpose. Computer 39, 6 (2006), 92–94.




How to Cite

Re Calegari, G., Nasi, G., & Celino, I. (2018). Human Computation vs. Machine Learning: an Experimental Comparison for Image Classification. Human Computation, 5(1), 13-30.