Send to

Choose Destination
Conserv Biol. 2016 Jun;30(3):520-31. doi: 10.1111/cobi.12695. Epub 2016 Apr 25.

A generalized approach for producing, quantifying, and validating citizen science data from wildlife images.

Author information

Department of Ecology, Evolution and Behavior, University of Minnesota, Saint Paul, MN 55108, U.S.A.
Department of Physics, University of Oxford, Denys Wilkinson Building, Oxford, OX1 3RH, U.K.
Current address: Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, MA 02138, U.S.A.


Citizen science has the potential to expand the scope and scale of research in ecology and conservation, but many professional researchers remain skeptical of data produced by nonexperts. We devised an approach for producing accurate, reliable data from untrained, nonexpert volunteers. On the citizen science website, more than 28,000 volunteers classified 1.51 million images taken in a large-scale camera-trap survey in Serengeti National Park, Tanzania. Each image was circulated to, on average, 27 volunteers, and their classifications were aggregated using a simple plurality algorithm. We validated the aggregated answers against a data set of 3829 images verified by experts and calculated 3 certainty metrics-level of agreement among classifications (evenness), fraction of classifications supporting the aggregated answer (fraction support), and fraction of classifiers who reported "nothing here" for an image that was ultimately classified as containing an animal (fraction blank)-to measure confidence that an aggregated answer was correct. Overall, aggregated volunteer answers agreed with the expert-verified data on 98% of images, but accuracy differed by species commonness such that rare species had higher rates of false positives and false negatives. Easily calculated analysis of variance and post-hoc Tukey tests indicated that the certainty metrics were significant indicators of whether each image was correctly classified or classifiable. Thus, the certainty metrics can be used to identify images for expert review. Bootstrapping analyses further indicated that 90% of images were correctly classified with just 5 volunteers per image. Species classifications based on the plurality vote of multiple citizen scientists can provide a reliable foundation for large-scale monitoring of African wildlife.


Snapshot Serengeti; Zooniverse; big data; camera traps; conjunto de datos; crowdsourcing; cámaras trampa; data aggregation; data validation; datos grandes; image processing; procesamiento de imágenes; validación de datos

[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Wiley Icon for PubMed Central
Loading ...
Support Center