Format

Send to

Choose Destination
Scientometrics. 2016;107:839-856. Epub 2016 Feb 9.

Estimating search engine index size variability: a 9-year longitudinal study.

Author information

1
Centre for Language Studies, Radboud University, PO Box 9103, 6500 HD Nijmegen, The Netherlands.
2
Department of Communication, Aalborg University Copenhagen, A.C. Meyers Vænge 15, 2450 Copenhagen, Denmark.
3
De Kunder Internet Media, Toernooiveld 100, 6525 EC Nijmegen, The Netherlands.

Abstract

One of the determining factors of the quality of Web search engines is the size of their index. In addition to its influence on search result quality, the size of the indexed Web can also tell us something about which parts of the WWW are directly accessible to the everyday user. We propose a novel method of estimating the size of a Web search engine's index by extrapolating from document frequencies of words observed in a large static corpus of Web pages. In addition, we provide a unique longitudinal perspective on the size of Google and Bing's indices over a nine-year period, from March 2006 until January 2015. We find that index size estimates of these two search engines tend to vary dramatically over time, with Google generally possessing a larger index than Bing. This result raises doubts about the reliability of previous one-off estimates of the size of the indexed Web. We find that much, if not all of this variability can be explained by changes in the indexing and ranking infrastructure of Google and Bing. This casts further doubt on whether Web search engines can be used reliably for cross-sectional webometric studies.

KEYWORDS:

Longitudinal study; Search engine index; Webometrics

Supplemental Content

Full text links

Icon for PubMed Central
Loading ...
Support Center