• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of amjphAmerican Journal of Public Health Web SiteAmerican Public Health Association Web SiteSubmissionsSubscriptionsAbout Us
Am J Public Health. 2009 October; 99(10): 1806–1810.
PMCID: PMC2741531

Reevaluating the Need for Concern Regarding Noncoverage Bias in Landline Surveys

Abstract

Objectives. We used recent data to reexamine whether the exclusion of adults from households with no telephone or only wireless phones may bias estimates derived from health-related telephone surveys.

Methods. We calculated the difference between estimates for the full population of adults and estimates for adults with landline phones; data were from the 2007 National Health Interview Survey.

Results. When data from landline telephone surveys were weighted to match demographic characteristics of the full population, bias was generally less than 2 percentage points (range = 0.1–2.4). However, among young adults and low-income adults, we found greater bias (range = 1.7–5.9) for estimates of health insurance, smoking, binge drinking, influenza vaccination, and having a usual place for care.

Conclusions. From 2004 to 2007, the potential for noncoverage bias increased. Bias can be reduced through weighting adjustments. Therefore, telephone surveys limited to landline households may still be appropriate for health surveys of all adults and for surveys of subpopulations regarding health status. However, for some behavioral risk factors and health care service use indicators, caution is warranted when using landline surveys to draw inferences about young or low-income adults.

In 2006, in this journal, we examined nationally representative survey data from 2004 and early 2005 to determine whether the exclusion of adults without landline telephones biased population-based estimates derived from health-related random-digit-dial telephone surveys.1 Noncoverage bias is determined both by the magnitude of the difference between persons with and without landline telephones for the variable of interest and by the percentage of persons without landline telephones in the population of interest.2 In 2004 and early 2005, only 7.2% of adults did not have landline telephones, and we concluded that “noncoverage is not presently a reason to reject the continued use of general population telephone surveys to help guide public health policy and program decisions.”1(p931)

In less than 3 years, the percentage of adults without landline telephones more than doubled. In 2007, 13.5% of adults lived in households with only wireless telephones, and an additional 1.7% of adults lived in households without any telephone service.3 Among certain subgroups, the percentage without landlines was even greater, reaching 30.6% for adults younger than 30 years and 21.6% for adults living in low-income households (defined as < 200% of the federal poverty level).

Our previously published conclusion, that noncoverage bias is not a concern,1,4 needed to be revisited. We therefore used more recent data to reexamine whether the exclusion of adults from households with no telephone or only wireless phones may bias estimates derived from health-related telephone surveys.

METHODS

The National Health Interview Survey (NHIS) remains the leading source for data on the wireless-only population of the United States,5,6,7 and every 6 months, the National Center for Health Statistics (NCHS) releases a report with the most up-to-date estimates available from the federal government concerning the size and characteristics of this population.3 This NCHS report describes the in-person household survey and the method for determining whether an adult lives in a household with landline telephones (landline household), with only cell phones (wireless-only household), or without any telephones (phoneless household).

We analyzed NHIS data collected in 2007. We selected health measures from data in the NHIS Sample Adult File, which includes health information for 1 civilian adult randomly selected from each interviewed family. In 2007, data on household telephone status and selected health measures were collected from 22 778 randomly selected adults. The overall response rate for the interview with the randomly selected adult was 67.8%.7

Variables

In accordance with the established NCHS definitions,3 we classified households as wireless-only if anyone living in the household had a working cellular telephone and if there were no working landline telephones inside the household. Individual ownership or use of the wireless telephone was not ascertained. All adults living in wireless-only households were classified as such. We used a similar approach to classify adults who lived in phoneless households with no working cellular or landline telephones.

Comparisons of adults from landline households with those from wireless-only households and those from phoneless households are regularly updated and disseminated by NCHS for 13 key indicators of health status, health behaviors, health care service use, and health care access.3 These indicators are a subset of the key adult health indicators published quarterly online by the NHIS Early Release Program.8 We used the specifications for these indicators that were published online.3,8

Analyses

We calculated the potential noncoverage bias in landline telephone surveys that exclude wireless telephones and phoneless households by subtracting the population estimate for all adults from the estimated value for adults living in landline households. The population estimate and an unadjusted value for adults living in landline households were estimated from the published NHIS sampling weights without further adjustments.

Previous studies showed that young adults, Hispanic adults, and adults living in poverty are more likely to live in wireless-only households.1,3 These demographic differences may explain some of the differences in health between adults with and without landlines. Therefore, we estimated an adjusted value for adults from landline households (technically, a predictive margin9) with a logistic regression that controlled for differences between adults from landline and nonlandline households on 4 demographic characteristics (age, race/ethnicity, gender, and education). The adjusted bias estimate (i.e., the predictive margin minus the true population value) approximated the actual bias that might be observed from a telephone survey that used sampling weights for adults living in landline households, after those weights were adjusted to match census estimates of the demographic composition of the population with and without landlines for these 4 demographic characteristics. A more complete explanation of this technique is available elsewhere.4 These 4 characteristics were selected because they are used for adjusting sampling weights for the Behavioral Risk Factor Surveillance System.10

Predictive margins, bias estimates, and standard errors were produced with SUDAAN software to account for the complex sample design of NHIS.11 Statistical significance tests (α = 0.05) to determine whether absolute bias was greater than zero were calculated with a standard error of the difference term that accounted for covariance between the 2 groups (adults from landline households and all adults). Then, to determine whether unadjusted and adjusted bias estimates differed from each other, we used a pooled standard error of the difference term to compare the 2 bias estimates.

Because biases for subgroups of the population may not be adequately reflected in observed biases for the overall population, we repeated the analyses for 2 subgroups: young adults aged 18 to 29 years and adults living in low-income households.

RESULTS

Our previous analysis of data from 2004 and 2005 found that the potential noncoverage bias in landline-based telephone surveys did not exceed 2 percentage points for any health-related variable, and we concluded that differences this small were not practically significant.1 By that admittedly debatable standard, random-digit-dial telephone surveys that excluded cell phone numbers were still viable in 2007 for health status surveys of adults (Tables 1 and and2).2). The noncoverage bias expected for these measures generally did not exceed 1 percentage point, even when sampling weights were unadjusted and when the sample was limited to populations more likely to live without landline telephones. The unadjusted bias estimate for self-reported health status for low-income adults was an important exception.

TABLE 1
Unadjusted and Adjusted Estimates Of Noncoverage Bias for Selected Measures of Health-Related Behaviors, Health Status, and Health Care Service Use: United States, 2007
TABLE 2
Unadjusted and Adjusted Estimates of Noncoverage Bias for Selected Measures of Health-Related Behaviors, Health Status, and Health Care Service Use, by Age and Household Poverty Status: United States, 2007

In addition, when the population of interest included all adults, the exclusion of cell phone numbers from random-digit-dial telephone surveys had little practical effect on adjusted estimates of health-related behaviors and health care service use. After adjustment, we found noncoverage bias greater than 1 percentage point (but less than 2.5 percentage points) only for binge drinking, having a usual place for care, and influenza vaccination. We had substantially similar findings, but with slightly higher levels of potential noncoverage bias, for low-income adults following adjustment.

However, when our population of interest was young adults, noncoverage was a reason to question the validity of landline-only surveys for studies of health-related behaviors and health care service use. In both unadjusted and adjusted estimates, the bias was greatest for binge drinking. Landline-only surveys could underestimate the prevalence of binge drinking among young adults by approximately 6 percentage points. We also found practically significant bias for smoking, HIV testing, financial barriers to medical care, and having a usual place for medical care. Adjustments did not significantly reduce the bias for these estimates.

The viability of landline-only surveys for health insurance estimates is also in doubt. With weighting adjustments, the noninsured rate was underestimated by 0.9 percentage points for all adults, 1.7 percentage points for low-income adults, and 2.9 percentage points for young adults. For all adults and for low-income adults, underestimations of this magnitude would qualify as practically insignificant by our original standard.1 However, annual changes in insurance rates that are smaller than this magnitude have been considered important.12 Without the weighting adjustments, the magnitude of the bias was significantly greater for all adults and for low-income adults.

DISCUSSION

For every measure, we found greater unadjusted noncoverage bias in the 2007 data than in the 2004 to 2005 data.1 The bias more than doubled for estimates of binge drinking, flu vaccination, HIV testing, and financial barriers to needed medical care. The bias nearly doubled for estimates of smoking, having a usual place for care, and health insurance. The increased bias is attributable largely to the rapid increase in the prevalence of adults from wireless-only households. Differences between adults with and without landline telephones did not change substantially.1(Table 3),3(Table 4)

The adjustment used to mimic the effect of demographic weighting attenuated, but did not eliminate, the potential noncoverage bias for some estimates. It is likely that noncoverage bias could be reduced further by the application of other poststratification adjustments. In the past decade, many telephone surveys increased the weights of data from transient telephone households (those that had an interruption in service during the previous year) to compensate for the exclusion of phoneless households.1315 Preliminary studies suggest that surveys may be able to compensate for the exclusion of wireless-only households by increasing the weights for wireless-mostly households (those landline households that receive nearly all calls on cell phones)16 or for transient wireless-only households (those that had only wireless phones at some time during the past year).17,18 Additional research will be necessary to prove the worth of such adjustments.

Data collection systems that have relied on random-digit-dial surveys for years have been testing methods for including persons from wireless-only households in their samples.19 For example, the Behavioral Risk Factor Surveillance System (whose findings of declines in smoking and drinking rates have been attributed to noncoverage bias20) recently conducted field tests dialing cell phone numbers21 and contacting adults from wireless-only households (and others) by mail.22 Best practices for cell phone surveys are not yet known, but a task force of the American Association for Public Opinion Research recently published an overview of the issues and challenges, including those related to coverage, sampling, nonresponse, measurement, weighting, ethics, and legal restrictions.23 When it is not feasible to sample persons living in wireless-only households, the task force recommended that researchers discuss how their exclusion may affect the results. Our study should help some researchers with that discussion.

Random-digit-dial surveys with only landline telephone numbers should not be rejected out of hand as a data collection method. This method of sampling and this mode of data collection are still appropriate when exclusion of wireless-only households does not lead to noncoverage bias. This will be true for some survey topics and for some populations, but studies such as ours should be conducted and updated often to be certain when this is the case. Moreover, some topics and populations for which landline-only telephone surveys may not yield valid results now may become acceptable because of advances in sample weighting methods, other methodological innovations, and even changes in the composition of the landline population.

Despite our findings, there may never be a reliable list of topics and populations known to be amenable to a method that excludes wireless-only households, especially because the prevalence of such households continues to grow at a strong and steady pace. The regular inclusion of other samples (of cell phone numbers or of addresses) along with—or in place of—landline telephone numbers may not always be required to generate valid conclusions from the data collected but may be necessary for peace of mind.

Acknowledgments

We thank Marcie L. Cynamon for her continuing support of this work and for her supervision on this project.

Human Participant Protection

Data collection procedures for the National Health Interview Survey were approved by the institutional review board at the National Center for Health Statistics, Centers for Disease Control and Prevention. No approval was necessary for analyses of deidentified data.

References

1. Blumberg SJ, Luke JV, Cynamon ML. Telephone coverage and health survey estimates: evaluating the need for concern about wireless substitution. Am J Public Health 2006;96:926–931. [PMC free article] [PubMed]
2. Anderson JE, Nelson DE, Wilson RW. Telephone coverage and measurement of health risk indicators: data from the National Health Interview Survey. Am J Public Health 1998;88:1392–1395. [PMC free article] [PubMed]
3. Blumberg SJ, Luke JV. Wireless Substitution: Early release of Estimates Based on Data From the National Health Interview Survey, July–December 2007. Hyattsville, MD: National Center for Health Statistics; 2008. Available at: http://www.cdc.gov/nchs/about/major/nhis/releases.htm. Accessed September 26, 2008.
4. Blumberg SJ, Luke JV. Coverage bias in traditional telephone surveys of low-income and young adults. Public Opin Q 2007;71:734–749.
5. Implementation of Section 6002(b) of the Omnibus Budget Reconciliation Act of 1993, Annual Report and Analysis of Competitive Market Conditions With Respect to Commercial Mobile Services, Twelfth Report. Washington, DC: Federal Communications Commission; 2008. Available at: http://hraunfoss.fcc.gov/edocs_public/attachmatch/FCC-08-28A1.pdf. Accessed September 26, 2008.
6. Wireless Quick Facts. Washington, DC: CTIA—The Wireless Association; 2008. Available at: http://www.ctia.org/advocacy/research/index.cfm/AID/10323. Accessed September 26, 2008.
7. 2007 National Health Interview Survey Public Use Data Release. Hyattsville, MD: National Center for Health Statistics; 2008. Available at: ftp://ftp.cdc.gov/pub/Health_Statistics/NCHS/Dataset_Documentation/NHIS/2007/srvydesc.pdf. Accessed September 26, 2008.
8. Heyman KM, Schiller JS, Barnes P. Early Release of Selected Estimates Based on Data From the 2007 National Health Interview Survey. Hyattsville, MD: National Center for Health Statistics; 2008. Available at: http://www.cdc.gov/nchs/about/major/nhis/releases.htm. Accessed September 26, 2008.
9. Graubard BI, Korn EL. Predictive margins with survey data. Biometrics 1999;55:652–659. [PubMed]
10. Battaglia MP, Frankel MR, Link MW. Improving standard poststratification techniques for random-digit-dialing telephone surveys. Surv Res Methods 2008;2:11–19.
11. SUDAAN User's Manual, Release 9.0. Research Triangle Park, NC: Research Triangle Institute; 2005.
12. DeNavas-Walt C, Proctor BD, Smith JC. Income, Poverty, and Health Insurance Coverage in the United States: 2007. Washington, DC: US Census Bureau; 2008. Current Population Reports, P60-235. Available at: http://www.census.gov/prod/2008pubs/p60-235.pdf. Accessed September 26, 2008.
13. Brick JM, Waksberg J, Keeter S. Using data on interruptions in telephone service as coverage adjustments. Surv Methodol 1996;22:185–197.
14. Davern M, Lepkowski J, Call KT, et al. Telephone service interruption weighting adjustments for state health insurance surveys. Inquiry 2004;41:280–290. [PubMed]
15. Frankel MR, Srinath KP, Hoaglin DC, et al. Adjustments for non-telephone bias in random-digit-dialing surveys. Stat Med 2003;22:1611–1626. [PubMed]
16. Blumberg SJ, Luke JV. Wireless-mostly households: estimates from the 2007 National Health Interview Survey. Paper presented at: Annual Meeting of the American Association for Public Opinion Research; May 15–18, 2008; New Orleans, LA.
17. Blumberg SJ, Luke JV, Cynamon ML, Frankel MR. Recent trends in household telephone coverage in the United States. Lepkowski JM, Tucker C, Brick JM, et al., editors. eds.Advances in Telephone Survey Methodology. New York, NY: Wiley; 2008:56–86.
18. Jay ED, DiCamillo M. Improving the representativeness of RDD telephone surveys by accounting for recent cell phone-only households. Paper presented at: Second International Conference on Telephone Survey Methodology; January 11–15, 2006; Miami, FL.
19. Lavrakas PJ, Shuttles CD, Steeh C, Fienberg H. The state of surveying cell phone numbers in the United States: 2007 and beyond. Public Opin Q 2007;71:840–854.
20. Delnevo CD, Gundersen DA, Hagman BT. Declining estimated prevalence of alcohol drinking and smoking among young adults nationally: artifacts of sample undercoverage? Am J Epidemiol 2008;167:15–19. [PubMed]
21. Link MW, Battaglia MP, Frankel MR, Osborn L, Mokdad AH. Reaching the U.S. cell phone generation: comparison of cell phone survey results with an ongoing landline telephone survey. Public Opin Q 2007;71:814–839.
22. Link MW, Battaglia MP, Frankel MR, Osborn L, Mokdad AH. Address-based versus random-digit-dial surveys: comparison of key health and risk indicators. Am J Epidemiol 2006;164:1019–1025. [PubMed]
23. AAPOR Cell Phone Task Force Guidelines and Considerations for Survey Researchers When Planning and Conducting RDD and Other Telephone Surveys in the U.S. With Respondents Reached via Cell Phone Numbers. Lenexa, KS: American Association for Public Opinion Research; 2008. Available at: http://www.aapor.org/uploads/Final_AAPOR_Cell_Phone_TF_report_041208.pdf. Accessed September 26, 2008.

Articles from American Journal of Public Health are provided here courtesy of American Public Health Association

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...