• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Public Health. Author manuscript; available in PMC Apr 1, 2010.
Published in final edited form as:
PMCID: PMC2718054
NIHMSID: NIHMS119772

Reliability of health-related quality-of-life indicators in cancer survivors from a population-based sample, 2005, BRFSS

Summary

Objective

The current emphasis in cancer survivorship research, which includes health-related quality of life (HRQoL), drives the need to monitor the nation’s cancer burden. Routine, ongoing public health surveillance tools, such as the Behavioral Risk Factor Surveillance System (BRFSS), may be relevant for this purpose.

Study design

A subsample of the 2005 Missouri BRFSS was used to estimate test–retest reliability of HRQoL questions among persons who did and did not report a personal cancer history.

Methods

Retest interviews were conducted by telephone 14–21 days after the initial data collection (n=540, 67% response rate). Reliability was estimated overall and by cancer history using intraclass correlation coefficients (ICCs) and kappa statistics.

Results

The majority of retest respondents were White, female and married, with 13% reporting a history of cancer. Overall, point estimates of the reliability coefficients ranged from moderate to excellent (κ=0.57–0.75). There were no statistically significant differences in test–retest reliability between persons with and without a history of cancer, except for self-reported pain (ICC=0.59 and ICC=0.78, respectively).

Conclusions

In general, BRFSS questions appear to have adequate reliability for monitoring HRQoL in this community-dwelling population, regardless of cancer history.

Keywords: Reproducibility of results, Neoplasms, Quality of life, Behavioral Risk Factor Surveillance System

Introduction

One in four deaths in the USA is attributed to cancer.1 Cancer is the second leading cause of death in the USA and, as such, is a leading public health concern.1 Cancer is distinguishable from other diseases by its related symptoms, such as high levels of pain, debilitating treatment sequelae2 and its long-term impact on function and well-being.3 Given advances in early detection and treatment, cancer survivorship research now focuses on late effects of treatment, quality of life, follow-up care and post-treatment surveillance.4

The National Cancer Institute’s Cancer Surveillance Research Implementation Plan5 has outlined population monitoring as an avenue for measuring the nation’s cancer burden. This includes monitoring outcomes beyond incidence, survival and mortality, such as health-related quality of life (HRQoL). Comparing HRQoL in cancer populations with the population at large requires a comparison with normative groups using a generic measure.2,3 Further, when the objective is to monitor the nation’s cancer burden for policy guidance, a HRQoL metric used in multiple diseases and the general population may be warranted.3 Few national data sources currently exist that can be used to monitor HRQoL among cancer survivors6 in comparison with the general population.

One data source for monitoring HRQoL among cancer survivors nationally as well as locally may be the Behavioral Risk Factor Surveillance System (BRFSS). A review of more than 60 studies from the Centers for Disease Control and Prevention (CDC) websites7,8 and of literature from a search for studies that assessed data quality, validity and reliability of indicators using the BRFSS found few reports on the reliability of HRQoL. A comprehensive review of BRFSS reliability from 20019 reported moderate reliability for the general health question (κ=0.42–0.47). However, this general health measure resulted from a study of 35–63-year-old adults in Finland, with a 1-year post test.10 The review also reported that the general health scale from the Medical Outcomes Study 36-Item Short Form had an intraclass correlation coefficient (ICC) of 0.87.11 No reliability studies had been undertaken regarding general health or quality of life (poor physical or mental health days in the past month) from the BRFSS at the time of the review.9 Only one known BRFSS study has since examined the reliability of HRQoL indicators, using a population-based sample in Missouri.12

The population of cancer survivors in the USA is increasing steadily, reaching almost 10 million by 2001, or 3.5% of the population.13 Due to treatment side-effects, late complications and secondary cancers, the reliability of HRQoL measures may differ between cancer survivors and the general population. To the authors’ knowledge, no population-based study has compared the reliability of HRQoL questions from the BRFSS across cancer history. This paper describes the results of a test–retest study of HRQoL indicators across cancer history using a subsample of the Missouri BRFSS participants.

Methods

Sample

The BRFSS is an ongoing state-based, computer-assisted telephone survey system used to track health conditions and risk behaviours. It is implemented annually by the 50 state health departments, the District of Columbia, Puerto Rico, Guam and the US Virgin Islands in collaboration with the CDC. The BRFSS is a cross-sectional survey that uses random-digit-dial techniques to survey non-institutionalized adults aged ≥18 years in the USA (http://www.cdc.gov/brfss). During 2005, Missouri completed 5164 BRFSS surveys14,15 for a median Council of American Survey Research Organizations (CASRO) response rate of 57.8% and cooperation rate of 78.5%.16 Of these participants, 867 Missourians were interviewed in April or May (CASRO rate: 55.2% and 55.1%; cooperation rate: 76.0% and 75.7%, respectively).17 Comparatively, the median CASRO rate for 2005 for all states was 51.1% (range 34.6–67.4%) and the median cooperation rate was 75.1% (range 58.7–85.3%).16 Of the 867 respondents potentially eligible for the retest interview, 56 were excluded from being re-interviewed for not completing or only partially completing (through the demographics section)18 the initial survey. The remaining 811 participants were invited to participate in a second (retest) interview 14–21 days after completing their initial BRFSS survey. All calls were made by trained interviewers at the University of Missouri-Columbia according to CDC’s standard BRFSS protocol.18

Instrument and definitions

Cancer history was determined by a ‘yes’ or ‘no’ response to ‘Have you ever been told by a doctor, nurse or other health professional that you had cancer?’

Several HRQoL indicators were collected, including self-rated health, social and emotional support, life satisfaction, physical and mental health status, pain and activity limitations. Response options to the general health question, ‘Would you say that, in general, your health is’ were: ‘poor’, ‘fair’, ‘good’, ‘very good’ or ‘excellent’. Response options to ‘How often do you get the social and emotional support you need?’ were ‘always’, ‘usually’, ‘sometimes’, ‘rarely’ or ‘never.’ Life satisfaction response options (‘In general, how satisfied are you with your life?’) ranged from ‘very satisfied’ to ‘very dissatisfied’. The following were treated as continuous variables: ‘Now thinking about your physical health, which includes physical illness and injury, for how many days during the past 30 days was your physical health not good?’ (physical distress); ‘Now thinking about your mental health, which includes stress, depression and problems with emotions, for how many days during the past 30 days was your mental health not good?’ (mental distress); ‘During the past 30 days, for about how many days did poor physical or mental health keep you from doing your usual activities, such as self-care, work or recreation?’; and ‘During the past 30 days, for about how many days did pain make it hard for you to do your usual activities such as self-care, work or recreation?’. ‘Yes’ or ‘no’ responses to activity limitation, asked as ‘Are you limited in any way in any activities because of physical, mental or emotional problems?’, were examined. Frequent mental distress was dichotomized at ≥14 days of poor mental health.19 Frequent physical distress, frequent poor physical or mental health, and frequent pain were also calculated.20

Several sociodemographic factors characterized the sample, including marital status, race, employment, age, education, income and gender. Marital status was categorized as married compared with those divorced, widowed, separated, never married or a member of an unmarried couple. Black and other races were combined due to small cell sizes. Employment status was categorized as employed or self-employed compared with out of work, homemaker, student, retired or unable to work.

Statistical analysis

Comparisons of descriptive characteristics across cancer history used Chi-squared or Fisher’s exact tests. The authors examined reliability for the HRQoL questions overall and stratified by cancer history. As the vast majority of cancers occur among persons aged ≥55 years21 and to reduce the difference in age between persons with and without cancer, reliability across cancer history was also examined among respondents aged ≥55 years. Cohen’s simple kappa statistic was used for dichotomous variables, weighted kappa was used for ordinal variables, and ICC from a one-way random effects model22 was used for continuous variables. The following rule of thumb was used to evaluate reliability: values >0.75 generally reflect excellent agreement, values <0.4 reflect poor agreement, and intermediate values reflect moderate agreement.23,24 Ninety-five percent confidence intervals (CI) accompany kappa statistics and ICCs. Significance tests were used to further examine statistically significant differences in reliability among persons who reported a cancer history compared with those who did not.

Only prevalence estimates from the entire Missouri BRFSS sample are weighted to the Missouri population. Retest data are not treated as a probability sample and therefore the BRFSS’ complex sampling design is not considered.25,26 Proportions are calculated for retest participants; therefore, percentages may not sum to 100 due to rounding or missing values. For demographic information alone, if data were missing from the initial interview, the retest data were used. For cancer history, responses at the initial interview were used for stratified analyses. All ‘don’t know/not sure’ and ‘refused’ responses were set to missing. Data were analysed using SAS/Stat Version 9.1 (Cary, NC, USA).27

Results

Out of 811 eligible subjects, 540 completed the retest survey (67% response rate). Compared with non-respondents, respondents were more likely to be White, aged ≥55 years and not employed (P<0.05). No differences were found for Hispanic ethnicity, education, marital status, gender or cancer history (results not shown). Table 1 describes the overall 2005 Missouri BRFSS interviews, as well as the respondents invited to complete and who completed the retest. Almost half of the retest respondents were aged ≥55 years, 10% reported Black or other race, over half (55%) were married, nearly 57% had at least some college education, half were employed or self-employed, the majority were female (64%), half reported an annual household income <$35,000 (49%), and 13% had been told by a health professional that they had cancer. The median number of days between interviews was 12 (range 7–35 days).

Table 1
Descriptive characteristicsa of Missouri 2005 Behavioral Risk Factor Surveillance System (BRFSS) and retest participants, overall and by cancer history.

Table 1 also compares respondents in the retest interview across cancer history. Those with a cancer diagnosis were significantly more likely to be aged ≥55 years (P<0.05) and not employed (P<0.05). Cancer history was not significantly different across other demographics.

Table 2 describes the HRQoL indicators at the initial interview and retest. Summarizing both the initial and retest interviews, the majority of the sample (75.2–79.4%) reported that their general health was good or better. Out of the last 30 days, respondents at the initial interview and retest reported respective means of 5.2 and 4.9 days of physical distress, 4.0 and 3.1 days of mental distress, 6.3 and 5.5 days of poor physical or mental health prohibiting usual activities, and 5.5 and 5.2 days of pain making it difficult to perform usual activities. Approximately 29% of the sample reported activity limitation. Respondents reported that they ‘sometimes’, ‘rarely’ or ‘never’ received the social and emotional support needed 22.6 and 18.9% of the time (respective to the initial and retest interviews). Approximately 8% of the sample was generally dissatisfied or very dissatisfied with their life. Frequent mental distress was less commonly reported than frequent physical distress or frequent pain.

Table 2
Test–retest reliability of health-related quality-of-life questions from the Missouri Behavioral Risk Factor Surveillance System retest sample, overall and by cancer history (n=540), 2005.

A personal cancer history demonstrated excellent reliability, although it was not perfect (κ=0.91, 95% CI 0.86–0.96). Overall results for activity limitation and pain suggest excellent reliability, while the other indicators demonstrated moderate reliability (Table 2). None of the measures indicated poor reliability. Across cancer history, reliability estimates did not differ statistically except for the pain measure. Results suggest significantly higher reliability for pain across cancer history, both overall (cancer history: ICC=0.59, 95% CI 0.41–0.73; no cancer history: ICC=0.78, 95% CI 0.74–0.81) and when restricting the sample to those aged ≥55 years (cancer history: ICC=0.51, 95% CI 0.28–0.69; no cancer history: ICC=0.73, 95% CI 0.65–0.79). A significant difference was not found for frequent pain.

Discussion

This is the first population-based study to compare reliability estimates of HRQoL indicators between cancer survivors and the general population using the BRFSS. Overall, reliability was moderate to excellent. The overall findings are consistent with those of another study examining HRQoL from the Missouri BRFSS.12 Additionally, the present finding that cancer diagnosis demonstrated excellent reliability (κ=0.91) is consistent with findings demonstrated for similarly worded BRFSS questions, such as hypertension28 and diabetes28 (κ=0.82 and 0.86, respectively). This study further adds that reliability for HRQoL measures across cancer history was generally moderate to excellent in this population, both overall and among respondents aged ≥55 years.

Using the BRFSS for cancer surveillance, while cross-sectional, may provide additional information for the National Cancer Institute’s Cancer Surveillance Research Implementation Plan.2,5 Such population-based monitoring could, for example, take the form of capturing ongoing, up-to-date HRQoL information on a large scale.2 Changes in health status over time, and variation by geography or population subgroups, including racial and ethnic disparities, could help to identify targeted areas for further investigation or intervention.2

This study has several limitations to consider. First, as these retest results using Missouri BRFSS data were not treated as a probability sample, statistical inferences are not made beyond this study. However, these results do provide an important look at how the HRQoL measures perform across cancer history in this population. Second, the sample size for those reporting a cancer history was modest in the retest sample. Confidence intervals were reported to assist with interpreting the precision of the estimates, especially for those with cell sizes <50. It would be important to confirm these findings in a larger sample and further examine the reliability of these HRQoL indicators based on cancer type, cancer stage and length of time since diagnosis. Third, the time interval between the BRFSS interview and retest ranged from 7 to 35 days. While this interval is consistent with other studies,12,25,28 longer intervals may be associated with greater variation and therefore lower reliability. These data cannot explain whether the lower reliability for the pain measure among those diagnosed with cancer reflects a change in the respondent’s treatment; however, the wording of questions regarding pain in the last 30 days may minimize any impact of the time interval. Despite these limitations, this study has several strengths including a strong response rate, consistency of results, trained interviewers who regularly call for the BRFSS, and some information to address a gap in the literature.

In summary, this study provides an initial look at estimates of the test–retest reliability of HRQoL questions from the BRFSS across cancer history. Measurement issues surrounding cancer survivorship are challenging and need to consider late and long-term medical effects as well as comorbid conditions.4 These data suggest that it is possible to monitor HRQoL in cancer survivors with similar reliability as in the general population using BRFSS data, with the exception of the pain measure. Replication studies are needed in other states to inform consideration of the BRFSS for population surveillance of HRQoL in cancer survivors.

Acknowledgements

The authors thank Mr. Dale Smith for his expert copy editing. The authors would also like to acknowledge CDC and the state coordinators and contractors for their commitment to administering the BRFSS and collecting high-quality data annually. Finally, the authors wish to thank the University of Missouri-Columbia BRFSS supervisors and interviewers who collected the data for this study.

Ethical approval

University of Missouri-Columbia’s Health Sciences Institutional Review Board.

Funding

This project was funded in part by a grant from CDC (U58/CCU722795-02 and U58/CCU722795-04) and the National Cancer Institute (CA1121590). These contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC or the National Cancer Institute.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errorsmaybe discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Competing interests

None declared.

References

1. Jemal A, Siegel R, Ward E, Murray T, Xu J, Thun MJ. Cancer statistics, 2007. CA Cancer J Clin. 2007;57:43–66. [PubMed]
2. Gotay CC. Assessing cancer-related quality of life across a spectrum of applications. J Natl Cancer Inst Monogr. 2004;33:126–133. [PubMed]
3. Gotay CC, Lipscomb J, Snyder CF. Reflections on findings of the Cancer Outcomes Measurement Working Group: moving to the next phase. J Natl Cancer Inst. 2005;97:1568–1574. [PubMed]
4. Aziz NM. Cancer survivorship research: state of knowledge, challenges and opportunities. Acta Oncol. 2007;46:417–432. [PubMed]
5. Surveillance Implementation Group. Cancer surveillance research implementation plan. Rockville, MD: National Cancer Institute, National Institutes of Health; [last accessed 28 September 2007]. Available at: http://cancercontrol.cancer.gov/sig/sigrept.pdf.
6. National Cancer Institute (US) Cancer trends progress report – 2005 update. Bethesda, MD: National Cancer Institute, National Institutes of Health, Department of Health and Human Services; 2005.
7. Centers for Disease Control and Prevention. Methods, validity, and reliability bibliography: selected articles related to BRFSS and other self-reported data. Atlanta, GA: United States Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adult and Community Health; [last accessed 28 September 2007]. Available at: http://www.cdc.gov/brfss/pubs/mvr.htm.
8. Centers for Disease Control and Prevention. BRFSS data quality, validity, and reliability. Atlanta, GA: United States Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adult and Community Health; [last accessed 28 September 2007]. Available at: http://www.cdc.gov/brfss/pubs/quality.htm.
9. Nelson DE, Holtzman D, Bolen J, Stanwyck CA, Mack KA. Reliability and validity of measures from the Behavioral Risk Factor Surveillance System (BRFSS) Soz Praventivmed. 2001;46 Suppl. 1:S3–S42. [PubMed]
10. Miilunpalo S, Vuori I, Oja P, Pasanen M, Urponen H. Self-rated health status as a health measure: the predictive value of self-reported health status on the use of physician services and on mortality in the working-age population. J Clin Epidemiol. 1997;50:517–528. [PubMed]
11. Andresen EM, Bowley N, Rothenberg BM, Panzer R, Katz P. Test–retest performance of a mailed version of the Medical Outcomes Study 36-Item Short-Form Health Survey among older adults. Med Care. 1996;34:1165–1170. [PubMed]
12. Andresen EM, Catlin TK, Wyrwich KW, Jackson-Thompson J. Retest reliability of surveillance questions on health related quality of life. J Epidemiol Community Health. 2003;57:339–343. [PMC free article] [PubMed]
13. Travis LB, Rabkin CS, Brown LM, et al. Cancer survivorship – genetic susceptibility and second primary cancers: research strategies and recommendations. J Natl Cancer Inst. 2006;98:15–25. [PubMed]
14. Centers for Disease Control and Prevention. Behavioral Risk Factor Surveillance System survey data. Atlanta, GA: United States Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adult and Community Health; 2005.
15. Centers for Disease Control and Prevention. Behavioral Risk Factor Surveillance System questionnaire. Atlanta, GA: United States Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adult and Community Health; 2005.
16. Centers for Disease Control and Prevention. Behavioral Risk Factor Surveillance System summary data quality report. Atlanta, GA: United States Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adult and Community Health; 2005. [last accessed 4 September 2007]. Available at: http://ftp.cdc.gov/pub/Data/Brfss/2005SummaryDataQualityReport.pdf.
17. Centers for Disease Control and Prevention. Behavioral Risk Factor Surveillance System survey. Missouri data quality report. Atlanta, GA: United States Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adult and Community Health; 2005.
18. Centers for Disease Control and Prevention. Behavioral Risk Factor Surveillance System operational and user's guide. Version 3.0. Atlanta, GA: United States Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Adult and Community Health; [last accessed 14 May 2007]. Available at: http://ftp.cdc.gov/pub/Data/Brfss/userguide.pdf.
19. Centers for Disease Control and Prevention. Self-reported frequent mental distress among adults – United States, 1993–2001. MMWR Morb Mortal Wkly Rep. 2004;53:963–966. [PubMed]
20. Strine TW, Chapman DP, Kobau R, Balluz L. Associations of self-reported anxiety symptoms with health-related quality of life and health behaviors. Soc Psychiatry Psychiatr Epidemiol. 2005:432–438. [PubMed]
21. American Cancer Society. Cancer facts & figures 2007. Atlanta, GA: American Cancer Society, Inc.; 2007.
22. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420–428. [PubMed]
23. Fleiss JL. Statistical methods for rates and proportions. 2nd ed. New York: Wiley; 1981.
24. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed]
25. Evenson KR, McGinn AP. Test–retest reliability of adult surveillance measures for physical activity and inactivity. Am J Prev Med. 2005;28:470–478. [PubMed]
26. Bombard JM, Powell KE, Martin LM, Helmick CG, Wilson WH. Validity and reliability of self-reported arthritis: Georgia senior centers, 2000–2001. Am J Prev Med. 2005;28:251–258. [PubMed]
27. SAS Institute Inc. SAS/STAT 9.1: user's guide. Cary, NC: SAS Institute, Inc.; 2004.
28. Brownson RC, Jackson-Thompson J, Wilkerson JC, Kiani F. Reliability of information on chronic disease risk factors collected in the Missouri Behavioral Risk Factor Surveillance System. Epidemiology. 1994;5:545–549. [PubMed]
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • Cited in Books
    Cited in Books
    PubMed Central articles cited in books
  • MedGen
    MedGen
    Related information in MedGen
  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...