Development of an Assessment to Examine Training of the Hospice Primary Caregiver
2 Melissa A. Clark, PhD,3 Carol Spence, PhD,4 and David Casarett, MD, MA5Abstract
Background: Key to high-quality care of dying hospice patients at home is whether the hospice provides adequate training so the caregiver can safely care for the patient.
Objective: The study objective was to develop and validate a survey of hospice training for caregivers to ensure safe, high-quality care in the home setting.
Design: Our survey design was cross-sectional. Bereaved respondents of individuals who died at home under the care of hospice were surveyed three to six months postdeath.
Measurements: Items were developed based on advice of an expert panel, focus groups of hospice caregivers, and literature review, with 12 items developed for testing and examining 8 key processes of care. We examined the validity and reliability of the assessment using factor analysis, correlational analyses, and multivariable modeling.
Results: Our sample consisted of 262 primary caregivers (mean age 62.4, 76.7% female, 58.8% non-Hispanic white). Six questions focused on providing the caregiver with information, while another six focused on the training that hospice provided. Based on model fit and Cronbach's alpha, we dropped the information items. The items that examined hospice training demonstrated a one-factor solution and a Cronbach's of 0.90. We examined correlations of the multi-item composite with overall rating of quality of care (0.53), overall distress (0.31), and whether the respondent would recommend this hospice to others (0.49). There were no significant sociodemographic correlates of concerns with training.
Conclusions: Sufficient preliminary reliability and validity warrants further testing of this composite to examine the adequacy of training provided to family members to care safely for the patient.
Introduction
Key to safe care of the hospice patient at home is that the hospice team empowers the primary caregiver regarding what to monitor, how to use medications to provide comfort, and when to call the hospice for further help. Safety in the home setting is different from that of an acute care hospital. As we previously proposed, hospice should be held accountable for providing appropriate education and training that allows the caregiver to safely care for the patient at home, including administering medications.1 Yet caregivers repeatedly report a lack of support in performing tasks such as pain and symptom management and medication management.2–5
In this report we outline the development and evaluation of a new survey module within an existing quality of care instrument that examines caregiver perspectives on whether hospice provided the needed information and training to allow primary caregivers to safely care for the dying patient at home. The survey for which we developed this new module is the Family Evaluation of Hospice Care (FEHC) instrument, developed and validated in 1999 to measure quality of care at the end of life from the perspective of family members and others who were particularly close to the deceased.6–8 Because patient safety is a necessary component of high-quality care,8 the results of patient safety measures such as this module are an important component of hospice program quality improvement.
Methods
Survey development
We relied on expert opinion, focus groups with bereaved family members, and existing guidelines9 to choose six key processes of care that family are required to do in safely caring for a patient at home. Based on this review, we developed 12 new items. Within the 12 items were two proposed composite measures. The first composite consisted of 6 items to determine whether the caregiver had received from hospice all the information needed to care for the patient at home, and the second composite consisted of 6 items to determine whether the caregiver had received all of the training that was needed to properly carry out the care processes of interest.
Focus groups consisted of 2–10 primary caregivers of individuals who died at home on hospice care. There were 6 focus groups and 39 total participants from hospices in 6 geographically diverse areas of the United States. Average age of participants was 63 years old, 74% were female, 72% were non-Hispanic white, and 67% were the child or spouse of the decedent. Members of each focus group participated in a card-sort that ranked the importance of hospice training in various processes of care. Two expert panels—one panel of experts on survey design and content and one panel of hospice program quality directors—reviewed each new item for relevance to high-quality end-of-life care and whether the process was under the control of the hospice provider. The survey underwent one-on-one, in-person cognitive testing among 26 respondents recruited from the data collection sites. The basic protocol involved reading the questions to the respondents, having them answer the questions, asking them to “think out loud” about the process that they used to answer each survey item, and using follow-up probes to ask for feedback on aspects of the questions that were considered potentially problematic. Revisions were then tested with a pre-test conducted among a subset of bereaved family members and was repeated 24 to 72 hours later to confirm the short-term intra-rater reliability of each item.
Data collection
Hospices were selected from six diverse geographic regions of the United States, including rural and urban locations and areas serving African American and Hispanic populations. Respondents were identified by the hospice as the primary person to contact regarding care for the patient, and underwent several screening questions to ensure they were the primary caregiver of the hospice patient. Surveys were available in English and Spanish and could be completed by mail or telephone. Surveys were administered to each respondent three, six, and nine months after the death of the hospice patient. Participants received a $20 gift card for each completed survey. All data collection was through the Survey Center in the School of Public Health at Brown University. For this analysis, six-month responses were used based on previous analysis that showed no significant differences in responses between three and nine months.10 If a respondent did not complete a six-month survey but did complete a three-month survey, we included the three-month one to maximize our sample size.
Additional measures
Along with the composite items we designed, several additional questions, which we will refer to throughout the manuscript as “validity measures,” were included for descriptive purposes and for examining the validity of the proposed composites. To examine the construct validity of the proposed composites, we included the five-item Mental Health Inventory (MHI-5)11 and the Inventory of Complicated Grief (ICG) Short Form. The ICG Short Form consists of 11 statements concerning the immediate bereavement-related thoughts and behaviors of the responder. There are five response options ranging from “never” to “always” and clients who score over 25 are considered at high risk for pathological grief. The Cronbach's alpha of the instrument is 0.94, and it has demonstrated strong reliability and validity.12 Our inclusion of the ICG as a measure of construct validity is based on previous literature that has identified a relationship between quality of end-of-life care of a patient and complicated grief among bereaved family members.13 The MHI-5 is a five-question assessment of mental health based on self-report. There are six response options ranging from “none of the time” to “all of the time,” and individuals who score high on the scale report feelings of nervousness and depression all the time for the past four weeks; the Cronbach's alpha of the instrument is 0.82.14 We included the MHI-5 as a measure of construct validity because of previous literature that has identified a relationship between quality of care at the end of life and mental health of the bereaved caregiver.
We also included a question asking respondents to rate their level of distress in the past week on a scale of 0 (no distress) to 10 (extreme distress) for construct validity analysis. The measure is a subjective scale, with “distress” defined by the rater. For further construct validity testing, we collected information about whether the respondent would recommend this hospice to others, and whether the patient was referred to hospice “too early,” “too late,” or “at the right time.” As a final measure of construct validity, we included an item asking respondents to rate quality of care received from the hospice using a five-point scale ranging from “excellent” to “poor.”
Finally, the FEHC collected information on the demographics of respondents and patients, the relationship between respondent and patient, and further characterized the respondent's experience with hospice and involvement in the patients' care.
Analytic approach
Because high-quality care strives for excellence, responses to each question in the composites were recoded to create dichotomous measures that indicated opportunities to improve (i.e., concerns with care). For example, responses to the question, “How much training did you receive from hospice about how to give medicines when the patient had difficulty swallowing?” ranged from “All the training that was needed” to “None of the training that was needed.” We considered an answer of anything other than “All” as an opportunity to improve and dichotomized the variable by assigning a value of zero to “All” and a value of one to all other responses. The two multi-item composites were created by summing the number of areas for improvement across the six items in each construct. Higher composite scores indicated more opportunities to improve quality of care (i.e., more concerns with information/training).
For each respondent we counted the number of items for which they had invalid/missing responses. If a composite was missing one or two items, we examined the possible bias of those persons with missing data and imputed by the mean composite score of the hospice from which the participant was recruited. Each individual was only permitted two imputed items; if an individual was missing responses on more than two items in a composite, that individual was not included in the analysis. Over 75% of our surveys were returned with no missing data on composite score measures. Sixteen percent were returned with one missing item, and 7% with two missing items. Only 2% of surveys had more than two composite items missing; these individuals were not included in our analysis.
The distribution of each item was reviewed to examine whether there was a ceiling effect. Confirmatory factor analysis with Varimax rotation was used to determine whether items loaded on the specified constructs for which they were intended. Decision on the number of factors to retain in the model was based on Kaiser's eigenvalue rule and scree test.15 Cronbach's alpha was used to examine internal consistency for each multi-item composite. We also tested the factor structure of the composites using a multifactor confirmatory factor analysis (CFA) model that was fit to the individual-level data. Evidence of model fit was evaluated based on widely used model fit index criteria, particularly Root Mean Square Error of Approximation (RMSEA) ≤0.08, Tucker-Lewis Index (TLI) ≥0.95, and Confirmatory Fit Index (CFI) ≥0.95.16,17
We assessed construct validity of the composites by correlating each with the following two validity measures: overall quality of care and likelihood of recommending the hospice to others. We expected to find a moderate correlation18 between the composites and these items. One-way ANOVA tested for differences in concerns with information/training received among respondents who felt they were referred to hospice too early, at the right time, or too late.
As a measure of construct validity, we used correlation coefficient testing to examine whether respondents with higher composite scores (indicating more opportunities for the hospice to improve it's information/training on critical care processes) had higher levels of emotional distress (measured by the MHI-5 and the question on distress in past week) or grief (measured by the ICG).
Next, an exploratory analysis examined the association between sociodemographic variables, characteristics of the respondent, and the composite measures. Univariate associations and an ordinal multivariate logistic model were used.
Results
Eligible individuals (n = 459) were invited to the study, and a total of 262 respondents completed the questionnaire postdeath (57.1% response rate). Table 1 shows the sample characteristics of patients and bereaved respondents. Respondents and patients were mostly white, female, and had at least some college education. Mean age of patient death was 76.2, and 55.2% had cancer. Mean age of respondents was 62.4, 45.0% were a spouse/partner, and 42.7% were a child of the deceased. The median length of hospice stay was 30 days, and 96.6% of respondents reported being the person involved in medical decision making for the patient. Thirty-five percent of respondents previously cared for a dying patient, and 28.2% had previous experience with hospice. We had limited information on eligible nonrespondents. Compared to respondents, decedents of nonrespondents were slightly older (76.4 versus 76.2, p = 0.60) and represented a higher proportion of African Americans (28.9% versus 24.8%, p = 0.13) and Hispanics (11.7% versus 11.1%, p = 0.76). These differences were not significant, suggesting that, based on the limited information we had on eligible nonrespondents, our survey respondents were a representative sample of the population.
Table 1.
Sample Description of Respondents And Patients
| Characteristic | Home sample n = 262 |
|---|---|
| Patient characteristics | |
| Patient gender (%, 95% CI) | |
| Female | 54.2 (48.1–60.3) |
| Patient age (mean, SD) | 76.2 (14.8) |
| Patient race/ethnicity (%, 95% CI) | |
| Non-Hispanic white | 60.3 (54.3–66.3) |
| Non-Hispanic black | 24.8 (19.5–30.1) |
| Hispanic | 11.1 (7.2–14.9) |
| Multiracial/other | 3.8 (0.0–7.6) |
| Patient education (%, 95% CI) | |
| Less than high school | 23.6 (18.4–28.8) |
| High school graduate/GED | 28.6 (23.0–34.1) |
| At least some college | 47.9 (41.8–54.0) |
| Length of hospice stay in days (median, 25th–75th percentile) | 30 (12–90) |
| Primary diagnosis (%, 95% CI) | |
| Cancer | 55.2 (49.1–61.3) |
| Heart/circulatory diseases | 9.7 (6.0–13.3) |
| Lung/breathing diseases | 5.0 (2.3–7.7) |
| Kidney disease | 0.8 (0.0–1.8) |
| Liver diseases | 1.9 (0.2–3.6) |
| Stroke | 0.8 (0.0–1.8) |
| Dementia/Alzheimer's | 7.3 (4.1–10.5) |
| AIDS/other infectious diseases | 0.8 (0.0–1.8) |
| Frailty and decline due to age | 6.9 (3.8–10.1) |
| Other | 11.6 (7.7–15.5) |
| Respondent characteristics | |
| Respondent gender (%, 95% CI) | |
| Female | 76.7 (71.6–81.9) |
| Respondent age (mean, SD) | 62.4 (12.7) |
| Respondent race/ethnicity (%, 95% CI) | |
| Non-Hispanic white | 58.8 (52.8–64.8) |
| Non-Hispanic black | 23.7 (18.5–28.8) |
| Hispanic | 12.6 (8.6–16.6) |
| Multiracial/other | 4.2 (0.0–8.4) |
| Respondent education (%, 95% CI) | |
| Less than high school | 10.4 (6.7–14.1) |
| High school graduate/GED | 22.3 (17.2–27.4) |
| At least some college | 67.3 (61.5–73.0) |
| Relationship to patient (%, 95% CI) | |
| Spouse/partner | 45.0 (37.9–52.2) |
| Child | 42.7 (36.7–48.8) |
| Parent | 1.5 (0.0–3.0) |
| Sibling | 5.3 (2.6–8.1) |
| Other relative | 1.5 (0.0–3.0) |
| Friend | 0.4 (0.0–1.1) |
| Other | 3.1 (1.0–5.2) |
| Previous experience caring for the dying (%, 95% CI) | 35.1 (29.3–40.9) |
| Previous experience with hospice (%, 95% CI) | 28.2 (22.8–33.7) |
| Involved in medical decisions for the patient (%, 95% CI) | 96.6 (94.3–98.8) |
| Power of attorney/health care proxy (%, 95% CI) | 73.3 (67.9–78.7) |
| Involved in care of patient while in hospice | |
| Sometimesa | 6.1 (3.2–9.0) |
| Usually | 11.5 (7.6–15.3) |
| Always | 82.4 (77.8–87.1) |
| Involved in treatment decisions in the last week of life (%, 95% CI) | |
| Not at all | 0.4 (0.0–1.1) |
| A little bit | 3.1 (1.0–5.2) |
| Somewhat | 5.0 (2.3–7.6) |
| Very much | 90.8 (87.3–94.4) |
| Days of contact with patient in last week of life (mean, 95% CI) | 6.5 (6.4–6.7) |
CI, confidence interval; GED, General Educational Development test; SD, standard deviation.
The percentage of missing responses for each item in the training composite ranged from 2.3% to 3.5% (see Table 2). Table 2 also lists the percentage of respondents in the highest category for each item; no training items demonstrated evidence of a ceiling effect.
Table 2.
Frequencies and Psychometric Properties of the Items Comprising the Information and Training Composite Measures
| Composite / item | Response set | % missing/ don't know | % responding in highest category | Mean number of concerns | Score range among hospices in sample |
|---|---|---|---|---|---|
| Information composite | 0.75 | n/aa | |||
| How much information did you receive about the medicines used to manage the patient's pain? | LTW/RA/MTWb | 2.2 | 91.5 | ||
| How much information did you receive about what side effects to watch for from the pain medicine? | LTW/RA/MTW | 9.2 | 80.2 | ||
| How much information did you receive about helping the patient with his/her breathing? | LTW/RA/MTW | 7.6 | 83.7 | ||
| How much information did you receive about what to do if the patient was agitated? | LTW/RA/MTW | 10.7 | 95.4 | ||
| How much information did you receive about how to safely move the patient? | LTW/RA/MTW | 5.0 | 93.9 | ||
| How much information did you receive about what to expect while the patient was dying? | LTW/RA/MTW | 6.5 | 82.5 | ||
| Training composite | 1.32 | 0.77–1.77 | |||
| How much training did you receive from hospice about what side effects to watch for from the pain medicine? | A/M/S/Nc | 2.6 | 64.8 | ||
| How much training did you receive from hospice about when to give extra doses of medicines to better control the patient's pain? | A/M/S/N | 3.5 | 78.3 | ||
| How much training did you receive from hospice about how to give medicines when the patient had difficulty swallowing? | A/M/S/N | 3.1 | 76.1 | ||
| How much training did you receive from hospice in helping the patient with his/her breathing? | A/M/S/N | 3.1 | 71.0 | ||
| How much training did you receive from hospice about what to do if the patient was agitated? | A/M/S/N | 2.7 | 71.9 | ||
| How much training did you receive from hospice about how to safely move the patient? | A/M/S/N | 2.3 | 69.0 |
We included all 12 items of the new module in the factor analysis. Factor analysis suggested a two-factor solution, with training items loading together with an eigenvalue of 4.33 and information items loading with an eigenvalue of 1.74 (see Table 3). No items in the training factor loaded below 0.38; all factor loadings indicated moderate to high relevance of each item in the factor. In contrast, three items in the information factor—pain medicines, pain medicine side effects, and helping the patient's breathing—had poor loading values (see Table 3), indicating low relevance of these items. Model fit indices were stronger for the training composite than for the information composite (RMSEA 0.110 versus 0.094; TLI 0.962 versus 0.851; and CFI 0.977 versus 0.901). Table 3 also displays psychometric properties of the composites, including the Cronbach's alpha of the composites and item-to-total correlations. The training composite had a Cronbach's of 0.90, indicating strong internal reliability. The alpha for the information composite was 0.60, which is below the recognized cut-off for internal consistency reliability for group-level analysis (0.70).19 Table 3 also illustrates the high item-total correlations for each item in the training composite, and demonstrates that all six items together make for the strongest composite. Due to poor factor loading, weak model fit indices, and low internal reliability, we decided to drop the information items from our analysis at this point.
Table 3.
Factor Analyses of the Information and Training Composite Measures
| Composite and item | Cronbach's alpha for composite | Eigenvalue of composite and factor loading of items | Item-total correlation | Cronbach's alpha with item deleted |
|---|---|---|---|---|
| Information composite | 0.60 | 1.74 | Average = 0.64 | Average = 0.55 |
| How much information did you receive about the medicines used to manage the patient's pain? | −0.05 | 0.62 | 0.55 | |
| How much information did you receive about what side effects to watch for from the pain medicine? | 0.06 | 0.75 | 0.52 | |
| How much information did you receive about helping the patient with his/her breathing? | 0.05 | 0.61 | 0.62 | |
| How much information did you receive about what to do if the patient was agitated? | 0.78 | 0.61 | 0.56 | |
| How much information did you receive about how to safely move the patient? | 0.91 | 0.51 | 0.58 | |
| How much information did you receive about what to expect while the patient was dying? | 0.36 | 0.74 | 0.47 | |
| Training composite | 0.90 | 4.33 | Average = 0.84 | Average = 0.88 |
| How much training did you receive from hospice about what side effects to watch for from the pain medicine? | 0.47 | 0.81 | 0.89 | |
| How much training did you receive from hospice about when to give extra doses of medicines to better control the patient's pain? | 0.43 | 0.85 | 0.88 | |
| How much training did you receive from hospice about how to give medicines when the patient had difficulty swallowing? | 0.66 | 0.82 | 0.89 | |
| How much training did you receive from hospice in helping the patient with his/her breathing? | 0.38 | 0.86 | 0.88 | |
| How much training did you receive from hospice about what to do if the patient was agitated? | 0.69 | 0.87 | 0.87 | |
| How much training did you receive from hospice about how to safely move the patient? | 0.69 | 0.81 | 0.89 |
There was variation by hospice site for the training composite, with average number of areas for improvement ranging from 0.77 to 1.77 (out of 6). Items with the most variability by hospice site included giving extra doses of pain medications (0.08–0.35) and helping an agitated patient (0.10–0.27).
When we tested construct validity for the training composite, there was a moderate correlation between the composite and overall quality of care (0.53) and likelihood of recommending the hospice in the future (0.49). Based on previous publications20 that demonstrated a moderate correlation between quality-of-care scores and timing of hospice referral, we used one-way ANOVA to test for differences among respondents who felt the patient was referred to hospice too early, at the right time, or too late. The training composite differed significantly across the three categories (F (4, 222) = 2.50, p = 0.043), with those who were referred to hospice at the right time having fewer concerns with their training than individuals referred to hospice too late or too early.
To test construct validity, we examined the association between the training composite score and emotional distress (as measured by MHI-5). The composite had a negligible but positive correlation (0.08). We also examined the association between training and grief. Individuals who reported a higher number of areas for improvement with training tended also to have higher grief (r = 0.31). Finally, we examined the correlation between training and overall distress. We found that respondents who reported more areas for improvement with training also reported more overall distress (low positive correlation, r = 0.31).
Table 4.
Correlations Between the Training Composite Score and Validity Measures
| Training composite correlation coefficienta | |
|---|---|
| Training composite | 1.00 |
| Emotional distress | 0.08 |
| Grief | 0.31 |
| Overall distress | 0.31 |
| Overall quality of care | 0.53 |
| Hospice recommendation | 0.49 |
Multivariate analysis suggested no significant sociodemographic correlates of the composite training score, although gender came close to statistical significance. On average, with all else held constant, females had 45% lower odds (OR: 0.55, 95% CI: 0.30–1.01) of reporting problems with training compared to males. Variables controlled for included respondent age, gender, and race; relationship to patient; patient diagnosis; previous experience with hospice; and level of involvement in care.
Discussion
A key aspect of safe, high-quality hospice care is ensuring the primary caregiver receives all necessary knowledge and training on what to monitor, steps to take if symptoms worsen, and when to call for help. After focus group interviews and expert review, we developed a new 12-item FEHC module to include questions on both information (six items) and training (six items) provided to the hospice caregiver. The information items were dropped based on their poor performance. Our data support the use of one six-item composite score that measures how well hospice trains the primary caregiver for his or her critical role in patient care.
Even beyond the hospice setting, caregiver training has been identified as an important contributor to quality of care. Bee et al.21 reviewed the literature on informal caregivers' needs in providing home-based end-of-life care to people with cancer. They found caregivers repeatedly report a lack of confidence in performing nursing tasks, such as pain and symptom management and medication management. Broback and Bertero22 note that within home-based palliative care services without adequate provision of training, families typically feel out of control, disempowered to make decisions, and unable to cope with the physical care required on a day-to-day basis,2–5 leading to lower quality of care for the patient and lower quality of life for the caregiver.
In this article we outline the development and initial testing of a new composite that examines whether the hospice adequately trained caregivers on key aspects of caring for the patient at home to allow a safe, comfortable death with high-quality end-of-life care. However, there are important limitations to bear in mind when interpreting findings from this study. First, despite our attempts to oversample Hispanic respondents, this demographic is still underrepresented in our sample. Second, we only examined caregiver experiences from six hospices; therefore, more analyses, using a larger number of hospices and bigger sample sizes, should confirm our results. Third, we have limited information on nonrespondents. Finally, all hospices sampled were not-for-profit, therefore our results may not be generalizable to for-profit hospices.
Overall, we find that the proposed composite, which focuses on adequacy of caregiver training, is ready for further testing in a larger sample. We shared these items with the team creating the Consumer Assessment of Healthcare Providers and Systems® (CAHPS) Hospice survey, a standardized bereaved family member survey program designed to measure the experience of patients who died while receiving care from Centers for Medicaid and Medicare (CMS) certified hospices. Four of these items are undergoing further national testing. Because of the inextricable link between safety and quality, the results of patient safety measures such as this module should be available with all publicly reportable data on hospice quality.
Author Disclosure Statement
This analysis was funded by Agency for Healthcare Research and Quality (AHRQ), Family Evaluation of Hospice Care, 1R01HS019675. All authors attest that no competing financial interests exist.