|
|
J Subst Abuse Treat. Author manuscript; available in PMC 2008 September 1. Published in final edited form as: Published online 2007 April 12. doi: 10.1016/j.jsat.2006.12.025. | PMCID: PMC1994649 NIHMSID: NIHMS28749 |
Influence of Organizational Functioning on Client Engagement in Treatment Jack M. Greener, Ph.D., George W. Joe, Ed.D., D. Dwayne Simpson, Ph.D., and Grace A. Rowan-Szal, Ph.D. Institute of Behavioral Research, Texas Christian University, Fort Worth, TX 76129 Wayne E. K. Lehman, Ph.D. The present study focused on the relationship between organizational functioning factors measured in a staff survey using the TCU Organizational Readiness for Change (ORC) assessment and client-level engagement measured by the TCU Client Evaluation of Self and Treatment (CEST) in drug treatment programs. The sample consisted of 531 clinical and counseling staff and 3475 clients from 163 substance abuse treatment programs located in 9 states from three regional Addiction Technology Transfer Centers (ATTC). Measures of client engagement in treatment (rapport, satisfaction, and participation) were shown to be higher in programs with more positive staff ratings of organizational functioning. In particular, these programs had fewer agency needs and more favorable ratings for their resources, staff attributes, and climate. These findings help establish the importance of addressing organizational factors as part of an overall strategy for improving treatment effectiveness. Keywords: Client engagement, Organizational functioning, Organizational climate, Substance abuse treatment, Treatment process outcomes Research on organizational behavior has long relied on the use of staff surveys as fundamental tools for measuring core domains of organizational functioning ( Kraut, 1996). Nadler (1996) described widespread applications of these “organizational assessments” using staff surveys to measure general functioning and health. Self-reports of staff have been used for multiple purposes, including diagnosing problem areas, evaluating progress over time, and in identifying and removing barriers that can block effective adoption of new ideas. Similar procedures are proving helpful to substance abuse treatment providers. In essence, assessments of staff perceptions serve to describe job satisfaction, group cohesion, adequacy of communication networks, and related attributes of a group of co-workers. More recent research also has addressed the role of climate strength pertaining to the extent that the climate perceptions are shared by members of the workgroup ( Chan, 1998; Schneider, Salvaggio, & Subirats, 2002). Such factors have virtually universal application to organizational effectiveness and are particularly appropriate for the study of treatment service agencies. An area of organizational studies known as “linkage research” ( Wiley, 1996) is of particular interest for drug treatment. In industry settings, this work focuses on the relationships between employee attitudes measured by employee surveys with employee performance, and other measures of business performance. Much of the emphasis has been on the relationship between organizational climate variables and customer satisfaction with service ( Johnson, 1996; Schneider, 1990; Schneider, White, & Paul, 1998). As in other research disciplines, Lundby, Fenlason, and Magnan (2001) stressed the importance of having a theoretical model to guide the development of a linkage project in order to understand the broader conceptual implications of findings. In substance abuse treatment settings, an important linkage to study is the relationship of organizational functioning with treatment effectiveness, and client satisfaction with treatment services received. This is better understood within the conceptual framework described by Simpson (2004) that discusses where program organizational factors fit within the overall treatment process. Using this framework as a guide, the current research examines how organizational attributes and counselor attributes relate to therapeutic process, starting at the early engagement phase of treatment. 1.1. Background The effect of program attributes has received attention in a number of treatment evaluation studies. However, most of the focus has been on structural factors such as program type, ownership, treatment setting, counselor education and training, and the characteristics of the clients being served ( D’Aunno & Vaughn, 1995; Etheridge, Hubbard, Anderson, Craddock, & Flynn, 1997; Friedmann, Alexander, & D’Aunno, 1999; Heinrich & Lynn, 2002; Roman & Johnson, 2002; Sun, 2006). Although Moos (2003) noted the general lack of research on the effects of organizational climate factors in the substance abuse treatment field, Moos and Moos (1998) previously reported that a supportive and goal directed climate was related to improved client treatment participation, greater satisfaction with treatment and better treatment outcomes at discharge. Knudsen, Johnson, and Roman (2003) similarly found that job autonomy was positively related to organizational commitment and negatively related to substance abuse counselors’ intention to quit. More directly relevant to this matter are the findings of Lehman, Greener, and Simpson (2002) showing that organizational factors, measured by the Texas Christian University (TCU) Organizational Readiness for Change (ORC) assessment, were related to three indicators of client engagement: counselor rapport, treatment satisfaction, and treatment participation. Therapeutic relationship seems to be a central factor for these indicators. The therapeutic relationship between clients and therapists has received considerable attention in the psychotherapy literature and is generally seen as a critical component in the therapeutic process. Martin, Garske, and Davis (2000) reported a meta-analytic study that included data from 79 studies that focused on measures of alliance (e.g., therapeutic alliance, working alliance, helping alliance) and reported finding modest but consistent relationships across a variety of outcomes. Tunis, Delucchi, Schwartz, Bany, and Sees (1995) noted that treatment outcomes might be improved by addressing the alliance between the client and both the counselor and peers. Belding, Iguchi, Morral, and McLellan (1997) investigated the impact of the helping alliance on treatment with a small sample of methadone maintenance clients and concluded that a strong helping alliance might be more an indicator of treatment progress rather than a precursor to other positive outcomes. Joe, Simpson, and Broome (1999) provided evidence that measures of treatment engagement and therapeutic involvement were predictive of treatment retention across three major treatment modalities (long-term residential, outpatient drug free, outpatient methadone). Models looking at treatment engagement components of the therapeutic relationship have consistently indicated that a stronger therapeutic relationship between counselors and clients was related to greater sessions attendance and less drug use during treatment ( Joe, Simpson, Greener, & Rowan-Szal, 1999; Simpson, Joe, Rowan-Szal, & Greener, 1995) and less drug use and criminal involvement at post-treatment follow-up ( Simpson, Joe, Greener, & Rowan-Szal, 2000). Joe, Simpson, Dansereau, and Rowan-Szal (2001) also found that higher counseling rapport during treatment was predictive of better post-treatment outcomes of less drug use and criminal activity. 1.2. Research questions The present study focused on the relationship between organizational functioning factors measured by the TCU Organizational Readiness for Change (ORC) assessment and treatment process outcomes measured by the TCU Client Evaluation of Self and Treatment (CEST) in Substance Abuse Treatment Programs (see Joe, Broome, Rowan-Szal, & Simpson, 2002). Rationale, scale descriptions, and psychometric properties of the ORC were previously reported by Lehman et al. (2002). They also described the direct relationships found between the ORC and the treatment process outcomes in a small number of programs, but statistical power was not sufficient to allow confidence in the stability of these initial results. Because the number and geographical diversity of programs available with both staff ORC data and client treatment process data has increased considerably over the intervening years, it is now possible to revisit this issue with a larger sample and more explicit objectives. Therefore, the central goal for this study was to examine whether an organization’s functioning is related to its effectiveness in treating drug clients. Expectations were that organizations with attributes that convey better functioning are more likely to be successful in their mission of engaging and keeping clients in treatment. Two specific research questions were addressed: (1) to what extent were the 18 ORC scales and composite scores for the four organizational functioning area domains (motivation for change, adequacy of resources, staff attributes, organizational climate) predictive of treatment process outcomes as measured independently by client ratings of counselor rapport, satisfaction with treatment, and treatment participation, and (2) would an overall composite organizational functioning index based on the four individual area domains also have predictive value as a summary indicator of climate. 2.1. Procedures for data collection Data were obtained from treatment programs located in 9 states from three regional Addiction Technology Transfer Centers (ATTCs). Cumulative data collection occurred over a period of approximately 5 years with slight variations in the procedures and format but generally using the same three forms, including the program staff version of the ORC (ORC-S), the Client Evaluation of Self and Treatment (CEST), and a program identification form (PID) completed by program directors. Staff and client records (based on the ORC-S and CEST, respectively) were aggregated for use as program-level indicators. Methods and procedures for collecting these forms were carried out in accordance with a protocol approved by the Institutional Review Board at Texas Christian University. Three regional ATTCs collaborated in TCU research on technology transfer by sponsoring training workshops for evidence-based treatment interventions. Program participation was solicited through the Prairielands ATTC (PATTC) and the Northwest Frontier ATTC (NFATTC), each contacting agencies in their respective areas of service. A brochure describing the planned training conferences was sent to about 200 agencies in each ATTC region (PATTC in February 2000 and NFATTC in January 2001). Research staff at TCU then contacted programs responding to this initial inquiry to further explain the nature and scope of the workshops, the requirements of completing assessment forms, and the topics to be covered at the conference. Subsequent data collection with the Gulf Coast ATTC (GCATTC) followed a similar format, including coordination with the state drug and alcohol agencies in two of the member states to accommodate their training initiatives. Data were collected in October 2002 in one state and during May and June 2003 in another. ORC data in the latter state was collected by the GCATTC using online services provided by PsychData Corporation. This followed a request that all agencies providing drug and alcohol treatment services with state contracts complete the ORC online. Programs with substantial racial/ethnic minority client representation were contacted to specifically encourage their participation. Each participating treatment unit in the three regional samples was asked to voluntarily administer a package of forms to be completed by the program director, program staff, and a sample of clients. The program director completed the ORC-D and PID, and counseling staff completed the ORC-S. They were administered before their regional training conferences with postage-paid, pre-addressed envelopes included with staff assessments so that participants could mail their completed surveys directly to TCU. A package of CEST forms was sent also to each treatment unit (see Joe et al., 2002). Typically, treatment units received up to 100 CESTs to be administered to a sequential sample of clients as they presented for treatment services during designated dates. CEST forms used in this study were administered 2-4 months before a planned training conference, and scheduled to be completed within a few weeks during the same time period that staff completed ORC assessments. 2.2. Participants The total database from the three ATTCs included 834 staff ORCs from 303 treatment programs, and 3963 CESTs from 197 programs. Not all programs were asked to provide client data (e.g., very small programs), and a few programs did not return both ORC and CEST data. The analytic sample was limited to programs that had both ORC and CEST linked-data available, resulting in a final sample of 531 ORC forms (64% of all ORC forms submitted) and 3475 CEST forms (88% of all CEST forms submitted) from 163 different treatment units, representing 83% of those submitting CEST data for clients. The comparatively high loss of ORC data was related in large part to the number of programs not asked to complete client-level (CEST) assessments. Overall, return rate for the CEST forms was approximately 52%, and although it was not possible to calculate participation rates for the ORC forms obtained online, the overall return rate for the ORC forms completed on paper was about 53%. The ORC return rate was slightly lower than the 56% to 64% rates for employees surveyed by mail as generally reported in the organizational literature ( Schneider, Parkington, & Buxton, 1980; Schneider et al., 1998), but the 52% return rate for client surveys was far better than the 15% to 41% average rates reported for customer surveys ( Johnson, 1996; Schneider et al., 1980; Wiley, 1996). This resulted in an average of approximately 21 CESTs and 3 ORCs per treatment unit. No minimum number of respondents was required for aggregation of treatment unit scores in order to use as much data as was available to establish a treatment unit score, using a simple additive composition model ( Chan, 1998). Staff were sampled from programs representing all major modalities, although half (49%) represented outpatient drug free settings. Residential programs represented 29%, and methadone maintenance and therapeutic communities each represented less than 2%. An “other” category that contained mostly detoxification units and units with mixed modalities represented 17% of all programs, while 3% of respondents did not indicate program type. More than half (56%) of the programs were free-standing, 8% were in community mental health centers, and 5% were in hospitals or university settings. In terms of staffing, 39% had 1-3 counselors, 42% had 4-7 counselors, and 19% had more than 7 counselors. The participating staff sample was 73% Caucasian, 16% African American, and 5% Hispanic; 63% were female. About 65% had a bachelor’s degree or higher, with 28% having a Master’s. Just over half (54%) had at least 5 years of experience in drug abuse counseling, and 28% had been on their present job for at least 5 years (with about 23% on their present job for less than a year). The client sample was 60% male and 34% female (6% failed to mark the gender item). Race/ethnicity was not obtained from the PATTC sample, but for those from the other ATTCs the composition was 58% Caucasian, 21% African American, 15% Hispanic, and 5% other or not reported. At the time of the survey, 38% of clients had spent less than 30 days in treatment, 27% had been treated 31-90 days, 20% had been treated 91-360 days, and 6% had been in treatment over a year. Time in treatment was not indicated by 9%. All of the analyses used unweighted data aggregated at the treatment unit level. 2.3. Measures The rationale, scale descriptions, and favorable psychometric properties of the ORC were previously reported in detail by Lehman et al. (2002). Brief description of the 18 scales in the Organizational Readiness for Change (ORC) are given in Table 1. Items use 5-point response categories (disagree strongly, disagree, uncertain, agree, agree strongly), and scale scores are calculated by reflecting items that need to be reverse scored and computing the mean and multiplying by 10. Thus, 30 represents a neutral score, and scores over 30 indicate stronger levels of agreement (similarly, scores below 30 indicate stronger levels of disagreement). Scale scores were computed as long as the respondent completed at least half of the items in a scale. | Table 1Brief Description of Scales in the Organizational Readiness for Change (ORC) Survey |
A specialized set of ORC composite scores were computed in this study for each of the four domains using all of the respective domain scales, and an overall organizational functioning index was computed from the domain composites. The program Needs and Pressures for change Index (NPI) had an alpha of .69; the Institutional Resources Index (IRI) had an alpha of .71; the Staff Attributes Index (SAI) had an alpha of .70; the Organizational Climate Index (OCI) with the stress scale reverse scored had an alpha of .88; and the Overall Organizational Functioning Index (OFI) - with NPI reverse scored - had an alpha of .73. Favorable psychometric properties of the client measures from the CEST were reported in detail by Joe et al. (2002). Response format and scoring procedures for the CEST was the same as the ORC. Brief descriptions of the CEST scales are presented below. Counseling Rapport (alpha = .92) Thirteen items in this scale reflect clients’ perceptions of core areas of therapeutic relationship with treatment counselors such as mutual goals, trust, and respect. Sample items include: “your counselor is easy to talk to”, “you are motivated and encouraged by your counselor”, and “your counselor respects you and your opinions”. Treatment Satisfaction (alpha = .81) Seven items in this scale reflect clients’ perceptions of how well the treatment program is meeting their needs. Sample items include: “this program is organized and run well”, “you are satisfied with this program”, and “you can get plenty of personal counseling at this program”. Treatment Participation (alpha = .86) Twelve items in this scale reflect clients’ perceptions of their own involvement and active engagement in treatment sessions and services. Sample items include: “you always attend the counseling sessions scheduled for you”, “you always participate actively in your counseling sessions”, and “you are following your counselor’s guidance”. Means and standard deviations of the 18 individual ORC scales, the four domain area composites, the overall organizational functioning index, and the three CEST treatment engagement scales for the study sample are presented in Table 2. | Table 2CEST and ORC Scale Means and Standard Deviations for Programs |
3.1. Correlations among domains As shown in Table 3, correlations based on program-level scores among the four domain composites ranged from -.30 to .59, with the needs domain composite negatively correlated with the others. Thus, the needs composite score was reversed when combined with the others to calculate the overall Organizational Functioning Index (OFI). Institutional resources and organizational climate domain scores were the most highly correlated pair (r = .59). | Table 3Reliabilities and Correlations among ORC Domain Composites (N = 163 treatment units) |
3.2. Relationship of ORC to client engagement The program-level relationships of the 18 ORC scales, four area domain composites, and the overall organizational functioning index scores with the three treatment engagement scales from the CEST are presented in Table 4. Counselor rapport was significantly related to 12 of the 18 ORC scales, including all six of the organizational climate scales (positively for mission, cohesion, autonomy, communication, change, and negatively for stress) and to four of the five resources scales (offices, staffing, equipment, and internet). It was negatively related to program needs in the motivation domain and positively related to the influence scale from the staff attributes domain. Counselor rapport also was related to the four domain composites and the overall organizational functioning index. A similar pattern of relationships was found for treatment satisfaction where 11 of the same scales yielded significant correlations, but staff cohesion was not correlated with client ratings of their satisfaction with treatment. In contrast to the results for counselor rapport and treatment satisfaction, however, only program needs (negative), influence, and mission scales were significantly correlated with client ratings of their treatment participation. The counselor rapport and treatment satisfaction measures were highly correlated with each other (.83), whereas treatment participation only correlated .65 with counselor rapport and .53 with treatment satisfaction. | Table 4Correlations of ORC Scales with Client Engagement (N = 163 treatment units) |
Six of the ORC scales (training needs, pressures for change, training resources, growth, efficacy, adaptability) were not related significantly to any of the client engagement scales. None of the four individual domain composites were correlated with treatment participation, but the overall organizational functioning index was. The overall functioning index accounted for about 9% of the variance in counselor rapport, 9% in treatment satisfaction, and 3% in treatment participation. Table 5 presents the results of multiple regression analyses predicting the client engagement outcomes from all 18 ORC scales. Multiple regression analyses were able to account for 19% of the variance in counselor rapport and 27% of the variance in treatment satisfaction. The regression equation predicting participation was not significant. Only the influence scale was significant in the equation for predicting rapport, and only influence and cohesion were significant in the equation predicting satisfaction (with cohesion receiving a negative weight even though its correlation with satisfaction was positive). However, the high degree of correlation among some of the predictors resulted in a situation where most of the prediction can be accounted for by the common variance among the predictors with little unique variance contributed by any of the individual predictors. For example mission correlated with communication .79, program needs correlated with training needs .72, and communication correlated with change .70. | Table 5Multiple regression using ORC Scales to predict Client Engagement (N = 160 treatment units) |
The findings confirm the general expectations that counselor rapport and treatment satisfaction were higher among clients treated in healthier organizational settings, and they had very similar relationships with the ORC indicators of organizational functioning. This seems to reflect the importance of counselor rapport in establishing a meaningful therapeutic relationship and also its role as a determinant of treatment satisfaction. Treatment participation, on the other hand, had lower correlations with the other measures and was much less predicable from the ORC scales. This might be explained by the fact that the items making up the rapport and satisfaction scales focus largely on interactions with counselors and program features, while treatment participation reflects mainly the client’s behavioral commitments, session attendance and recovery initiatives. More importantly, measures of “participation” are difficult to apply consistently across the diversity of residential and outpatient treatment settings included. The evidence that programs with high program needs, as perceived by the counselors, is associated with low scores on all three client engagement criteria is not surprising. Likewise, recognition of program deficiencies is related to inadequate institutional resources. These views are consistent with the finding that programs with more resources also had higher client ratings of rapport and satisfaction. Similarly, programs with high scores on the organizational climate scales also had higher ratings on the rapport and satisfaction scales, and the organizational climate domain score was most highly correlated with the institutional resources domain. An unexpected finding was the positive correlation between the personal influence scale (from the staff attributes domain) and client engagement indicators. It implies that counselors with high influence scores may possess some combination of personal characteristics (empathy, self-confidence, good interpersonal skills, persuasiveness, counseling expertise) that enables them to establish good therapeutic relationships with clients as well as form positive relationships with fellow staff members. These are the likely traits of opinion leaders in their respective treatment programs ( Moore et al., 2004; Rogers, 2003). This study supports and extends the previous work reported by Lehman et al. (2002). Although the ORC was originally developed as a measure to assess organizational readiness for change in treatment programs as part of technology transfer initiatives, its scales also serve as effectiveness measures of organizational functioning (also see Broome, Flynn, Knight, & Simpson, this issue). The results here point to the value of identifying deficiencies in organizational functioning that, if corrected, should contribute to improved client outcomes in the treatment process. The domain composite indices serve as general indicators of overall organizational functioning of a program. However, if a program seeks out strategies to improve functioning in specific areas of deficiency, it is expected that it would be better to focus on low scores at the scale level in targeting areas for improvement and developing the appropriate interventions ( Simpson & Dansereau, in press). Additionally, in some cases it is advisable to examine the results for some scales (particularly program needs) at the individual item level to help target particular topics for improvement. 4.1. Limitations This study is limited in several respects. The data are cross-sectional in nature and the participant sample is not a randomized representation of the field as a whole. Participants generally did not include programs already involved in other large-scale studies, such as the Clinical Trials Network (CTN) initiatives funded by the National Institute on Drug Abuse. Although efforts were made to achieve diversity in geographic and racial/ethnicity representation, there was no centralized or highly controlled sampling plan in the recruitment of participants. Unfortunately this data collection strategy and the moderate response rates leave the study in a position of uncertainty regarding its representativeness. The sampling objective was to collect data that would facilitate study of the natural process of program change and technology transfer in the substance abuse treatment field, focusing mainly on counselors and programs interested in attending training workshops to learn new counseling techniques. The analyses were limited to pre-training baseline data, but the participants had in some cases already expressed an interest in attending training workshops and therefore may be biased in attitudes toward change. Additionally, the small staff size of some treatment units, combined with low response rates from other programs, limit generalizations of the findings. Because the data collection protocol relied on anonymous voluntary participation at the treatment unit level, limits were imposed both on ability to resolve cases of missing or conflicting survey responses and on choices of analysis for the linkage data. For instance, it was possible only to establish links between the ORC and CEST measures aggregated to the treatment unit level. Although the sample was considerably larger than in the previous research by Lehman et al. (2002), it still lacked sufficient statistical power to allow for detailed sample breakdowns on other demographic and program variables of interest. In the present study, very few predictors were significant in the multiple regression analyses. High correlations among some of the ORC scales resulted in a multicollinearity situation that limited the interpretability of the regression weights in these analyses. Because of the pattern of high correlations among the independent variables, this situation can lead to suppressor effects where the sign of the zero order correlation of a predictor is different from the regression weight and serves to suppress or control for irrelevant variance that reduces the magnitude of the relationships of other predictors with the criterion variable ( Pedhazur, 1982). It is important to keep in mind that while multiple regression maximizes the variance accounted for in the sample, it also capitalizes on chance relationships and needs to be subjected to cross validation to estimate the true correlation in the population ( Cohen, Cohen, West, & Aiken, 2003; Pedhazur, 1982). These analyses illustrate the limitations of using multiple regression to understand the comparative importance and dynamics of the individual ORC scales in this situation. 4.2. Summary and future research In summary, results indicated that organizational functioning is predictive of treatment engagement outcomes. Treatment implications include the potential value of addressing organizational functioning issues and the effects they may have as well as identification of specific client needs and changes in treatment regimens to improve client functioning in treatment programs. Additional research is needed to confirm the generalizability of these findings, and longitudinal records should be obtained to assess whether specific interventions aimed at improving organizational functioning do lead to positive changes in client outcomes over time (see Simpson, Joe, & Rowan-Szal, this issue). More specifically, future research should include efforts to overcome some of the weaknesses of this study by having an improved strategy for obtaining higher response rates within a more highly structured sampling plan. In general, more attention also needs to be given to the treatment context and infrastructure support as it relates to treatment intervention studies. Research also needs to be expanded to cover additional populations such as adolescent and correctional populations. Some recent research has started to investigate international cross-cultural applicability of these instruments ( Rampazzo, De Angeli, Serpelloni, Simpson, & Flynn, 2006; Rowan-Szal, Joe, Bartholomew, Greener, & Simpson; 2006), which adds evidence for generalizability. The role of climate strength is another area that deserves to be examined to determine whether the degree of agreement within treatment units moderates the relationships between the ORC scores and client outcomes. And finally, the growing evidence showing the importance of organizational functioning on services effectiveness suggests that intervention strategies devoted to “institutional change” deserve more attention. The authors would like to thank the Prairelands, Northwest Frontier, and Gulf Coast Addiction Technology Training Centers (ATTCs) for their assistance with recruitment and training. We would also like to thank the individual programs (staff and clients) who participated in the assessments and training in the DATAR Project. This work was funded by the National Institute of Drug Abuse (Grant R37 DA13093). The interpretations and conclusions, however, do not necessarily represent the position of NIDA or the Department of Health and Human Services. More information (including intervention manuals and data collection instruments that can be downloaded without charge) is available on the Internet at www.ibr.tcu.edu, and electronic mail can be sent to ibr/at/tcu.edu.
|
The publisher's final edited version of this article is available at 