|
|
J Subst Abuse Treat. Author manuscript; available in PMC 2008 September 1. Published in final edited form as: Published online 2007 April 16. doi: 10.1016/j.jsat.2006.12.029. | PMCID: PMC2104560 NIHMSID: NIHMS28739 |
Organizational Readiness for Change in Adolescent Programs Criterion Validity Lisa Saldana, Ph.D., Jason E. Chapman, Ph.D., Scott W. Henggeler, Ph.D., and Melisa D. Rowland, M.D. Family Services Research Center, Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina, Charleston, South Carolina 29451, (December 18, 2006) This study examined the convergent and concurrent validity of the Organizational Readiness for Change (ORC; Lehman, Greener, & Simpson, 2002) Scale among practitioners who treat adolescents. Within the context of a larger study, the ORC and measures of practitioner attitudes toward evidence-based practices (EBPs) and treatment manuals were administered to a heterogeneous sample of 543 community-based therapists in the state mental health and substance abuse treatment sectors. Using a contextual random-effects regression model, the associations between ORC domains and measures of practitioner characteristics and attitudes were examined at both therapist and agency levels. Results supported the convergent and concurrent validity of several domains. Namely, the motivational readiness and training needs domains were associated with higher appeal and openness to innovations. Program resources and climate, however, were less related. Discussion focuses on the utility of the ORC in helping to evaluate the needs of programs considering the adoption of an EBP. Keywords: organizational readiness, validation, contextual modeling, evidence-based practice The transport of evidence-based substance abuse treatments to community-based practitioners and provider agencies is a public health priority ( National Institute on Drug Abuse, 2006). A key factor in the successful transport of evidence-based practices (EBPs), however, is practitioner and organizational readiness to adopt new practices. Clearly, some practitioners and organizations are more amenable to adopting EBPs than are others ( Real & Poole, 2005; Schoenwald & Henggeler, 2003), and the identification of agreeable parties would facilitate the dissemination efforts of both treatment developers and service funders. Moreover, in the current era of policy makers mandating the adoption of EBPs, it is essential to develop tools that will enable organizations to develop an infrastructure that is most likely to address the strengths and needs of the practitioners “on the front lines” to promote following of the mandate. Yet, empirically-based methods for assessing amenability to the adoption of EBPs have not been well validated and do not exist specifically for adolescent substance abuse treatment programs. The primary purpose of this study, therefore, is to provide an additional step toward the validation of a promising measure of practitioner and organizational readiness to adopt EBPs and to evaluate the measure with a sample of practitioners from the adolescent treatment field. Based on a well-conceived conceptual framework for transferring research to practice ( Simpson, 2002), the Organizational Readiness for Change (ORC; Lehman, Greener, & Simpson, 2002) scale was developed to assess the motivation, attributes of program leadership, institutional resources, and organizational climate of programs that are considering the adoption of new substance abuse practices. In contrast with some organizational measures that focus on physical resources and work environment (e.g., Moos & Moos, 1998), the ORC was designed to measure staff and organizational characteristics related to importing new technologies. Items on the ORC tap agency staff perceptions and cognitive appraisals regarding motivation and organizational capacity for change across five domains: (1) motivational readiness for change, (2) adequacy of resources, (3) staff attributes, (4) organizational climate, and (5) practitioner access to and utilization of training ( Institute of Behavioral Research, 2005). Within each of these domains, subscales have been constructed for items that are contextually similar (e.g., immediate training needs, computer access, communication, adaptability), and the majority of these subscales have demonstrated acceptable levels of reliability at both staff and program levels. Moreover, the ORC has demonstrated the ability to distinguish between agency staff versus directors on factors consistent with these positions (e.g., directors reported higher propensity for change in program orientation than did staff). The ORC, therefore, seems to be a promising measure of a key construct in efforts to bridge the science-practice gap -- evaluating practitioner and agency readiness to adopt new technologies. As noted previously, the ORC has been used in the evaluation of organizations conducting adult substance abuse treatment, whereas the current study focuses on adolescent substance abuse treatment. Thus, a secondary goal of the current paper is to validate this measure with practitioners who treat substance abusing youth. Although initial properties of the ORC are promising, further evaluation of the instrument is needed to support its intended use. As with all psychometrically sound instruments, after reliability has been established, validation of the measure can follow. The current study examines two forms of criterion related validity of the ORC. Specifically, data from a larger study were used to evaluate the criterion--convergent and concurrent--validity of the measure and its subdomains. Convergent validity refers to the degree of association between two measures of similar constructs. Thus, in order to evaluate whether a tool is assessing what it is thought to measure, the associations between the respective tool and other already validated instruments of a similar construct are examined. On the other hand, concurrent validity refers to a measure’s ability to distinguish between individuals or groups that theoretically should be expected to respond differently on the measure. Thus, in order to assess the concurrent validity of an instrument, hypotheses are made with regard to person characteristics that might distinguish one group from another. In the current study, criterion validity of the ORC was evaluated for practitioners who encounter adolescent substance abuse. For the assessment of convergent validity, the ORC was evaluated in relation to practitioner attitudes toward factors thought to be important for the dissemination of EBPs. Specifically, the association between ORC domains and measures of practitioner attitudes toward EBPs (i.e., Evidence-Based Practice Attitude Scale; Aarons, 2004) and the use of treatment manuals (i.e., questionnaire developed and validated by Addis and Krasnow, 2000) was evaluated. Favorable practitioner attitudes toward EBPs and the use of treatment manuals were expected to converge with high ORC scores. In addition, the concurrent validity of the ORC was examined by considering variables that might discriminate between readiness for change for groups of practitioners that differed in professional background and experience characteristics such as addition certification, service sector, caseload size, and years of experience ( Roman & Johnson, 2002). Given the limitations of the data available at the time of this writing, a third type of criterion validity, predictive, was not considered in this evaluation of the ORC. That is, the current manuscript relies on data collected from practitioners at the baseline assessment of a longitudinal study. Thus, data were not available to evaluate subsequent practitioner use (i.e., implementation) of evidence-based practice. An evaluation of the concurrent and convergent validity of the ORC specific to practitioners’ attitudes toward evidence-based practice, however, is a critical step in assessing the measure’s utility to gauge organizational readiness to change specific to the adoption of EBP. Practitioners from 44 public sector substance abuse and mental health treatment agencies across the state of South Carolina participated in this project, providing a highly diverse sample of adolescent treatment providers. 2.1. Design The data from this study represent a subset of that collected during a larger longitudinal project that was conducted across the state of South Carolina ( Henggeler et al., in press). The primary purposes of the larger project are to examine public sector practitioner adoption and implementation of contingency management for treating adolescent substance abuse. As described subsequently, the present ORC validation study uses the self-report data obtained at the start of the study from the consenting practitioners. 2.2 Site and Participant Recruitment In the state of South Carolina, adolescents are served by 33 Department of Alcohol and Other Drug Abuse Services (DAODAS) publicly funded provider organizations and 17 Department of Mental Health (DMH) community mental health centers. Considerable resources were devoted to agency and practitioner recruitment to obtain a diverse and unbiased sample. First, the project investigators met with all executive and treatment directors from DAODAS and DMH agencies to provide an overview of the project and facilitate support. Leadership in 88% of these agencies agreed to support the project. Second, letters were sent to each practitioner treating adolescents at each site from the respective DAODAS and DMH state commissioners and project investigators to introduce the study and invite attendance to a research recruitment session provided at the agency site. Third, the research team visited each site at a time convenient to the site staff, and individualized arrangements (e.g., counselors not available at that time) were made when necessary. To facilitate attendance, the visiting research team provided beverages and snacks for the adolescent-treating practitioners at the site who attended the meeting. The research team introduced and described all aspects of the project, emphasizing the voluntary nature of research participation and that participation would have no impact on the practitioners’ job performance evaluation. After answering any questions that the eligible practitioners might have, they were consented to the study per approval from the University’s Institutional Review Board. Ninety-seven percent of the eligible DAODAS practitioners (178 of 183) and 81% of the eligible DMH practitioners (365 of 453) consented to participate, yielding a final sample size of 543 practitioners. The number of therapists per agency ranged from 2 to 82, with an average of 12.3 (SD = 14.1). 2.3 Participants Demographically, 80% of the 543 consenting participants were female; 41% were African-American, 57% white non-Hispanic, 1% Hispanic, and 1% other; and their average age was 40 years. Professionally, 16% had Bachelor’s degrees, 82% had Master’s degrees, and 1% had doctorates. On average, these practitioners had almost 10 years of professional experience, and 22% were certified addictions counselors. 2.4 Research Procedures As part of consenting procedures, participants were provided with a description of the study. They were informed that the purpose of the study was to determine who would attend a workshop providing instruction in contingency management for the treatment of adolescent marijuana abuse, and of those who attended, who would try to implement the intervention. At this time they were not provided with a detailed overview of the contingency management protocol, rather they were informed that the workshop opportunity would be a single day. Participants who consented were told that although workshop attendance was voluntary, if they did not attend a researcher would call them to ask about their reasons for not attending. Likewise, they were informed that if they did attend, they would have access to resources for implementing the intervention for 6 months following the workshop and they would receive brief monthly calls to ask about their use of the intervention. Immediately following informed consent, practitioners were administered a self-report assessment battery, described subsequently, lasting approximately 45 minutes. Practitioners who completed the questionnaires were reimbursed $20 for their time. 2.5 Measures 2.5.1 Practitioner demographics, professional experience, and service sector The Personnel Data Inventory ( Schoenwald, 2003) was used to collect demographic information as well as information regarding therapists’ professional characteristics and experience (i.e., highest degree, years of experience, caseload size, proportions of adults and youths in caseload, type of service delivery model used, and certification in the addictions field). In addition, the current service sector (i.e., mental health or substance abuse) of the practitioner was noted. 2.5.2 Organizational characteristics As noted previously, the Organizational Readiness for Change instrument - Program Staff Version (ORC-S; Lehman et al., 2002) was the primary assessment of interest in the current study. The ORC includes 129 5-point Likert scale items developed to tap key constructs in a theoretical process model of program change ( Simpson, 2002). Domains measure Motivational Readiness (e.g., perceived need and pressure for change, immediate training needs), Adequacy of Resources (e.g., offices, staffing, training, computer access, e-communications), Staff Attributes (e.g., growth, efficacy, influence, adaptability), Organizational Climate (e.g., clarity of mission, cohesion, autonomy, communication, stress, change), and Training Exposure and Utilization (e.g., frequency of attendance, adoption of new techniques). Scoring procedures that are suggested by the developers ( Lehman et al., 2002) were followed. Each domain has demonstrated satisfactory reliability and internal consistency at the staff, director, and program levels of evaluation ( Lehman et al., 2002). Internal consistency of the ORC also was evaluated for the current sample due to a) the focus of this study on this particular measure; b) the use of a sample of practitioners who treat adolescents versus only adults; and c) the need to establish reliability prior to testing dimensions of validity. Estimation of the internal consistency of each domain of the ORC is complicated, however, by the nesting of therapists within agencies. Failure to partition the variance in ORC responses according to the nested data structure has the potential to yield ambiguous and misleading estimates of the reliability of these domains ( Raudenbush, Rowan, & Kang, 1991). Using a multilevel measurement model as described by Raudenbush et al. (1991), item responses for each ORC domain (level-1) were nested within therapists (level-2) who were nested within agencies (level-3), yielding a total of five models. Each estimate of internal consistency was computed based on the formulas provided by Raudenbush and Bryk (2002), resulting in a therapist- and agency-level reliability estimate for each domain. The average reliabilities for the ORC domain scores across therapists were adequate, albeit modest (i.e., .80, .44, .64, .48, and .67, for Motivational Readiness for Change, Adequacy of Resources, Staff Attributes, Organizational Climate, and Training Exposure and Utilization, respectively) as were the reliabilities across agencies (i.e., .57, .64, .42, .32, and .62, respectively). 2.5.3 Practitioner attitudes toward evidence-based practice The 15-item Evidence-Based Practice Attitude Scale (EBPAS; Aarons, 2004) was used to assess general attitudes toward the adoption of evidence-based practices. Items are rated on 5-point Likert scales (0 = not at all; 1 = to a slight extent; 2 = to a moderate extent; 3 = to a great extent; 4 = to a very great extent). The EBPAS includes four theoretically derived subscales (i.e., Appeal, Requirements, Openness, and Divergence) that assess: the intuitive appeal of EBPs for adoption (e.g., “If you received training in a therapy or intervention that was new to you, how likely would you be to adopt it if it was being used by colleagues who were happy with it?”); extent of practitioner adoption if required by authorities (e.g., “If you received training in a therapy or intervention that was new to you, how likely would you be to adopt it if it was required by your agency?”); practitioner openness to new interventions (e.g., “I like to use new types of therapy/interventions to help my clients.”); and practitioner perceptions of EBPs as less relevant than clinical experience (i.e., Divergence; e.g., “Clinical experience is more important that using manualized therapy/interventions.”). Confirmatory factor analysis of the EBPAS supports a four scale factor structure, with adequate internal consistency for both the subscales and the overall total scale. Scores have demonstrated association in meaningful directions with important practitioner and organizational characteristics (Aarons, 2004; 2005). 2.5.4 Practitioner attitudes toward treatment manuals A questionnaire developed and validated by Addis and Krasnow (2000) was used to evaluate staff attitudes toward treatment manuals. Principal-components analyses of the 17 items, rated on 5-point Likert scales (1 = strongly disagree, 5 = strongly agree), suggest a two-factor structure: Positive Outcomes and Negative Process. Positive Outcomes items reflect practitioner perceptions that treatment manuals are valuable in guiding clinicians toward favorable outcomes with their clients (e.g., “Following a treatment manual will enhance therapeutic outcomes by insuring that the treatment being used is supported by research.”). Negative Process items characterize treatment manuals as having a dehumanizing effect on the therapeutic process and emphasizing technique at the expense of relationship skills (e.g., “Using a treatment manual keeps a therapist from using his or her intuition in responding to a client.”). These attitudes have been shown to vary in predictable directions with practitioner theoretical orientations and work settings. 2.6 Data Analysis 2.6.1 Within agency agreement Prior to analysis, the level of within agency agreement in therapist ORC reports was evaluated ( Chan, 1998; Glisson & James, 2002). High levels of agreement indicate that the organizational aggregate score should be used in analyses. Two sources of information were used to guide the decision of whether to treat ORC scores at the individual versus agency level. First, r wg, a widely adopted estimate of within-organization interrater agreement ( Castro, 2002; James, Demaree, & Wolf, 1984), was computed for each organization based on the observed and expected variance and the number of items ( James, Demaree, & Wolf, 1993). These estimates were averaged across agencies, and scores in excess of .70 provide justification for organizational level aggregation (e.g., George, 1990). Second, the intraclass correlation coefficient (ICC; Snijders & Bosker, 1999) was used to estimate the percentage of variability in therapist ORC scores residing at the agency level. The ICCs for each ORC domain were computed through the estimation of an unconditional (i.e., no predictor variables entered) random-effects regression model for each domain. The variance components for level-1 (i.e., therapist level) and level-2 (i.e., agency level) were used to calculate ICCs according to τ 2/(σ 2 + τ 2), where τ 2 is the level-2 variance component, σ 2 is the level-1 variance component, and ICC is the proportion of total model variance represented at level-2. ICCs also were computed for each of the six subscales on the EBPAS and measure of attitudes toward treatment manuals. ICCs greater than or equal to .10 are considered meaningful and support the use of a nested model. 2.6.2 Random-effects regression models Two features of the present data required the use of random-effects regression models (RRMs). First, therapists (level-1) are nested within - agencies (level-2), necessitating modeling of outcome variability at each level. Second, dependent upon ICCs and r wg, ORC scores might be modeled most appropriately at therapist and/or agency levels ( Castro, 2002). Continuous RRMs were performed using restricted maximum likelihood estimation in Hierarchical Linear and Nonlinear Modeling software (HLM 6; Raudenbush, Bryk, Cheong, & Congdon, 2004). The decision to model level-1 covariates as fixed or randomly-varying was made on the basis of the chi-square test for the variance components. Given the small number of participating agencies, regular rather than robust standard errors were used for the computation of test statistics (Maas & Hox, 2002; Raudenbush & Bryk, 2002). Centering decisions were guided by the substantive research questions ( Hofmann & Gavin, 1998). The percentage of explained variance was computed for significant level-1 and level-2 predictors according to the approach described by Snijders and Bosker ( 1994, 1999). 3.1 Preliminary Analyses 3.1.1. Within agency agreement ICCs and r wg scores for the five ORC domains are presented in Table 1. The agency mean r wg scores for each domain exceeded .70. Likewise, ICCs for four of five domains exceeded .10. Together, these results support aggregation of ORC predictor variables at the agency level. That is, therapists in a given agency generally agreed in their ORC reports such that the average of their scores should be used as an agency-level predictor in the RRM. | Table 1Intraclass Correlation Coefficients (ICCs) and Mean Agency rwg Scores on the Organizational Readiness to Change (ORC) Scale. |
On the other hand, ICCs for the six outcome variables (i.e., scales for practitioner attitudes toward EBPs and treatment manuals; not presented in the table) revealed only a small portion of variance at agency level (i.e., ICCs ranged from .001 to .03). Thus, although the ORC domains can be aggregated at the agency-level, there is little between-agency variance in attitudes toward EBPs and the use of treatment manuals for these scores to predict. Additionally, Lehman and colleagues (2002) argue that ORC scores are expected to operate differently at the staff versus agency level, despite being derived from the same questionnaire. Consequently, both therapist- and agency-level ORC scores (i.e., individual therapist reports and the average score for all therapists within a given agency) were used to predict therapist attitudes toward EBPs and treatment manuals using a contextual model presented next. 3.1.2 Contextual model As described by Hofmann and Gavin (1998) and Raudenbush and Bryk (2002), the following contextual RRM was tested: In this model, ( Xij - Xj) represents the group mean centered level-1 ORC score (i.e., the value of the predictor is the deviation of the individual therapist’s report from his/her agency average). At level-2, (  ) represents the deviation of each agency average ORC score from the agency grand mean. ORC scores were entered simultaneously, resulting in five group mean centered level-1 predictors of within-agency variation and five grand mean centered level-2 predictors of between-agency variation. Chi-square tests for the variance components were not significant for the level-1 ORC scores suggesting no significant variation in the way the ORC operated in relation to therapist attitudes between agencies. As a result, ORC scores were modeled as fixed effects, as indicated by the omission of the unique agency effect for ORC (i.e., u 1j). Of note, service sector was tested as a moderator of the results presented subsequently. In all cases, however, the results did not differ significantly by service sector (i.e., substance use treatment agency versus mental health agency). 3.1.3 Evaluation of multicollinearity A benefit of the centering strategy for the contextual model is that, for each domain, the therapist-level score is orthogonal to the agency-level aggregate score ( Raudenbush, 1989). The potential for collinearity between the five therapist-level domain scores as well as the five agency-level domain scores remains, however ( Shieh & Fouladi, 2003). To evaluate the magnitude of these associations, the bivariate correlations were computed for (a) the therapist-level deviations from the respective agency means and (b) the agency-level scores for the five domains. For the therapist-level deviation scores, the ten correlation coefficients ranged in magnitude from .07 to .54, with one correlation exceeding .50. For the agency-level scores, the ten correlation coefficients ranged in magnitude from .01 to .73, with three correlations exceeding .50. Despite the presence of some moderate correlations, the present data have several protective factors against potential bias from collinearity, including: (a) no cross-level interaction terms, (b) a moderate number of agencies and number of therapists within agencies, (c) small intraclass correlation coefficients for the outcome variables, and (d) estimation of a random intercept only. According to the simulations of Shieh and Fouladi (2003), the fixed effects and the therapist-level variance component are estimated with little bias, though the agency-level variance component may exhibit bias. In this case, this only refers to the random intercept as random effects were not modeled for the therapist-level scores. The standard errors ( SEs) for the fixed effects (i.e., the effects of the model covariates) are expected to be conservatively estimated in the presence of multicollinearity. Thus, based on the correlations between predictors as detailed above, the expectation is that model SEs are estimated conservatively, yielding a conservative statistical test. 3.2 Convergent Validity Tables 2 and 3 provide the specific results with regard to the association between ORC scores and scores on each of the outcome measures (i.e., Evidence Based Practice Attitudes Scale and Addis and Krasnow’s measure of attitudes toward treatment manuals). As shown, the relationship between ORC scores and practitioner attitudes toward EBPs are most significant at the therapist level of the contextual model versus the agency level. These results suggest that, in general, individual therapists’ responses on the ORC provided a better estimate of their attitudes toward EBP and treatment manuals than did the average score for their agency. | Table 2Contextual Model of the Association of Practitioner Ratings on the Organizational Readiness to Change and Their Attitudes Toward Evidence-Based Practices. |
| Table 3Contextual Model of the Association of Practitioner Ratings on the Organizational Readiness to Change and Their Attitudes Toward Use of Treatment Manuals. |
3.2.1 Appeal Therapists’ responses on the ORC Motivational Readiness, Staff Attributes, and Training Exposure and Utilization domains were associated positively with their ratings of the intuitive Appeal of adopting EBPs. On the other hand, the perceived ORC Adequacy of Resources domain was associated negatively with Appeal. Collectively, the therapist-level ORC scores accounted for 5% of the variance in Appeal. Agency-level ORC scores were not significantly associated with agency-average scores for the intuitive appeal of adopting EBPs. 3.2.2 Requirements At the therapist-level, ratings on the ORC Training Exposure and Utilization domain were associated positively with the likelihood of adopting EBPs when required by authorities. Collectively, the level-1 ORC scores accounted for 2% of the variance in Requirements. Agency-level ORC scores were not significantly associated with practitioner ratings of adopting EBPs when required by authorities. 3.2.3 Openness Therapist-level ratings of ORC Motivational Readiness, Staff Attributes, and Training Exposure and Utilization were associated positively with their openness to new interventions. In addition, at the agency-level, Motivational Readiness was associated positively with openness to new interventions. Collectively, the level-1 and level-2 ORC scores accounted for 9% and 1% of the variance in Openness, respectively. 3.2.4 Divergence Therapist-level ratings of ORC Staff Attributes were associated positively with perceptions of EBPs as being less relevant than clinical experience (i.e., Divergence). On the other hand, therapist-level perceptions of ORC Organizational Climate were associated negatively with Divergence (i.e., giving greater value to EBPs). Collectively, the level-1 predictors accounted for 2% of the variance in Divergence. Agency-level ORC scores were not significantly associated with practitioner perceptions of EBPs as being less relevant than clinical experience. 3.2.5 Positive Attitudes As shown in Table 3, the therapist-level ORC Motivational Readiness for Change domain was associated positively with Positive Attitudes toward the use of treatment manuals. Agency-level Motivational Readiness also was associated positively with Positive Attitudes toward the use of treatment manuals. Collectively, the level-1 and level-2 predictors accounted for 6% and 13% of the variance in Positive Attitudes, respectively. 3.2.6 Negative Attitudes Therapist-level ratings on the ORC Training Exposure and Utilization domain were inversely related with Negative Attitudes toward the use of treatment manuals. Collectively, however, the level-1 predictors accounted for less than 1% of the variance in Negative Attitudes. Agency-level ORC scores were not significantly associated with Negative Attitudes toward the use of treatment manuals. 3.3 Concurrent Validity The next set of analyses tested the extent to which individual practitioner ORC reports (level-1) differed according to practitioner-level (i.e., professional training, caseload characteristics) and agency-level (i.e., service sector) covariates. 3.3.1 Training Practitioners with a Masters degree or higher [γ10 = 0.10, SE = .044, t (531) = 2.29, p = .02], school-based counselors [γ20 = 0.10, SE = .046, t (531) = 2.18, p = .03], and more experience practitioners [γ70 = .01, SE = .003, t (531) = 2.48, p = .01] provided higher ratings on the ORC Staff Attributes domain than did their respective counterparts. Similarly, ratings on the ORC Training Exposure and Utilization domain were higher for more experienced practitioners [γ70 = .01, SE = .004, t (531) = 2.41, p = .02]. Addiction certification was not associated with the ORC domains. 3.3.2 Caseload characteristics Practitioners with larger caseloads provided lower ratings on the ORC Adequacy of Resources [γ10 = -0.03, SE = .014, t (531) = -2.04, p = .04], Organizational Climate [γ10 = -0.03, SE = .012, t (531) = -2.15, p = .03], and Training Exposure and Utilization [γ10 = -0.04, SE = .015, t (531) = -2.67, p = .01] domains. Furthermore, practitioners treating a higher proportion of youth as opposed to adult clients provided lower ratings on the ORC Staff Attributes domain [γ20 = -0.03, SE = .012, t (531) = -2.54, p = .01]. 3.4.3 Service sector Practitioners in drug and alcohol provider agencies reported higher average ratings on the ORC Adequacy of Resources [γ01 = 0.27, SE = .098, t (42) = 2.76, p = .01] and Training Exposure and Utilization [γ01 = 0.29, SE = .079, t (42) = 3.71, p < .01] domains than did their mental health sector counterparts. The findings support both the convergent and concurrent validity of the ORC and suggest potentially important findings regarding the assessment of an organization’s readiness to adopt EBPs for youth substance abuse. These findings are particularly important as the field of adolescent substance abuse treatment moves toward transporting EBPs into community settings. These findings suggest that the ORC shows promise in being able to identify those organizations, and within organizations, those therapists, that are most receptive to adopting new treatment technologies. Such identification can assist in informing decisions regarding the strengths and needs of agencies to help increase the likelihood of success in importing EBPs. Although the focus of the present paper is the validity of the ORC, a separate but related finding warrants mention. A substantial proportion of the variance in the ORC domains was found to reside at the agency level; however, the opposite was found for therapist attitudinal scores. Specifically, nearly all of the variance in therapist attitudes toward evidence-based practice and treatment manuals was at the level of the individual therapist. This has important implications for programmatic change, suggesting that the implementation of new evidence based practices should consider the individual attitudes of the therapists who are affected by the change. That is, the practitioners “on the front lines” are those who are most affected by organizational change when that change directs treatment decisions. Thus, it is critical for the decision-makers to value the input of these therapists when creating change and to have an understanding of what is driving their attitudes (e.g., perceptions of the use of manuals, of EBP). 4.1 Therapist Attitudes toward EBPs and Treatment Manuals In support of Simpson’s (2002) process model of program change for transferring research to practice, the ORC Motivational Readiness and Training Exposure and Utilization domains examined at the individual practitioner level were consistently related to indices of practitioner amenability to adopt EBPs. Specifically, Motivational Readiness was associated positively with the intuitive appeal of EBPs to practitioners, practitioner openness to new interventions, and a view that treatment manuals can facilitate favorable outcomes with clients. The latter two findings also were observed when Motivational Readiness was examined at the agency level. Thus, those practitioners who have positive attitudes and perceptions of EBP and manualized protocols are more likely to demonstrate a high motivation and readiness to adopt EBP for adolescent substance abuse. Likewise, Training Exposure and Utilization was positively associated with the intuitive appeal of EBPs to practitioners, practitioner openness to new interventions, and the extent of adoption if required by authorities; and negatively associated with pejorative characterizations of treatment manuals. In combination, these findings suggest that there is an association between exposure of the therapists to new treatments with an emphasis on positives of EBPs and manualized protocols, and practitioners’ perceived value in learning such methods. These findings provide support for the convergent validity of the Motivational Readiness and Training Exposure and Utilization domains. Moreover, and importantly, the findings are consistent with the proximal connections among Personal Readiness, Training Exposure, and Adoption proposed in the process model of program change ( Simpson, 2002). Thus, these ORC domains appear to be effectively tapping therapist interest in adopting EBPs. On the other hand, the ORC Adequacy of Resources and Organizational Climate domains were not associated clearly with the indices of practitioner amenability to adopt EBPs. That is, practitioner perceptions of resources such as supplies, materials, and adequate space and their reports of the environment and personal dynamics within their agency did not appear to be related to their amenability to adopt EBPs. Although a more favorable Organizational Climate was associated with practitioner perceptions of the relevance of EBPs in comparison with clinical practice experience, this relationship was relatively small and Organizational Climate was not associated with the other measures of therapist attitudes toward EBPs and treatment manuals. Moreover, Adequacy of Resources was associated with only one variable (i.e., the intuitive appeal of EBPs), and that was in the direction opposite of expectations (i.e., high resources were associated with low appeal of EBPs). Although these findings do not seem to support the convergent validity of these particular ORC domains, the results are not necessarily incompatible with the process model of program change ( Simpson, 2002). The process model posits that Adequacy of Resources and Organizational Climate are most pertinent in determining the quality of implementation of EBPs. The practitioner attitudinal variables in the present study, however, are likely tapping constructs closer to interest in adoption of EBPs than to the quality of implementation after adoption. Indeed, data presented in this study were collected at baseline and did not assess behaviors associated with treatment implementation. Thus, the design of the present study does not necessarily provide a fair evaluation of the convergent validity of the Adequacy of Resources and Organizational Climate domains. The fifth ORC domain, Staff Attributes, also provided mixed results. Supporting the convergent validity of this instrument, favorable staff attributes (e.g., confidence in their own counseling skills, ability to adapt to changing environment) were positively associated with the intuitive appeal of EBPs to practitioners and to their openness to new interventions. On the other hand, favorable staff attributes also were associated with the view that clinical experience is more important than research evidence (i.e., Divergence). Yet, when one considers that practitioners with high scores on Staff Attributes generally feel confident in their own abilities (as indicated by the subscales that make up the Staff Attributes domain), the high value placed on their personal clinical experience is understandable. Regarding fit with the process model of program change ( Simpson, 2002), staff attributes are posited as a primary determinant of whether innovations become incorporated into standard practice. As noted previously, the present study did not examine this final stage in the technology transfer process. In sum, the findings provide support for the convergent validity of those ORC domains (i.e., Motivational Readiness and Training Exposure and Utilization) that are most proximally linked with practitioner amenability to the adoption of EBPs. Although less evidence was observed for the convergent validity of those ORC domains that are posited as key determinants of EBP implementation and sustainability (i.e., Adequacy of Resources, Organizational Climate, and Staff Attributes), the emphasis of the present study on practitioner attitudes toward versus adoption of EBPs and treatment manuals suggests that the evaluation of these relationships do not provide a strong test of the convergent validity of these particular ORC domains. Rather, implementation research would be more pertinent for validating these measures ( Fixen, Naoom, Blasé, Friedman, & Wallace, 2005). 4.2 Therapist Training, Caseload, and Service Sector The ORC domains are intended to identify organizational traits and barriers that can predict program change and help diagnose structures that are more or less amiable to change ( Lehman et al., 2002). Thus, a test of the concurrent validity of the ORC domains examines the measure’s ability to distinguish between practitioners faced with different practical realities or barriers in providing substance abuse services to adolescents. Identification of such barriers can facilitate the development of dissemination strategies that promote the successful adoption of EBPs ( Simpson, 2002). One set of findings that stands out in support of the concurrent validity of the ORC is the negative associations between caseload size and the ORC Training Exposure and Utilization, Adequacy of Resources, and Organizational Climate domains. Therapists with large caseloads clearly perceive themselves to be under resourced, having little opportunity for training, and embedded within organizational climates (e.g., low autonomy, high stress) that are not conducive to fulfilling the organization’s mission. These results also are pertinent to the findings that practitioners in the substance abuse treatment sector reported higher scores on the ORC Training Exposure and Utilization and Adequacy of Resources domains than did their counterparts in the mental health service sector. That is, mental health sector therapists have significantly higher caseloads than substance abuse sector practitioners [t (541) = 3.62, p <.001]. Together, these findings further support the concurrent validity of the ORC domains and, importantly, suggest that the adoption of EBPs is more likely among practitioners with lower caseloads. The ORC Staff Attributes domain was significantly associated with several therapist characteristics. Master’s level clinicians, more experienced practitioners, school-based therapists, and therapists with a relatively high percentage of adults in their caseloads reported higher scores on the Staff Attributes domain (e.g., confidence in their own skills) than did their respective counterparts. Although the bases of these findings are not entirely clear, they might be related to the degree of clinical challenge experienced by the practitioners. Master’s level and more experienced therapists should, logically, feel more confident in their work than their less educated and experienced counterparts. School-based therapists typically function in relatively controlled and circumscribed (e.g., little family contact) environments and often focus on less serious clinical problems. Similarly, many therapists are more confident in treating adults, who are usually voluntary clients, than adolescents, who rarely seek treatment themselves and present a myriad of family and social system problems. Finally, more experienced therapists reported higher scores on the ORC Training Exposure and Utilization domain as would be expected. These findings as well as those pertaining to caseload size support the emerging concurrent validity of the ORC. 4.3 Limitations The study included several limitations at the psychometric level of evaluation that support the need for continued research on the ORC. First, the nature of the measurement methods (i.e., focus on therapist attitudes toward EBPs and treatment manuals) provided fair tests of the convergent validity of some ORC domains (i.e., Motivational Readiness and Training Exposure and Utilization), but not necessarily others (i.e., Adequacy of Resources, Organizational Climate, and Staff Attributes). Thus, the ability to draw firm conclusions about the validity of the entire instrument is incomplete. A more thorough evaluation would examine the relationship of the ORC with measures of EBP implementation. Second, given the moderate level of correlation between some of the domains, the potential effect on the results of simultaneous entry of the scores into the model, versus individual entry, should be considered. Although the present data have several “protective” factors against bias from multicollinearity (e.g., moderate number of agencies, moderate cluster size, small intraclass correlation coefficients; Shieh, & Fouladi, 2003), the actual impact of these correlations on the findings is unclear. The alternative analytic strategy of separate entry of each ORC domain, however, would serve to inflate Type I error and would inaccurately treat each domain as being entirely independent. Thus, given that the correlations could be suggestive of the ORC domains not forming unique dimensions, further evaluation of the dimensionality and factor structure of the ORC should be considered in future psychometric work. Third, although there were limited significant findings at the agency level, these results should be viewed with caution. As noted in the preliminary analyses, there was little between-agency variance in the outcome measures used to assess for concurrent validity. Thus, it would be inaccurate to assume that the ORC will never show association at the agency level with other assessment tools; rather, when compared to other measures that might also be better analyzed with aggregation at the agency level, greater agency level findings might emerge. Other limitations of the study pertain to the confines of the design. One participant related issue is the public sector nature of the sample. That is, the findings should not be generalized to private sector practitioners and organizations. Another limitation is that this validation study was conducted with practitioners from the adolescent substance use field, whereas the instrument has been used previously with adult treatment providers. A more thorough test of validity would be to draw from a mixed sample of practitioners and compare their responses using a multigroup method of analysis. Finally, but importantly, although convergent and concurrent validity are essential in the criterion validation of a research instrument, the determination of predictive validity is critical as well. Although such longitudinal analyses are not available at this time, future research will examine the predictive abilities of the ORC. Findings from predictive analyses might allow users to predict practitioner and organizational related outcomes (e.g., adherence to EBP model, success of implementation) and eventually to help assess variables related to client treatment outcomes. As noted next, the ultimate value of the highly promising ORC domains will be determined by their prospective associations with the adoption, implementation, and sustainability of EBPs. 4.4 Conclusion and Future Directions Findings from the current study are consistent with the intention of the ORC to identify those practitioners and organizations that have positive attitudes toward implementation of new technologies for the treatment of substance abuse. Each of the five ORC domains is theorized to be influential in evaluating the process model of program change ( Simpson, 2002), and relatively strong support for convergent validity was observed for several of the domains. The cross-sectional nature and measurement limitations of the current study, however, restricted a more complete evaluation of the associations between the ORC domains and the adoption, implementation, and sustainability phases of the process model of program change. Indeed, and as noted previously, the data from the present study are part of a larger longitudinal project that is examining both the adoption of an EBP for the treatment of adolescent substance abuse (i.e., contingency management) and the fidelity of its implementation. Thus, data from this longitudinal study will provide the opportunity to examine the predictive validity of the ORC domains with regard to important components of Simpson’s (2002) process model of program change. Nevertheless, despite the continued needs for validation, findings from the present study support the ORC as one of the first organizational measures that can help identify practitioners and agencies within the adolescent service field that are likely amenable to importing EBPs and those for whom additional supports might need to be provided to “bring them on board” successfully. Such organizational measures provide an encouraging step toward the ability to bridge the gap between research and practice. This manuscript was supported by grant R01DA17487 from the National Institute on Drug Abuse. We sincerely thank the many executive and treatment directors of the DAODAS and DMH provider organizations for their support in facilitating the success of this project, and special thanks are extended to the state level leadership including Dr. George Gintoli, W. Lee Catoe, Louise Johnson, James Wilson, and Ruthie Johnson. Special appreciation is also extended to the research staff that performed at an extremely high level of professionalism, including Jennifer Shackelford, Kevin Armstrong, Ann Ashby, and Geneene Thompson. - Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale. Mental Health Services Research. 2004;6:61–72. [PubMed]
- Aarons GA. Measuring provider attitudes toward evidence-based practice: Consideration of organizational context and individual differences. Child and Adolescent Psychiatric Clinics of North America. 2005;14:255–271. [PubMed]
- Addis ME, Krasnow AD. A national survey of practicing psychologists’ attitudes toward psychotherapy treatment manuals. Journal of Consulting and Clinical Psychology. 2000;68:331–339. [PubMed]
- Castro SL. Data analytic methods for the analysis of multilevel questions: A comparison of intraclass correlation coefficients, rwg(j), hierarchical linear modeling, within- and between-analysis, and random group resampling. The Leadership Quarterly. 2002;13:69–93.
- Chan D. Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology. 1998;83:234–246.
Fixen DL, Naoom SF, Blasé KA, Friedman RM, Wallace F Implementation research: A synthesis of the literature 2005. University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; Tampa, FL: (FMHI Publication #231). - George JM. Personality, affect, and behavior in groups. Journal of Applied Psychology. 1990;75:107–116.
- Glisson C, James LR. The cross-level effects of culture and climate in human service teams. Journal of Organizational Behavior. 2002;23:767–794.
Henggeler SW, Chapman JE, Rowland MD, Halliday-Boykins CA, Randall J, Shackelford J, Schoenwald SK (in press) If you build it, they will come: A statewide study of practitioner and organizational willingness to learn about an evidence-based practice for adolescent substance abuse Journal of Substance Abuse Treatment . - Hofmann DA, Gavin MB. Centering decisions in hierarchical linear models: Implications for research in organizations. Journal of Management. 1998;24:623–641.
- Institute of Behavioral Research. TCU survey of organizational functioning (program staff version): Scales and item scoring guide for program staff survey of Organizational Readiness for Change. TCU Institute of Behavioral Research; Fort Worth, Texas: 2005.
- James LR, Demaree RG, Wolf G. Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology. 1984;69:85–98.
- James LR, Demaree RG, Wolf G. rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology. 1993;78:306–309.
- Lehman WEK, Greener JM, Simpson DD. Assessing organizational readiness for change. Journal of Substance Abuse Treatment. 2002;22:197–210. [PubMed]
- Maas CJM, Hox JJ. Robustness issues in multilevel regression analysis. Statistica Neerlandica. 2004;58:127–137.
- Moos RH, Moos BS. The staff workplace and the quality and outcome of substance abuse treatment. Journal of Studies on Alcohol. 1998;59:43–51. [PubMed]
- Raudenbush S. “Centering” predictors in multilevel analysis: Choices and consequences. Multilevel Modelling Newsletter. 1989;1:10–12.
- Raudenbush SW, Bryk AS. Hierarchical linear models: Applications and data analysis methods. 2nd ed. Sage; Thousand Oaks, CA: 2002.
- Raudenbush SW, Bryk AS, Cheong YF, Congdon R. HLM 6: Hierarchical linear & nonlinear modeling [computer software and manual] Scientific Software International; Lincolnwood, IL: 2004.
- Raudenbush SW, Rowan B, Kang SJ. A multilevel, multivariate model for studying school climate with estimation via the EM algorithm and application to U.S. high-school data. Journal of Educational Statistics. 1991;4:295–330.
- Real K, Pool MS. Innovation implementation: Conceptualization and measurement in organizational research. Research in Organizational Change and Development. 2005;15:63–134.
- Roman PM, Johnson JA. Adoption and implementation of new technologies in substance abuse treatment. Journal of Substance Abuse Treatment. 2002;22:211–218. [PubMed]
- Schoenwald SK. Personnel Data Inventory. Family Services Research Center, Medical University of South Carolina; Charleston, SC: 2003.
- Schoenwald SK, Henggeler SW. Current strategies for moving evidence-based interventions into clinical practice: Introductory comments. Special Series, Current strategies for moving evidence-based interventions into clinical practice. Cognitive and Behavioral Practice. 2003;10:275–277.
- Shieh YY, Fouladi RT. The effect of multicollinearity on multi-level modeling parameter estimates and standard errors. Educational and Psychological Measurement. 2003;63:951–985.
- Simpson DD. A conceptual framework for transferring research to practice. Journal of Substance Abuse Treatment. 2002;22:171–182. [PubMed]
- Snijders TAB, Bosker RJ. Modeled variance in two-level models. Sociological Methods & Research. 1994;22:342–363.
- Snijders TAB, Bosker RJ. Multilevel analysis: An introduction to basic and advanced multilevel modeling. Sage; London: 1999.
|
The publisher's final edited version of this article is available at 