• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of implemsciBioMed CentralBiomed Central Web Sitesearchsubmit a manuscriptregisterthis articleImplementation Science : ISJournal Front Page
Implement Sci. 2011; 6: 49.
Published online May 23, 2011. doi:  10.1186/1748-5908-6-49
PMCID: PMC3120719

A realist evaluation of the role of communities of practice in changing healthcare practice

Abstract

Background

Healthcare organisations seeking to manage knowledge and improve organisational performance are increasingly investing in communities of practice (CoPs). Such investments are being made in the absence of empirical evidence demonstrating the impact of CoPs in improving the delivery of healthcare. A realist evaluation is proposed to address this knowledge gap. Underpinned by the principle that outcomes are determined by the context in which an intervention is implemented, a realist evaluation is well suited to understand the role of CoPs in improving healthcare practice. By applying a realist approach, this study will explore the following questions: What outcomes do CoPs achieve in healthcare? Do these outcomes translate into improved practice in healthcare? What are the contexts and mechanisms by which CoPs improve healthcare?

Methods

The realist evaluation will be conducted by developing, testing, and refining theories on how, why, and when CoPs improve healthcare practice. When collecting data, context will be defined as the setting in which the CoP operates; mechanisms will be the factors and resources that the community offers to influence a change in behaviour or action; and outcomes will be defined as a change in behaviour or work practice that occurs as a result of accessing resources provided by the CoP.

Discussion

Realist evaluation is being used increasingly to study social interventions where context plays an important role in determining outcomes. This study further enhances the value of realist evaluations by incorporating a social network analysis component to quantify the structural context associated with CoPs. By identifying key mechanisms and contexts that optimise the effectiveness of CoPs, this study will contribute to creating a framework that will guide future establishment and evaluation of CoPs in healthcare.

Background

With a focus on knowledge sharing and learning, communities of practice (CoPs) are being promoted in the healthcare sector as a means of improving practice and patient care [1]. By definition, a CoP is a group of people 'who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise on this area by interacting on an ongoing basis' [2]. It is argued that CoPs nurture and harness knowledge, particularly in terms of promoting the exchange of tacit knowledge, and drive innovation to help individuals and organisations improve practice and performance [3-5]. Such claims have contributed to the widespread adoption of CoPs in healthcare and other sectors seeking to effectively manage knowledge in order to improve organisational performance.

Reflecting the increased uptake of CoPs in the healthcare sector, the number of peer-reviewed papers reporting on CoPs is steadily increasing [6], and includes the publication of two systematic reviews in 2009 [7,8]. Despite this increase, there is a lack of empirical evidence demonstrating the impact of CoPs in improving healthcare practice. Much of the published literature is limited to describing the establishment or activities of CoPs [6,9]. If organisations and sponsors are to foster CoPs for their value in knowledge management and for improving organisational performance, there is a need to understand better the role of CoPs in improving healthcare practice. To this end, a realist evaluation of CoPs is proposed.

In contrast to traditional evaluation methods that examine the success of an intervention based on whether or not a predefined outcome has been achieved, the realist approach seeks to answer the questions -- how, why, and when does the intervention work [10]? A realist evaluation is a theory-driven approach to understanding what it is about a program that achieves a particular outcome in one setting and a different outcome in another. It is well suited for social interventions where outcomes are determined through stakeholder action and interaction, which in turn is likely to be influenced by social and cultural norms [10,11]. Underpinned by the principle that context (C) will trigger mechanisms (M) to yield outcomes (O), a realist evaluation goes beyond focussing purely on inputs and outputs; it involves exploring and identifying the mechanisms by which the inputs are converted inside a 'black box' into outputs, and recognises the need for particular conditions (or contexts) to be present inside the black box for the causal mechanisms to be triggered and yield a particular outcome. The relationship between context, mechanism, and outcome is presented as a 'CMO configuration' [10]. Based on these principles, the objective of this study is to identify CMO configurations that will explain the role of CoPs in improving healthcare practice. This objective will be achieved by seeking to answer the following questions: What outcomes do CoPs achieve in the healthcare sector? Do these outcomes translate into improved practice in healthcare? What are the contexts and mechanisms by which CoPs impact on improved practice in healthcare?

Methods

This realist evaluation will be conducted in four stages as shown in Table Table11 corresponding to the four components of the realist evaluation cycle (theory, hypotheses generation, observations, and program specifications) as described by Pawson and Tilley [10]. The stages will be undertaken sequentially such that the findings from each stage will inform the next stage, and the final stage will involve reviewing the findings from stage three to confirm, modify, or reject the theory-based hypotheses generated in stage two [12].

Table 1
Four-stage approach to the realist evaluation of communities of practice

An opportunistic sample of four CoPs will be identified, and sponsors, facilitators, and members of each CoP will be interviewed and surveyed using semi-structured interviews and online surveys. These four CoPs will act as case studies to enable the in-depth exploration required to understand the happenings within the 'black box' linking CoPs to improved practice.

Stage one: theory

The first stage of the realist evaluation involves developing candidate theories on the role of CoPs in improving healthcare practice and potential CMO configurations. This begins with a systematic search and review of the healthcare literature identifying characteristics of CoPs and outcomes achieved. By focusing the review specifically on the healthcare sector, the information collected will be context specific. Our systematic review [6] identified particular features by which CoPs in the healthcare sector differ (Table (Table2),2), providing us with a starting point for formulating contexts and potential mechanisms by which CoPs in the healthcare sector influence change in practice.

Table 2
Characteristics of communities of practice identified from the literature [6]

The next activity in stage one is to interview sponsors and facilitators with the objective of identifying CMOs that will, in turn, be used to generate candidate theories on CMO configurations explaining the role of CoPs in improving healthcare practice. Additional data will be collected on resources offered to members and the means by which impact is assessed. The questions used to guide these interviews are in Additional File 1.

When collecting data from published literature and from interviews, contexts will be defined as the settings in which the CoP operates. To a large extent, these will be the characteristics of the CoP in terms of constitution of membership, level of maturity of the CoP, and activities organised by the CoP. Contexts will be also be determined by examining the connections, interactions, and knowledge flow that occur within each CoP. Mechanisms will be defined as the factors and resources that the CoP offers its members to influence a change in behaviour or action [12,13]. A mechanism may be an enabler or a disabler depending on the context.

To date, there is no consistency in the way in which outcomes of CoPs are defined or measured [6]. For the purpose of this study, an outcome will be defined as a change in behaviour or work practice that occurred, influenced by participating in a CoP activity or through accessing resources provided by the CoP. The change may be to a process (such as adoption of a new system or process, or reduced time to achieve a goal that is related to improved care); an innovation (such as development of a new product or technology that will improve the delivery of healthcare); or change in level of customer (patient) satisfaction [14]. Financial outcomes will not be considered due to the focus of this study on clinical practice. Individual as well as organisational level outcomes will be sought, recognising that improved organisational performance is achieved through changing the work practice of individuals who contribute to the organisation.

The end product of stage one will be a list of CMOs and possible CMO configurations that explain the role of CoPs in improving healthcare. Figure Figure11 presents a preliminary list based on background research.

Figure 1
Preliminary list of CMOs that potentially play a role in CoPs improving healthcare practice.

Stage two: Hypotheses generation

The second (hypotheses generating) stage involves rephrasing the CMO configurations theorised in stage one as hypotheses for testing in stage three. These hypotheses will be framed around the theoretical relationships between specific CMOs that could explain the different outcomes of CoPs, depending on the context in which the CoP operates.

Stage three: Observation

The hypotheses developed in stage two will be tested during stage three. All members of the participating CoPs will be asked to respond to a survey that will seek their level of agreement (using a five-point Likert scale) with hypotheses (see Additional File 2). Testing these hypotheses will help identify CMO configurations that occur with regularity, and provide possible explanations for the role of CoPs in improving healthcare practice.

The second part of the third stage will involve obtaining contextual information on the connections, relationships, and knowledge exchange that occur within a CoP. These are, by definition, essential elements of a CoP and form the context in which the hypotheses are tested. As identified in the literature and presented in Table Table22 regular communication, interaction, and knowledge exchange between members are characteristics associated with CoPs. These elements have been linked to improved organisational performance through the concept of social capital as follows: Social capital is created by developing connections among practitioners that foster 'relationships that build a sense of trust and mutual obligation, and (by) creating a common language and contexts that can be shared by community members' [4]. The connections, relationships, and common contexts that generate social capital, in turn, positively impact on organisational performance [4]. The study will utilise social network analysis (SNA) methods to examine the professional connections and relationships within the CoP, represented by the strength of ties, so as to understand how the CoP network features may relate to improved healthcare practice. The social interactions that occur within the CoP will also be examined, recognising their role in the exchange of tacit knowledge [15].

Questions used to collect data on professional and social connections and knowledge exchange will be based on social network questions used by other researchers [16-18], modified where necessary to achieve the objectives of this study (See Additional File 3). The questionnaire will be validated by testing on a convenience sample of ten people with clinical practice and health management roles similar to those of members of the CoPs. Reliability will be tested using Kappa coefficient methods on test and retest of the instrument five days apart.

The network data collected in the study will be analysed using UCInet [19]. The NetDraw feature of this software allows visual examination of each of the relationships (i.e., professional connections, social interactions, and information and knowledge flow) for strength of connectedness based on the frequency of contact. It also aids the identification of cliques (or subgroups), cut-points (referring to a person whose departure will result in a break in flow of information/knowledge), and isolated individuals [20].

When surveying CoP members, knowledge will be defined as 'internali(s)ed or understood information that can be used to make decisions' [21]. Knowledge will be differentiated from information by the fact that '(k)nowledge is information possessed in the mind of individuals: it is personali(s)ed information (which may or may not be new, unique, useful, or accurate) related to facts, procedures, concepts, interpretations, ideas, observations, and judgements' [22]. The reason for differentiating knowledge from information is to examine how much of the information that CoPs provide their members is translated into knowledge that influences change in their own work and practice.

The analysis functions in UCInet will be used to quantify the connectivity and stability of the community by measuring degree, closeness and betweenness centrality, reciprocity of relationships, and multiplexity [23]. Degree centrality is the number of persons (or nodes) to which a particular person is directly linked; a higher score indicates a well connected person [24]. This measure will help identify key persons in the community, with the facilitator likely to score highly. A high average density score at the CoP level indicates a high level of direct links or interactions between members of the CoP. Closeness centrality recognises the importance of indirect connections for exchange of resources (such as knowledge) and measures the shortest path connecting a key node (CoP member, in this case) to any other node [25]. Betweenness centrality also takes into account the importance of indirect links in maintaining links between nodes not otherwise connected [25]. This, too, is relevant in terms of examining the flow of resources (such as information or knowledge) [24]. A CoP scoring highly in the knowledge exchange relationship would indicate high connectivity with little threat to knowledge exchange due to lost links. Reciprocity of each relationship will also be examined to identify bidirectional links, with suggestions that high level of reciprocity is characteristic of a more stable network [26]. The knowledge relationship will be examined further for path length, to assess the efficiency of information and knowledge flow and exchange within the CoP and will help identify how best to optimise this process [27].

As this study examines multiple relationships (that is, professional connections, social interactions, and information and knowledge flow), multiplexity will be examined as an indication of the strength of the link between members; with members linked by more than one relationship said to have stronger ties than those linked by one relationship [24,28,29].

Stage four: Program specification

The fourth and final stage is program specification, during which the theorised role of CoPs in improving healthcare and potential CMO configurations from stage one will be reviewed in light of the findings in stage three. The CMO configurations that were supported with regularity will form the basis for specifying possible explanations for the role of CoPs in improving healthcare practice.

Discussion

This paper describes a protocol that uses mixed methods to examine systematically and understand how, why, and when CoPs improve healthcare practice. Realist evaluation is being used increasingly in the healthcare sector, recognising the fact that programs and interventions requiring behavioural change operate within a complex social and cultural context, and that the operating context plays an important role in determining impact. In such circumstances, the traditional approach of evaluating success based on whether or not a pre-defined outcome is achieved does not provide decision makers with sufficient information to assess the value of the program outside the context in which it was tested. There is a need for methods that are able to tease out the mechanisms by which a program results in change, and study the interactions between these causal mechanisms and context [10,30].

Following the application of a realist approach to evaluate a modernisation initiative in the UK, Greenhalgh et al. discussed the difficulties in identifying the mechanisms of change and drawing realist conclusions around CMO configurations. They refer to this process as typically requiring 'a three-hour face-to-face meeting as well as lengthy email exchanges and numerous iterations and counteriterations' [31]. Work undertaken to date on this project affirms the difficulty of identifying mechanisms and outcomes, and generating the list presented in this paper has required lengthy discussions and iterations. Since our proposal also includes using SNA methods to examine the connections and knowledge exchanges within the CoP as a means of providing contextual information, we need to strike a fine balance between making significant demands on participants' time and securing the high response rates required for SNA. As a means of achieving this balance, we have chosen to limit the in-depth interviews and discussions to facilitators and sponsors of CoPs. Members will participate in testing the hypotheses generated by the discussions and will respond to the SNA questions. To help this process further and taking into consideration the length of the hypotheses testing survey, the SNA survey will be administered at a later date.

A challenging aspect of developing this study protocol was identifying and defining an outcome that would demonstrate the impact of CoPs in improving work practice. A finding from our systematic review [6] was that the vast majority of existing research had assessed impact though self-reported perceived benefits, with very limited effort to substantiate these claims through triangulation. This study will attempt to overcome this limitation by defining an outcome as a demonstrated change in work practice at the individual member level as well as the organisation level. The difficulty in drawing conclusions around CMO configurations will be addressed to some extent by looking for patterns that occur with regularity supporting the occurrence of such causal interactions.

The realist evaluation method, by seeking to understand how, why, and when a program works, is well suited for separating out and examining the multiple components in a program individually and in the context of the program. This feature is particularly useful given the difficulty experienced in directly attributing outcome to a CoP in studies that have measured and reported outcomes from multi-faceted interventions [6]. CoPs offer more than one resource to their members with the intention of facilitating knowledge creation and sharing. Knowing the role that each of these components play in influencing change in healthcare practice will help maximise value and return on investment in CoPs.

This study further enhances the value of realist evaluations by incorporating a SNA component to quantify the structural context associated with CoPs. To our knowledge, these two methods have not been previously combined. By examining the connections and relationships that occur within the community or network, SNA methods quantify the structural component of the context within which CoPs operate.

Overall, this paper proposes a research study to understand the complexity of CoPs, taking into consideration its multi-component nature and the influence of context in determining impact. The systematic approach proposed will help identify key mechanisms that operate within particular contexts, which in turn will help optimise the establishment and effectiveness of CoPs. The study will contribute to creating a framework that will guide the future development and evaluation of CoPs in the healthcare sector [9].

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

JB and JIW conceptualised the overarching research project and are the chief investigators of the research grant funding this research activity. GR developed the study protocol presented in this paper in consultation will all other co-authors and wrote the first draft. All authors provided input into various aspects of the study, provided ongoing critique and approved the final version of the manuscript.

Supplementary Material

Additional file 1:

Questions guiding interviews with sponsors and facilitators - for stage 1 of the realistic evaluation.

Additional file 2:

Survey of CoP members to test context, mechanism and outcome configurations.

Additional file 3:

Survey of CoP members to map the current structure, available expertise and knowledge exchange.

Acknowledgements and funding

This research is supported by the Australian Research Council's Discovery Project funding scheme (DP 0986493), and has been approved by the Human Research Ethics Committee, University of New South Wales (HREC 09085). The authors would like to acknowledge and thank Sue Huckson and Scott Bennett for their input into identifying CMOs as they relate to CoPs; and David Greenfield and Nerida Creswick for their valuable input at various stages of this study.

References

  • le May A. In: Communities of practice in health and social care. le May A, editor. Wiley-Blackwell; 2009. Introducing communities of practice; pp. 3–16.
  • Wenger E, McDermott R, Snyder WM. Cultivating Communities of Practice. Boston, Massachusetts: Harvard Business School Press; 2002.
  • American Productivity & Quality Centre. Building and sustaining communities of practice: Continuing success in knowledge management. Texas: American Productivity and Quality Centre (APQC); 2001.
  • Lesser EL, Storck J. Communities of practice and organizational performance. IBM Syst J. 2001;40:831–841.
  • Schenkel A, Teigland R. Improved organizational performance through communities of practice. J knowledge management. 2008;12:106–118. doi: 10.1108/13673270810852421. [Cross Ref]
  • Ranmuthugala G, Plumb J, Cunningham F, Georgiou A, Westbrook J, Braithwaite J. Communities of practice in the health sector: A systematic review of the peer-reviewed literature. Sydney: University of New South Wales, Australian Institute of Health Innovation; 2010.
  • Li LC, Grimshaw JM, Nielsen C, Judd M, Coyte PC, Graham ID. Use of communities of practice in business and health care sectors: A systematic review. Implement Sci. 2009;4 [PMC free article] [PubMed]
  • Fung-Kee-Fung M, Watters J, Crossley C, Goubanova E, Abdulla A, Stern H, Oliver TK. Regional collaborations as a tool for quality improvements in surgery: a systematic review of the literature. Ann Surg. 2009;249:565–572. doi: 10.1097/SLA.0b013e31819ec608. [PubMed] [Cross Ref]
  • Braithwaite J, Westbrook JI, Ranmuthugala G, Cunningham F, Plumb J, Wiley J, Ball D, Huckson S, Hughes C, Johnston B. et al. The development, design, testing, refinement, simulation and application of an evaluation framework for communities of practice and social-professional networks. BMC Health Serv Res. 2009;9:162. doi: 10.1186/1472-6963-9-162. [PMC free article] [PubMed] [Cross Ref]
  • Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.
  • Hunt KS, Sridharan S. A Realist evaluation approach to unpacking the impacts of the sentencing guidelines. Am J Eval. 2010;31:463–485. doi: 10.1177/1098214010371817. [Cross Ref]
  • Thistleton L. A realistic evaluation of the work of a speech and language therapy service in primary schools (the first schools project) using the perceptions of some of the important stakeholders (teachers, SLTs, and parents) PhD thesis. The University of Birmingham, School of Education; 2008.
  • Kazi M. In: Realist evaluation in practice: Health and social work. Kazi M, editor. London, California, New Delhi: Sage Publications Ltd; 2003. A realist evaluation of practice: the NSPCC's Shield project and intensive analysis; pp. 95–109.
  • Rhodes J, Lok P. An integrative model of organizational learning and social capital on effective knowledge transfer and perceived organizational performance. J Workplace Learning. 2008;20:245–258. doi: 10.1108/13665620810871105. [Cross Ref]
  • Helms R, Bosua R, Ignatio R. Impact assessment of knowledge sharing bottlenecks:The Knowledge Sharing Environment Model (KSEM) Presented at the 20th Australasian Conference on Information Systems, Melbourne, Australia; 2009.
  • Cross R, Parker A. The hidden power of social networks. Boston, Massachusetts, USA: Harvard Business School Publishing Corporation; 2004.
  • Sales AE, Estabrooks CA, Valente TW. The impact of social networks on knowledge transfer in long-term care facilities: Protocol for a study. Implement Sci. 2010;5:49. doi: 10.1186/1748-5908-5-49. [PMC free article] [PubMed] [Cross Ref]
  • Ferlie E, Fitzgerald L, McGivern G, Dopson S, Exworthy M. Networks in health care: A comparative study of their management, impact and performance. London: King's College London; 2010. http://www.sdo.nihr.ac.uk/files/project/102-final-report.pdf Accessed 13 March 2011.
  • Borgatti S, Everett M, Freeman L. UCInet 6.3 For Windows: Software for social network analysis. Lexington, KY; 2010.
  • Helms R. In: Hands-On Knowledge Co-Creation and Sharing: Practical Methods and Techniques. Kazi A, Wohlfart L, Wolf P. Stuttgart, editor. Germany: KnowledgeBoard; 2007. Redesigning communities of practice using knowledge network analysis; pp. 251–274.
  • Zins C. Conceptual approaches for defining data, information, and knowledge. J Am Soc Inf Sci Technol. 2007;58:479–493. doi: 10.1002/asi.20508. [Cross Ref]
  • Alavi M, Leidner DE. Review: Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly. 2001;25:107–136. doi: 10.2307/3250961. [Cross Ref]
  • Creswick N, Westbrook JI. Social network analysis of medication advice-seeking interactions among staff in an Australian hospital. Int J Med Inform. 2010;79:e116–125. doi: 10.1016/j.ijmedinf.2008.08.005. Epub 2008 Nov 2012. [PubMed] [Cross Ref]
  • Provan KG, Nakama L, Veazie MA, Teufel-Shone NI, Huddleston C. Building community capacity around chronic disease services through a collaborative interorganizational network. Health Educ Behav. 2003;30:646–662. doi: 10.1177/1090198103255366. [PubMed] [Cross Ref]
  • Provan KG, Fish A, Sydow J. Interorganizational networks at the network level: A review of the empirical literature on whole networks. J Management. 2007;33:479–516.
  • Hanneman R, Riddle M. Introduction to social network methods. Riverside, CA: University of California; 2005. Chapter 8: Embedding.
  • Grippa A. A social network scorecard to monitor knowledge flow across communication media. Knowledge management and practice. 2009;7:317–328. doi: 10.1057/kmrp.2009.24. [Cross Ref]
  • Holschuh J, Segal S. Factors related to multiplexity in support networks of persons with severe mental illness. Social Networks and Health. 2002;8:293–321.
  • Provan KG, Milward HB. Do networks really work? A framework for evaluating public-sector organizational networks. Public Admin Review. 2001;61:414–423. doi: 10.1111/0033-3352.00045. [Cross Ref]
  • Berwick DM. The science of improvement. J Amer Med Assoc. 2008;299:1182–1184. doi: 10.1001/jama.299.10.1182. [PubMed] [Cross Ref]
  • Greenhalgh T, Humphrey C, Hughes J, Macfarlane F, Butler C, Pawson R. How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Q. 2009;87:391–416. doi: 10.1111/j.1468-0009.2009.00562.x. [PMC free article] [PubMed] [Cross Ref]
  • Hildreth PM, Kimble C. The duality of knowledge. Information Research. 2002;8
  • Lathlean J, le May A. Communities of practice: an opportunity for interagency working. J Clin Nurs. 2002;11:394–398. doi: 10.1046/j.1365-2702.2002.00630.x. [PubMed] [Cross Ref]
  • Kimball L, Ladd A. Facilitator toolkit for building and sustaining virtual communities of practice. Idea Group Publishing. 2004.
  • Render ML, Brungs S, Kotagal U, Nicholson M, Burns P, Ellis D, Clifton M, Fardo R, Scott M, Hirschhorn L. Evidence-based practice to reduce central line infections. Joint Comm J Qual Patient Saf. 2006;32:253–260. [PubMed]
  • Chandler L, Fry A. Can communities of practice make a meaningful contribution to sustainable service improvement in health and social care? J Integr Care. 2009;17:41–48.
  • Jiwa M, Deas K, Ross J, Shaw T, Wilcox H, Spilsbury K. An inclusive approach to raising standards in general practice: working with a 'community of practice' in Western Australia. BMC Med Res Methodol. 2009;9:13. doi: 10.1186/1471-2288-9-13. [PMC free article] [PubMed] [Cross Ref]

Articles from Implementation Science : IS are provided here courtesy of BioMed Central
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...