• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of milbankLink to Publisher's site
Milbank Q. Jun 2003; 81(2): 221–248.
PMCID: PMC2690219

How Can Research Organizations More Effectively Transfer Research Knowledge to Decision Makers?

John N Lavis, Dave Robertson, Jennifer M Woodside, Christopher B McLeod, and Julia Abelson
McMaster University; Institute for Work & Health; Canadian Institute for Advanced Research; Queen's University
Knowledge Transfer Study Group

Applied research organizations invest a great deal of time, and research funders invest a great deal of money generating and (one hopes) transferring research knowledge that could inform decisions about health and health care. Basing these knowledge-transfer activities on our evolving understanding of the most effective approaches to knowledge transfer will help us achieve value for money in our individual and collective investments in health services and health policy research. Research organizations and research funders can probably be excused for not basing their activities on research evidence until now, however, because the variety of relevant questions, target audiences, and disciplinary perspectives and methodological approaches used in empirical studies has made the identification of take-home messages from this field of research a very difficult task.

We provide an organizing framework for a knowledge-transfer strategy and an overview of our understanding of the current knowledge for each of the five elements of the framework. The framework provides an overall approach to knowledge transfer that can be evaluated as a whole over long periods of time, as well as specific elements that can be evaluated and fine-tuned over shorter periods of time. We also illustrate how opportunities for improving how research organizations transfer research knowledge can be found in the differences between what the research literature suggests that research organizations should do and what the directors of research organizations say that they do. We surveyed the directors of applied health and economic/social research organizations in Canada regarding how their organizations transfer research knowledge to decision makers. We conclude by providing suggestions for both action and further research.

A Framework for Knowledge Transfer

Five questions provide an organizing framework for a knowledge-transfer strategy: What should be transferred to decision makers (the message)? To whom should research knowledge be transferred (the target audience)? By whom should research knowledge be transferred (the messenger)? How should research knowledge be transferred (the knowledge-transfer processes and supporting communications infrastructure)? With what effect should research knowledge be transferred (evaluation)? The details of these elements vary according to the target audience. We distinguish among four audiences for applied health and economic/social research: general public/service recipients (e.g., citizens, patients, and clients), service providers (e.g., clinicians), managerial decision makers (e.g., managers in hospitals, community organizations, and private businesses), and policy decision makers at the federal, state/provincial, and local levels (Goldberg et al. 1994; Lomas 1990; Power and Eisenberg 1998).

We conducted a qualitative review of both systematic reviews and original studies across the five questions, four target audiences, and full range of disciplinary perspectives and methodological approaches used in empirical studies. We identified relevant articles and documents from the lead author's collection, as well as from an ongoing, systematic review by several authors. We consider our answers to the five questions to be preliminary because each combination of question, target audience, and disciplinary perspective and/or methodological approach could (and ultimately should) be subjected to a systematic review. But we hope that by asking the salient questions, placing them in a logical order, and providing preliminary answers to them, we will spur such systematic reviews. The answer to the questions may change as these results become available. We recognize, however, that the challenges of evaluating the effectiveness of the different approaches may mean that we can never convincingly address some questions, such as the optimal allocation of resources to knowledge transfer within research organizations.

What Should Be Transferred to Decision Makers?

The research literature strongly suggests that research organizations should transfer actionable messages from a body of research knowledge, not simply a single research report or the results of a single study. A message can, however, profile and place in context a particular study when relevant. Enhanced validity provides one justification for this approach. Years of research on systematic reviews have taught us that individual studies can often lead to a conclusion very different from that of a systematic review of all available studies (Egger and Davey Smith 1997). For example, a single study comparing the mortality rates of for-profit and not-for-profit hospitals may have found a lower risk of death for patients treated in for-profit hospitals, but a metanalysis of more than 26,000 hospitals and 38 million patients found a higher risk of death for patients treated in for-profit hospitals (Devereaux et al. 2002).

Empirical research on the “types” of research that influence decision making also justifies this approach. Research on managerial and policy decision making has taught us that research in the form of “ideas,” not “data,” most influences decision making (Weiss 1991). Decision makers rarely use a regression coefficient to help them solve a particular problem. Rather, over long periods of time, “ideas” enlighten decision makers about a particular issue and how to handle it. For example, it took decades for nurse practitioners to be seen as a viable policy alternative to solve the particular problem of physician shortages in rural and remote areas, and even then most decision makers were probably not aware of the particular studies that demonstrated the safety and cost effectiveness of nurse practitioners (Spitzer 1984).

We offer two caveats to this approach. First, not all research can or should have an impact. Some bodies of research knowledge will not generate a “take-home” message, because either the research has no apparent application for decision makers or the findings are not conclusive. That said, this excuse can be overused. A recent survey of prominent health services researchers identified that their last decade of work had been spent primarily on identifying problems, generating hypotheses, and developing new methodologies, not informing decision making (Stryer et al. 2000). Someone needs to tell decision makers about solutions. Second, accountability mechanisms must be in place to ensure that when take-home messages can be generated, they are appropriate to the decision-making environments to which they are directed (Black 2001b). Research ethics boards, for example, may need to begin assessing the back end of the research, not just the front end (i.e., the proposal). Even then, take-home messages should be seen as only a starting point for a discussion with decision makers, given that researchers will rarely be able to predict the range of incentives and constraints that various decision makers face and what this context means for the applicability of their message (Shapiro 1993).

To Whom Should Research Knowledge Be Transferred?

The research literature makes clear that a message's target audiences must be clearly identified and the specifics of a knowledge-transfer strategy must be fine-tuned to the types of decisions they face and the types of decision-making environments in which they live or work. The same message from a body of research knowledge about stroke care, for example, would clearly not work for patients, clinicians (paramedics, nurses, doctors, and rehabilitation specialists), managers, and public policymakers. Rather, multiple audience-specific messages are needed. Moreover, the same approach to knowledge transfer would clearly not work for patients having to make a decision about cancer treatment that could be informed by a consumer decision aid (Estabrooks et al. 2001), for clinicians having to make a decision about a diagnosis or treatment that could be informed by a clinical practice guideline (Lomas et al. 1989), for managers having to make a decision about resource allocation that could be informed by economic evaluations (Drummond, Cooke, and Walley 1997; Hoffmann and von der Schulenburg 2000), for public policymakers in the health sector having to make a decision about regulations that could be informed by health services research (Eisenberg 1998), and for public policymakers in finance departments having to make a decision about a tax-and-transfer policy that could be informed by research on nonmedical determinants of health (Lavis, Ross, Stoddart et al. 2003).

Learning about these decision-making environments often requires a significant investment of time and financial resources. As one moves from citizen, patient, and clinical (i.e., individual) decision-making environments to managerial and policy decision-making environments, for example, the organizational and political factors with which research knowledge must compete to influence the decision-making process become more apparent (Black 2001a; Walshe and Rundall 2001). We have learned that public policymakers must contend with not only research knowledge but also the values and opinions of the governing party, its key supporters, interested and affected stakeholders, and the general public; with calculations of who wins, who loses, and by how much (i.e., what political scientists call “interests”); and with rules for making decisions and with past policies that may shape and constrain future policies (i.e., what political scientists call “institutions”) (Lavis et al. 2002).

The research literature does not explain how to select the target audience(s) for a message, only that once a target audience is identified, the specific knowledge-transfer strategy should be fine-tuned to the types of decisions the decision makers face and the decision-making environments in which they live or work. But what if knowledge transfer to patients is better at creating change in clinical practice than targeting clinicians, hospital managers, public policymakers, or some combination of them? When deciding to whom the research knowledge should be transferred, the first step should be to ask who can act on the basis of the available research knowledge; the second step should be to ask who can influence those who can act; and the third step should be to ask with which of these target audience(s) we can expect to have the most success and which messages pertain most directly to each of them.

By Whom Should Research Knowledge Be Transferred?

According to the research literature, the credibility of the messenger delivering the message—whether the messenger is an individual, group, or organization—is important to successful knowledge-transfer interventions but has never been tested (Shonkoff 2000). The opinion leaders who have been used in clinical decision-making environments, for example, are, by definition, credible messengers. Presumably credibility pertains to both the research/academic arena and the target audience, and the likelihood of credibility not mattering is so low as to make it unlikely that anyone will ever evaluate its effect directly. We do, though, have indirect evidence of its effect. An authoritative endorsement by a respected physician organization or a respected physician colleague, for example, has been shown to influence physicians’ adoption of clinical practice guidelines (Hayward et al. 1997). We also have some evidence regarding who is perceived to be credible by different target audiences. Public policymakers, for example, report that they consider organizations of government professionals, such as the Association of State and Territorial Health Officers, to be trusted sources of information (Sorian and Baugh 2002).

Building credibility and acting as a messenger can be very time-consuming and skill-intensive processes, which makes it impossible to use a one-size-fits-all approach to decide who should act as the messenger. When researchers have the skills and experience to act as the principal messenger, their credibility will likely make them the ideal choice. Having researchers work with and through trusted intermediaries (i.e., knowledge brokers) may constitute a way around the time constraints faced by individual researchers and the limited interest in and skills applicable to knowledge transfer of some researchers while at the same time enhancing the messenger's credibility.

How Should Research Knowledge Be Transferred?

The research literature on which processes are best at transferring research knowledge suggests that passive processes are ineffective and that interactive engagement may be most effective, regardless of the audience. For instance, research on the transfer of research knowledge to and its uptake by clinical audiences has demonstrated that interventions like academic detailing and opinion leaders appear to be effective in many settings (Davis et al. 1995; Davis et al. 1999; Grimshaw et al. 2001; Oxman et al. 1995). The hallmark of these interventions is interaction: interaction between the clinician and an “expert” who has been trained in the principles of academic detailing or interaction between the clinician and someone to whom he or she routinely turns for guidance (Lomas et al. 1991; Soumerai and Avorn 1990).

Research on the transfer of research knowledge to and its uptake by managerial and policy audiences has demonstrated that interaction between researchers and these audiences (or representative members of these audiences) appears to be important to explaining why some types of research knowledge are used and not others, albeit using observational and not randomized, controlled trial designs (Innvaer et al. 2002; Landry, Amara, and Lamari 2001; Lavis et al. 2002; Lomas 2000; Soumerai et al. 1997). Interaction can occur at many stages in both the research process and the decision-making process. In theory, interaction can introduce bias into research studies if decision makers press researchers to conduct research in ways that are likely to yield results favorable to decision makers’ preexisting beliefs or positions (Innvaer et al. 2002). Whether such outcomes occur frequently has not yet been studied systematically.

The research literature also goes beyond researchers transferring research knowledge using one-way (and sometimes one-off) processes (i.e., beyond “producer-push” efforts). Over long periods of time, two-way “exchange” processes that give equal importance to what researchers can learn from decision makers and what decision makers can learn from researchers can produce cultural shifts. A decision-relevant culture can be created among researchers, and a research-attuned culture can be created among decision makers (Huberman 1994; Roos and Shapiro 1999). Such cultural shifts can facilitate the ongoing use of research knowledge in decision making, not just one-off uses. In addition, the research literature supports the effectiveness of at least one noninteractive intervention, namely, individualized feedback (Davis et al. 1995; Grimshaw et al. 2001; Oxman et al. 1995), which we used at the end of this study to help research organizations move toward the best practices in knowledge transfer. Supporting infrastructure like Web sites and newsletters can augment interactive efforts, though not replace them, particularly if the material provides targeted information to clearly identified audiences and/or more general information in a searchable form when an intervention or event generates a demand for this information.

With What Effect Should Research Knowledge Be Transferred?

Performance measures for knowledge transfer should be appropriate to the target audience and to the objectives (Lavis, Ross, McLeod et al. 2003). For clinicians, for example, the objective may be to change behavior so it is more in line with the available evidence. For public policymakers, given the organizational and political factors with which research knowledge must compete, the objective may be to inform debate. Measures can be categorized according to whether they capture a process associated with the pursuit of the research's impact (e.g., presentations to decision makers), an intermediate outcome (e.g., a change in awareness, knowledge, or attitudes), or an outcome (e.g., a decision to select one course of action over others because research knowledge supports its effectiveness).

When measuring outcomes, moving beyond measuring whether research knowledge is used in decision making to measuring how research knowledge is used becomes important (Pelz 1978; Weiss 1979). Research knowledge may be used in instrumental, conceptual, or symbolic ways. Instrumental use is defined as acting on research in specific and direct ways, such as to solve a particular problem at hand (e.g., developing the first iteration of Medicare's “Resource-Based Relative Value Scale” physician fee schedule). Conceptual use involves a more general and indirect form of enlightenment (e.g., resisting a move toward more for-profit hospitals because of a general sense that not-for-profit hospitals offer a survival advantage for patients compared with for-profit hospitals, but without knowing about the particular studies or their strengths and limitations). Conversely, symbolic use pertains to a use of research knowledge, but not to inform decision making; here research knowledge is used to justify a position or action that has already been taken for other reasons (sometimes called a “political use of research”) or the fact that research is being done is used to justify inaction on other fronts (called a “tactical use of research”).

Moving beyond decision-making outcomes to health, economic, and social outcomes, however, is almost certainly asking too much. Research organizations simply want to know whether the research knowledge that they produce is having an impact on decision making. Tracing the complex pathways through which informed decisions translate into improved implementation or performance and ultimately into better health is best left to stand-alone research initiatives (Lavis 2002). The same can be said of economic and social outcomes.

Current Practices in Transferring Research Knowledge

Opportunities for improving how research organizations transfer research knowledge can be found in the differences between what the research literature suggests that research organizations should do and what the directors of research organizations say that they do. We surveyed the directors of applied health and economic/social research organizations in Canada regarding how their organizations transfer research knowledge to decision makers. We had two research objectives: (1) to assess the extent to which Canadian research organizations were transferring research knowledge in ways consistent with our understanding of the research evidence, and (2) to examine whether either sector (i.e., health versus the economic/social policy sector) or target-audience orientation (i.e., none, some, or all four target audiences) explained any variation in their responses. We did not collect data that would have allowed us to establish which individual-level factors (e.g., directors’ personal interest in knowledge transfer), organization-level factors (e.g., size of budget and affiliation with a university) or system-level factors (e.g., availability of funding for syntheses and for knowledge transfer) explained any variation in their responses, although we do plan to pursue this line of inquiry in future research.

We hypothesized that most Canadian research organizations were not transferring research knowledge in ways consistent with the available evidence (primarily because take-home messages from the field of research on knowledge transfer have been so difficult to identify), that health organizations were more likely than economic/social research organizations to transfer research knowledge in ways consistent with the available research evidence (primarily because leadership in this domain in the health sector was strong and the legislative mandate of Canada's largest health research funder focused attention on it), and that research organizations with a smaller number of target audiences were more likely to transfer research knowledge in ways consistent with the available research evidence (primarily because specialization would increase the likelihood that they could identify take-home messages from the subfield of research on knowledge transfer pertaining to their principal target audience).

The two past attempts to describe current practices in knowledge transfer in Canadian health research organizations were not designed to provide comprehensive descriptions (Canadian Population Health Initiative 2001; Lewis 2000). These surveys were less comprehensive than ours in the number of research organizations surveyed (17 in one case and 12 in the other), in the type of research organizations surveyed (committed to knowledge transfer to public policymakers in one case and committed to health technology assessment in the other), and in the number and type of questions asked about knowledge-transfer activities (to whom, when, and how were asked in one case, and how and with what effect were asked in the other). Differences in either the survey questions or the rating scale or both precluded comparing responses across surveys to determine whether the types of knowledge-transfer activities used by Canadian research organizations have been changing over time.

Methods

We mailed a survey to 265 directors of applied research organizations in Canada about their organizations’ current practices in transferring research knowledge to decision makers. Our study population covered two groups: applied health research organizations (N = 134) and applied economic/social research organizations (N = 131). We defined applied research organizations as research groups producing research that could be acted on by any one of four target audiences: general public/service recipients, service providers, managerial decision makers, and policy decision makers. This definition ruled out laboratory-based research groups producing research intended primarily for biomedical firms. We defined applied health research organizations as research groups studying the effectiveness and efficiency of clinical services and health care systems. We defined economic/social research organizations as research groups studying the effectiveness and efficiency of government economic/social programs and economic/social policy systems.

We included autonomous research groups, semiautonomous research groups in university departments and schools, semiautonomous research groups in federal or provincial government departments, and semiautonomous research groups in Quebec's largest regional health authorities. We excluded university departments or schools, virtual networks of researchers (e.g., National Centres of Excellence), management-consulting firms, marketing-research firms, and professional membership organizations. We also excluded research groups that had existed for less than one year.

We identified research organizations by reviewing published lists of Canadian research organizations (Abelson and Carberry 1998), reviewing unpublished lists of Canadian research organizations compiled by groups like the federal government's Policy Research Initiative, searching Canadian funding agency and university Web sites, and contacting the six members of our research advisory group (who were drawn from across Canada) and the three members of our decision-maker advisory group. We determined each research organization's eligibility through its Web site or a telephone or e-mail interaction. After mailing the survey and responding to telephone or e-mail queries from research organization directors who had received the survey, we determined that six of the research organizations were ineligible to participate. These organizations do not appear in either the numerator or denominator of our response rate calculations.

The four-page survey instrument covered five principal domains, each corresponding to one element of a knowledge-transfer strategy:

  1. What do research organizations transfer to their target audiences, and at what cost?
  2. To whom do research organizations transfer research knowledge, and with what investments in targeting them?
  3. By whom is the research knowledge transferred, and with what investments in assisting them?
  4. How do research organizations engage target audiences in the research process (and do they use supporting communications infrastructure such as Web sites and newsletters to transfer research knowledge)?
  5. Do research organizations perform evaluative activities related to knowledge transfer?

Some of these domains, such as “what is transferred,” can be considered as a continuum from more traditional approaches (e.g., providing full reports on research projects) to more innovative approaches (e.g., developing actionable messages based on bodies of research knowledge). Other domains, such as “with what effect is research knowledge transferred,” can be considered an “either/or” approach that is employed either across the board or not at all: a research organization either does or does not survey its target audiences about their awareness of, knowledge of, attitudes toward, and self-reported use of the research knowledge that has been the focus of the research organization's knowledge-transfer efforts. That said, some research organizations may assess awareness and not self-reported use of research knowledge because the latter is more vulnerable to criticism of social desirability bias. With few exceptions, questions were followed by a five-point Likert scale that captured the frequency with which a particular approach is used or activity is undertaken (i.e., never, rarely, occasionally, frequently, or always). A one-page insert was provided for respondents’ optional written comments.

We mailed the cover letter and survey instrument in the late summer of 2001, and then to nonresponders we mailed (at one-month intervals) a postcard reminder, a follow-up letter along with another copy of the survey instrument, and a second follow-up letter. We also telephoned nonresponders one to two months after the final mailing. The health research organizations received an endorsement letter from the president of the Canadian Institutes for Health Research along with their second follow-up letter. We sent a thank-you card to all directors who returned a survey and an individualized feedback form to all directors who requested one.

All data were entered separately by two individuals, and any discrepancies in the two datasets were resolved by a third individual in consultation with the data-entry personnel. Data were analyzed by type of research organization (health research organizations and economic/social research organizations) and by target-audience orientation (none, between one and three of the four target audiences identified previously, and all four target audiences). We established the target audiences for research organizations according to which of the four target audiences they reported frequently or always transferring research knowledge. We used a t-test to compare health research organizations with economic/social research organizations and an ANOVA to compare research organizations with different target-audience orientations.

We supplement the description of our findings with illustrative examples of some of the knowledge-transfer activities of the research organizations with which we are affiliated. By doing so, however, we do not mean to imply that these activities were representative of all research organizations engaging in a particular type of activity or that these activities were necessarily grounded in research evidence supporting their effectiveness. Little empirical evidence exists to support one approach over another to, for example, spending time with a target audience. We hope to spur evaluations of these alternatives by providing illustrative examples.

Results

The response rate was reasonably high overall (0.66), but much higher in the health sector (0.77) than in the economic/social sector (0.55). We found no difference in the overall response rate by language of correspondence (0.66 for English-speaking directors and 0.67 for French-speaking directors). We found few cases in which responses differed between health and economic/social research organizations. Therefore all tables list knowledge-transfer activities in descending order of frequency for the full study sample (not the order in which the activities were listed on the survey instrument), and most of our comments focus on our results on the full study sample.

About one-third of research organizations have moved beyond transferring project-specific reports and report summaries to developing messages for their target audiences that transcend particular research reports (0.34) or that specify possible action (0.30) (table 1). As an example, the Institute for Work & Health, an independent research organization, routinely develops actionable messages for clinicians who diagnose and treat injured workers. One such message pertains to the management of acute low-back pain: when a focused history and physical exam rule out red flags, neither x-rays nor lab tests are necessary and interventions should be kept to a minimum, with patients being reassured about the normal course of recovery, educated in self-management of pain, and encouraged to stay active and resume activities as soon as possible.

TABLE 1
What Do Research Organizations Transfer to Their Target Audiences and at What Cost?

Health research organizations target three groups—service providers (e.g., clinicians), managers, and public policymakers—with roughly equal frequency (table 2). Economic/social research organizations target public policymakers more frequently than either service providers or managers. A higher proportion of economic/social research organizations (0.77) than health research organizations (0.60) target public policymakers, and a lower proportion of economic/social research organizations (0.45) than health research organizations (0.74) target service providers. While many research organizations tailor their knowledge-transfer approach to their specific target audiences (0.60), a lower proportion dedicate resources to getting to know their target audiences (0.39), and an even lower proportion dedicate resources to skill building among their target audiences (0.20). The Canadian Cochrane Network and Centre is among the one in five research organizations that dedicate resources to skill building: affiliated researchers and staff conduct workshops individually designed for public policymakers in provincial health departments to guide them in the practical application of the actionable messages provided by Cochrane systematic reviews.

TABLE 2
To Whom Do Research Organizations Transfer Research Knowledge and with What Investments in Targeting Them?

Few research organizations dedicate resources to enhancing their internal capacity to transfer research knowledge through skill building among their knowledge-transfer staff (0.22), getting to know the research literature regarding effective approaches to knowledge transfer (0.21), or learning what constitutes a credible messenger for their target audiences and ensuring that their knowledge-transfer staff meet these expectations (0.17) (table 3). That said, many research organizations do employ dedicated staff with knowledge-transfer duties (0.63), and a fair number of them create explicit incentives (e.g., performance goals/measures and targeted stipends) for research staff to engage in knowledge-transfer activities (0.42). Moreover, research organizations dedicate a sizable proportion of their research budget to knowledge transfer (mean = 0.14; median = 0.10). The Institute for Work & Health is among the small group of research organizations that report investing heavily in enhancing their internal capacity for knowledge transfer. The institute invests almost one-fifth of its budget in its knowledge-transfer functions, and it employs three knowledge-transfer associates, one each for clinicians, workplace parties, and public policymakers. These associates—two of whom were drawn from the decision-making environments in which their target audiences are located—invest both time and resources in keeping abreast of the issues that their target audiences are facing. In addition, each member of the institute's scientific staff commits to a particular knowledge-transfer goal at the beginning of each year, and his or her performance against this goal is evaluated as part of the staffer's annual performance appraisal.

TABLE 3
By Whom Is Research Knowledge Transferred and with What Investments in Assisting Them?

Between one-third and two-thirds of research organizations engage representatives of their target audiences in different stages of the research process (table 4). A higher proportion of health research organizations (0.64) than economic/social research organizations (0.45) involve their target audiences in establishing the overall direction of their organization. McMaster University's Centre for Health Economics and Policy Analysis frequently uses representatives of one of its target audiences (the Ontario Ministry of Health and Long-Term Care) to develop specific research questions for ministry-responsive projects, but it always uses representatives of all its target audiences in its knowledge-transfer activities. The centre replaced its annual health policy conference with small, invitation-only, interactive workshops designed to engage representatives of its target audiences in discussions about the implications of the research knowledge that the centre produces. Each workshop session is facilitated by an individual who is perceived to be credible by all participating target audiences. The first ten minutes of each session is spent listening to a presentation by a researcher about an actionable message that has been developed from a body of research knowledge, and the remainder of the two-hour session is spent grappling with its implications for both action and further research.

TABLE 4
How Do Research Organizations Engage Representatives of Their Target Audiences in the Research Process?

Almost all research organizations supplement interactive processes with Web sites (0.91); fewer supplement them with other supporting communications infrastructure such as newsletters (0.60) or list-serves (0.33). The Web sites tend to be fairly passive, however, in how they engage target audiences: only about one-third notify target audiences when new material of potential interest to them has been posted (0.37). A lower proportion introduce research projects (while they are still in progress) that may have important implications for target audiences (0.21), and a still lower proportion provide a dedicated entry point (with dedicated text) for each target audience (0.11) (results not shown). Newsletters appear to be similar to Web sites in their lack of fine-tuning for different target audiences (results not shown). The Canadian Cochrane Network and Centre offers dedicated entry points on its Web site for each of its target audiences (citizens, clinicians, and public policymakers, as well as reviewers and the media) and the Program for Policy Decision-Making, a research program affiliated with McMaster University's Centre for Health Economics and Policy Analysis, does the same for its target audiences (research funders, research organizations, knowledge brokers, and public policymakers).

Only about one in ten research organizations performed any given type of evaluative activity related to knowledge transfer (results not shown). A consistently small proportion (between 0.08 and 0.12) of research organizations reported frequently or always assessing any changes in their target audiences’ awareness of, knowledge of, or attitudes toward particular bodies of research knowledge or assessing any changes in their target audiences’ self-reported behavior or actual behavior that could be attributed to their knowledge-transfer activities. McMaster University's Centre for Health Economics and Policy Analysis incorporated before-and-after assessments of awareness, knowledge, attitudes, and behavior into evaluations of select knowledge-transfer workshops and compared changes in these measures with those in a control group that included representatives of the same target audiences.

While responses rarely differed between health and economic/social research organizations, responses did differ among research organizations with different target-audience orientations (results not shown). A higher proportion of research organizations that targeted all four target audiences (compared with none or between one and three target audiences) (1) moved beyond transferring project-specific reports and report summaries to developing messages for their target audiences that transcended particular research reports or specified possible action; (2) tailored their knowledge-transfer approach to their specific target audiences and both dedicated resources to getting to know their target audiences and spent time with them discussing research reports and ideas transcending particular research reports; (3) dedicated resources to enhancing their internal capacity for knowledge transfer; (4) engaged their target audiences in several different stages of the research process; and (5) made better use of Web sites and newsletters. Groups with different target-audience orientations did not differ in their use of supporting communications infrastructure or in their lack of attention to evaluative activities related to knowledge transfer. A larger proportion of health organizations (0.24) than economic/social research organizations (0.19) targeted all four target audiences.

Interpretation

Self-reports of current knowledge-transfer activities can tell us where the most room for improvement is, regardless of whether responses tap into actual behaviors or a social desirability bias. Biased responses are presumably based on directors’ awareness and knowledge of what they think we want them to say, with the added constraint that in some domains they may be concerned that they could be questioned further. Our findings suggest that the directors of applied research organizations in Canada are at least reasonably aware of and knowledgeable about what the research literature suggests they should be doing. One finding that led us to believe that there is some social desirability bias in their responses is that more than one-third of research organizations (0.36) reported that they involved their target audiences in establishing the preferred research design and methods, a domain usually jealously guarded by researchers. At the same time, the directors of research organizations were remarkably frank about their not evaluating their knowledge-transfer activities, which suggests that this type of bias was not pervasive.

Canadian research organizations quite often report transferring research knowledge in ways consistent with our understanding of how best to undertake such activities, even though take-home messages from the field of research on knowledge transfer have been so difficult to identify. This finding, which was not what we had hypothesized, may reflect, in part, the strong leadership on knowledge-transfer issues in Canada by organizations such as the Canadian Health Services Research Foundation (a research funder), the Health Evidence Application and Linkage Network (a research network), and the Policy Research Initiative (a federal government initiative to increase the capacity of the Canadian government to identify, understand, and address the longer-term policy issues facing Canada and Canadians). These findings may also reflect the strong pressures for action on knowledge transfer from research funders such as the Canadian Institutes of Health Research and the Social Sciences and the Humanities Research Council of Canada. The newly created Canadian Institutes of Health Research, for example, has as its objective to “excel … in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened health care system.”1

Leadership and legislative mandates in this area may also exert reciprocal influences on one another over time. These forces may have been helped along by the nature of Canada's research community, with its relatively small size creating pressures to seek value for money in health services and health policy research and with its interconnectedness creating opportunities for the rapid diffusion of new approaches to knowledge transfer. These forces will likely feed on themselves in the future as graduate training experiences are shaped by exposure to a new knowledge transfer–specific training center and by more general health services and health policy research training centers emphasizing training in knowledge transfer.

The combination of few statistically significant differences between health and economic/social research organizations and higher response rates among health research organizations likely means that health research organizations more frequently transfer research knowledge in ways consistent with the available research evidence. The economic/social research organizations that responded to the survey are likely more interested in knowledge transfer than those that did not respond and are therefore more likely to have sought out better ways of transferring research knowledge. This finding is consistent with what we had hypothesized. That said, the economic/social research organizations for which we have data quite often reported transferring research knowledge in ways consistent with our understanding of how best to undertake such activities. Research organizations in other sectors with which we are familiar, such as the cultural policy sector, also appear to be headed in this direction. While similar clusters of innovation may exist in other countries, such as with the Agency for Healthcare Research and Quality in the United States, they may not have created the same degree of momentum. Questions about when and how these innovative clusters emerged in Canada and why more clusters may have emerged in Canada than in some other countries warrant more detailed exploration.

Canadian research organizations that identified all four possible groups as target audiences tended to report transferring research knowledge in ways consistent with our understanding of how best to undertake such activities more frequently than did research organizations that identified fewer possible groups as target audiences. This finding, which was contrary to what we had hypothesized, may reflect a greater commitment to knowledge transfer among those research organizations that exist to serve multiple target audiences (i.e., serving multiple target audiences is more complicated, so more resources are dedicated to it, a possibility borne out by the gradient in both the mean and median proportions of research budgets spent on knowledge transfer). The finding may also reflect a greater need for multiple approaches to knowledge transfer among those research organizations that serve multiple target audiences (i.e., serving multiple target audiences requires multiple approaches, so there is a greater likelihood that at least one approach will be consistent with the research evidence on knowledge transfer). Finally, the finding may reflect a greater appreciation of our understanding of how best to transfer research knowledge among those research organizations that have identified multiple target audiences (i.e., having identified target audiences is a marker for understanding the research evidence for knowledge transfer).

While we feel quite confident about the validity of our findings for Canadian health research organizations, we found three reasons why our sample of economic/social research organizations may not be representative of all economic/social research organizations. First, the research team and advisory groups were drawn from the health sector, so we may not have identified all eligible economic/social research organizations. Second, the cover letter and survey instrument clearly noted that the research team and advisory groups were drawn from the health sector, and the first question on the survey instrument (covering target audiences) provided health-related examples, so our perceived credibility among and relevance to economic/social research organizations may have affected their willingness to complete the survey. Third, the lack of an endorsement letter from a highly respected individual in the economic/social sector in our second follow-up attempt may also have affected their willingness to complete the survey. This possibility is partly supported by the change in response rates following our second follow-up letter: an increase of nine percentage points for health research organizations and only one percentage point for economic/social research organizations. Those economic/social research organizations that did respond may be more interested in knowledge transfer and therefore more likely than their nonresponding colleagues to report transferring research knowledge in ways consistent with our understanding of how best to undertake such activities.

Given the current practices in knowledge transfer, the opportunities for Canadian research organizations’ improvement appear to lie in developing actionable messages for decision makers (what is transferred), developing knowledge-uptake skills among target audiences (to whom is it transferred) and knowledge-transfer skills within their own organizations (by whom is it transferred), and evaluating the impact of knowledge-transfer activities (with what effect). Evaluation appears to be a particularly underexplored area. Possible reasons include a lack of infrastructure for evaluation, a lack of knowledge of how to undertake such an evaluation, the difficulties associated with undertaking such an evaluation, and concern about how the findings of an evaluation will be acted on (Lavis, Ross, McLeod et al. 2003). We recognize, however, that such implications pertain primarily to the research organizations that produce (or recognize the need to produce) syntheses or systematic reviews, not single studies, because only these research organizations provide (or have the ability to provide) the focus on bodies of research knowledge that we believe to be critical to the development of actionable messages.

Research funders may want to consider helping research organizations take advantage of these opportunities for improvement. Funders could structure the knowledge-transfer requirements for the research organizations they fund in ways conducive to these opportunities. For example, a funder could require research organizations to move beyond transferring reports on research projects to transferring actionable messages based on whole bodies of research knowledge. Such a move would help counter the academic incentives for focusing on peer-reviewed publications and against transferring research knowledge to decision makers. Academic tenure and promotion decisions are based on peer-reviewed grants, not just on peer-reviewed publications. If funders introduced particular knowledge-transfer requirements, academic institutions would not need to debate the relevance of and to quantify contributions to this area because their existing criteria would capture commitments to knowledge transfer. A funder could also require research organizations that produce similar types of research knowledge for similar types of target audiences to work together to undertake small-scale evaluations of their knowledge-transfer activities. Furthermore, a funder could sponsor skill-building workshops for both the target audiences of its funded research organizations and the research and knowledge-transfer staff of its funded research organizations.

Individualized feedback may also help research organizations take advantage of these opportunities to improve their knowledge-transfer activities. To promote a dialogue within the organizations we surveyed about how to improve their knowledge-transfer efforts, we provided confidential, individualized reports to the 110 directors who requested one (from a total pool of 171 respondents). Each report compares an organization's current practices with those of other research organizations. The precedent for such an endeavor was set on the knowledge-uptake front. The Canadian Health Services Research Foundation (2000) developed a self-assessment tool for decision-making organizations to use to promote a dialogue within their own organizations about how to increase their knowledge-uptake capacity. Individualized feedback was shown to facilitate clinicians’ uptake of research knowledge (Davis et al. 1995; Grimshaw et al. 2001; Oxman et al. 1995). Time will tell if the same proves true for research organizations.

Conclusion

Five questions—What should be transferred to decision makers? To whom should research knowledge be transferred? By whom? How? With what effect?—provide an organizing framework for a knowledge-transfer strategy. Opportunities for research organizations’ improvement can be found in the differences between the answers suggested by our understanding of the research literature and those provided by research-organization directors asked to describe what they do. In Canada, these opportunities include developing actionable messages for decision makers, developing knowledge-uptake skills among target audiences and knowledge-transfer skills in research organizations, and evaluating the impact of knowledge-transfer activities. Research funders can help research organizations take advantage of these opportunities. Opportunities for improvement in the research base from which answers to the five questions have been drawn can be found by conducting systematic reviews for each combination of question, target audience, and disciplinary perspective and/or methodological approach. We hope that asking the salient questions, placing them in a logical order, and providing preliminary answers to them will spur such reviews, as well as evaluations of the overall strategy.

Acknowledgments

This study was funded through an open grants competition by the Ontario Ministry of Health and Long-Term Care. The opinions expressed in this article are those of the authors and do not necessarily represent the views of the ministry. Lavis holds a Canada Research Chair in Knowledge Transfer and Uptake. He also receives partial salary support as Liberty Health Scholar, Canadian Institute for Advanced Research. He completed this study while he was a Canadian Associate of the Harkness Fellows Program. Abelson holds a Career Scientist Award from the Ontario Ministry of Health and Long-Term Care.

The authors thank their coauthors of the report on which this article was based (Tom Abernathy and Irving Gold), staff of the Commonwealth Fund who provided feedback on a presentation of the preliminary results from the survey (Karen Davis and Robin Osborn), an early participant in our planning group (Jane Coutts), members of our researcher advisory group (Carolyn DeCoster, Dianne Ferguson, Judy Birdsell, Kim McGrail, Pierre-Gerlier Forest, and Stephen Tomblin), members of our decision maker advisory group (Marion Emo, Ron Sapsford, and Mel Sweetnam), researchers who provided feedback on our survey instrument (Doreen Neville, Duncan Hunter, Renee Lyons, Steven Lewis, and Suzanne Ross), and decision makers who supported our funding proposal (Frank Fedyk and George Browman). We also thank Elisabeth Martin for translation assistance, Kelly Ross and Lisa Kitchen for data entry, and Liza Thong for administrative assistance.

Endnote

1An Act to establish the Canadian Institutes of Health Research, to repeal the Medical Research Council Act and to make consequential amendments to other Acts, 2000, S.C. 2000, c. 13.

References

  • Abelson DE, Carberry CM. Following Suit or Falling Behind? A Comparative Analysis of Think Tanks in Canada and the United States. Canadian Journal of Political Science. 1998;31(3):525–55.
  • Black N. Evidence Based Policy: Proceed with Care. British Medical Journal. 2001a;323:275–9. [PMC free article] [PubMed]
  • Black N. Public Accountability: One Rule for Practitioners, One for Scientists? Journal of Health Services Research and Policy. 2001b;6(3):131–2. [PubMed]
  • Canadian Health Services Research Foundation. Self Assessment Tool for Decision Making Organizations. Ottawa: 2000.
  • Canadian Population Health Initiative. An Environmental Scan of Research Transfer Strategies. Ottawa: Canadian Institute for Health Information; 2001.
  • Davis DA, Thomson O'Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of Formal Continuing Medical Education: Do Conferences, Workshops, Rounds, and Other Traditional Continuing Education Activities Change Physician Behavior or Health Care Outcomes? Journal of the American Medical Association. 1999;282(9):867–74. [PubMed]
  • Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing Physician Performance: A Systematic Review of the Effect of Continuing Medical Education Strategies. Journal of the American Medical Association. 1995;274(9):700–5. [PubMed]
  • Devereaux PJ, Choi PTL, Lacchetti C, et al. A Systematic Review and Meta-Analysis of Studies Comparing Mortality Rates of Private For-Profit and Private Not-for-Profit Hospitals. Canadian Medical Association Journal. 2002;166(11):1399–406. [PMC free article] [PubMed]
  • Drummond M, Cooke J, Walley T. Economic Evaluation under Managed Competition: Evidence from the U.K. Social Science & Medicine. 1997;45:583–95. [PubMed]
  • Egger M, Davey Smith G. Potentials and Promise. British Medical Journal. 1997;315:1371–4. [PMC free article] [PubMed]
  • Eisenberg JM. Health Services Research in a Market-Oriented Health Care System. Health Affairs. 1998;17(1):99–108. [PubMed]
  • Estabrooks C, Goel V, Thiel E, Pinfold P, Sawka C, Williams I. Decision Aids: Are They Worth It? A Systematic Review. Journal of Health Services Research and Policy. 2001;6(3):170–82. [PubMed]
  • Goldberg HI, Cummings MA, Steinberg EP, et al. Deliberations on the Dissemination of PORT Products: Translating Research Findings into Improved Patient Outcomes. Medical Care. 1994;32(7):JS90–110. [PubMed]
  • Grimshaw J, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, Grilli R, Harvey E, Oxman A, O'Brien MA. Changing Provider Behaviour: An Overview of Systematic Reviews of Interventions. Medical Care. 2001;39(suppl. 2):II2–45. [PubMed]
  • Hayward RSA, Guyatt GH, Moore K-A, McKibbon KA, Carter AO. Canadian Physicians’ Attitudes about and Preferences Regarding Clinical Practice Guidelines. Canadian Medical Association Journal. 1997;156(12):1715–23. [PMC free article] [PubMed]
  • Hoffmann C, von der Schulenburg JM-G. The Influence of Economic Evaluation Studies on Decision Making: A European Survey. Health Policy. 2000;52:179–92. [PubMed]
  • Huberman M. Research Utilization: The State of the Art. Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization. 1994;7(4):13–33.
  • Innvaer S, Vist GE, Trommald M, Oxman AD. Health Policymakers’ Perceptions of Their Use of Evidence: A Systematic Review. Journal of Health Services Research and Policy. 2002;7(4):239–44. [PubMed]
  • Landry R, Amara N, Lamari M. Utilization of Social Science Research Knowledge in Canada. Research Policy. 2001;30:333–49.
  • Lavis JN. Ideas at the Margin or Marginalized Ideas? Nonmedical Determinants of Health in Canada. Health Affairs. 2002;21(2):107–12. [PubMed]
  • Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, Abelson J. Examining the Role of Health Services Research in Public Policymaking. Milbank Quarterly. 2002;80(1):125–54. [PMC free article] [PubMed]
  • Lavis JN, Ross SE, McLeod CB, Gildiner A. Measuring the Impact of Health Research. Journal of Health Services Research and Policy. 2003 In press.
  • Lavis JN, Ross SE, Stoddart GL, Hohenadel JM, McLeod CB, Evans RG. Do Canadian Civil Servants Care about the Health of Populations? American Journal of Public Health. 2003;93:658–63. [PMC free article] [PubMed]
  • Lewis S. The Evolution of Health Research in Canada: The Premise, the Promise, and the Performance to Date. Paper presented to the National Academy of Medicine (Italy) Meeting on Health Policies and Research in Health Services, New York, February 23. 2000.
  • Lomas J. Finding Audiences, Changing Beliefs: The Structure of Research Use in Canadian Health Policy. Journal of Health Politics, Policy and Law. 1990;15:525–42. [PubMed]
  • Lomas J. Using “Linkage and Exchange” to Move Research into Policy at a Canadian Foundation: Encouraging Partnerships between Researchers and Policymakers Is the Goal of a Promising New Canadian Initiative. Health Affairs. 2000;19:236–40. [PubMed]
  • Lomas J, Anderson GM, Domnick-Pierre K, Vayda E, Enkin MW, Hannah WJ. Do Practice Guidelines Guide Practice? New England Journal of Medicine. 1989;321:1306–11. [PubMed]
  • Lomas J, Enkin MW, Anderson GM, Hannah WJ, Vayda E, Singer J. Opinion Leaders vs. Audit and Feedback to Implement Practice Guidelines: Delivery after Previous Caesarean Section. Journal of the American Medical Association. 1991;265:2202–7. [PubMed]
  • Oxman AD, Thomson MA, Davis DA, Haynes RB. No Magic Bullets: A Systematic Review of 102 Trials of Intervention to Improve Professional Practice. Canadian Medical Association Journal. 1995;153:1423–31. [PMC free article] [PubMed]
  • Pelz DC. Some Expanded Perspectives on Use of Social Science in Public Policy. In: Yinger JM, Cutler SJ, editors. Major Social Issues: A Multidisciplinary View. New York: Free Press; 1978. pp. 346–57.
  • Power EJ, Eisenberg JM. Are We Ready to Use Cost-Effectiveness Analysis in Health Care Decision Making? A Health-Services Research Challenge for Clinicians, Patients, Health Care Systems, and Public Policy. Medical Care. 1998;36(5):MS10–7. [PubMed]
  • Roos NP, Shapiro E. From Research to Policy: What Have We Learned? Medical Care. 1999;27(6):JS291–305. [PubMed]
  • Shapiro E. Gerontological Research and Health Policy: Ritual Dance or Policy Relevant? The Gerontologist. 1993;33(3):419–21. [PubMed]
  • Shonkoff JP. Science, Policy, and Practice: Three Cultures in Search of a Shared Mission. Child Development. 2000;71(1):181–7. [PubMed]
  • Sorian R, Baugh T. Power of Information: Closing the Gap between Research and Policy. Health Affairs. 2002;21(2):264–73. [PubMed]
  • Soumerai SB, Avorn J. Principles of Educational Outreach (“Academic Detailing”) to Improve Clinical Decision Making. Journal of the American Medical Association. 1990;263:549–56. [PubMed]
  • Soumerai SB, Ross-Degnan D, Fortress EE, Walser BL. Determinants of Change in Medicaid Pharmaceutical Cost Sharing: Does Evidence Affect Policy? Milbank Quarterly. 1997;75(1):11–34. [PMC free article] [PubMed]
  • Spitzer WO. The Nurse Practitioner Revisited: Slow Death of a Good Idea. New England Journal of Medicine. 1984;310(16):1049–51. [PubMed]
  • Stryer D, Tunis S, Hubbard H, Clancy C. The Outcomes of Outcomes and Effectiveness Research: Impacts and Lessons from the First Decade. Health Services Research. 2000;35(5):980–93. [PMC free article] [PubMed]
  • Walshe K, Rundall TG. . Evidence-Based Management: From Theory to Practice in Health Care. Milbank Quarterly. 2001;79(3):429–57. [PMC free article] [PubMed]
  • Weiss CH. The Many Meanings of Research Utilization. Public Administration Review. 1979;39:426–31.
  • Weiss CH. Policy Research: Data, Ideas, or Arguments? In: Wagner P, Weiss CH, Wittrock B, Wollmann H, editors. Social Sciences and Modern States: National Experiences and Theoretical Crossroads. Cambridge: Cambridge University Press; 1991. pp. 307–32.

Articles from The Milbank Quarterly are provided here courtesy of Milbank Memorial Fund

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • Cited in Books
    Cited in Books
    PubMed Central articles cited in books
  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...