Given the complexity and diversity of research, program evaluation is critical to understanding and documenting the effectiveness of funded research in illuminating the linkages between the environment and human health. Mandates such as the Government Performance and Results Act of 1993 have required research agencies to look beyond measures of output (e.g., publications produced) toward metrics related to long-term outcomes on public health. Guidance from the U.S. Office of Management and Budget Program Assessment Rating Tool (PART) requires that outcomes of a program (managed by a single entity) be linked to a clear set of program and agency goals, yet be external to the research program (Office of Management and Budget 2006). When reviewing fundamental research programs using the PART guidance, managers of these programs face significant challenges in demonstrating a link between traditional research outputs and outcomes (Cozzens 1997). Health and environmental research organizations such as NIEHS have been challenged to define and measure outcomes distant in time and space from environmental health research (Van Houten et al. 2000). Outcome-based measures of accountability for research grants are inherently difficult, because by definition in the Federal Grants and Cooperative Agreement Act of 1977, grants have indirect benefit to and little substantial involvement by federal agencies.

To broaden this conception and to incorporate requirements for outcome-based program evaluation, our logic model of a research program provides a visual and conceptual representation of what broad impacts the research program is likely to have and how the impacts are achieved. The simplest structure that defines the impact of research on society is a linear progression:

We chose this format because much of the theoretical and methodological literature describing the research process either explicitly or implicitly provides information on the inputs, activities, outputs, or outcomes of research, as well as describing how these elements of the research process can be linked to one another (Powers et al. 2006). Even though the process may not be linear, our focus is on the influence of specific research program inputs on a range of outcomes and does not attempt to evaluate all the influences of a particular outcome. Definitions of the logic model components are presented below.

Inputs are resources that feed into the research program from NIEHS, other federal agencies, research institutions, and community and business partners (e.g., funding, staff qualifications, technical assistance, grantees, organizational resources, community resources).

Activities are actions that describe how the inputs are used to carry out the research program or project (e.g., grant awarding, exposure/risk assessments).

Outputs are the direct products of the research activities, such as publications, presentations, and new funding applications, as well as patents and products.

Outcomes are benefits or changes resulting from the use of the research outputs. Outcomes are defined further as short term, intermediate, long term, and ultimate. Assigning time frames to the four levels of outcomes is difficult, because the length of time taken is highly variable depending on the individual outcome and the many factors that may affect it. Short- to long-term outcomes may include:

Ultimate outcomes of environmental health research may include:

Two other components of the logic model as they related to the NIEHS extramural research portfolio include contextual factors and reservoir of knowledge. Contextual factors could potentially affect the research environment through availability of resources or shifts in research or policy priorities that create constraints or opportunities for the research program. Examples include political or society interests, external triggers such as a disease outbreak, state of the economy, and other national and global socioeconomic influences.

Knowledge of the human health effects of ambient airborne pollutants has increased over the last several decades, from an initial focus on ozone and pulmonary diseases such as asthma, to a growing scientific understanding of the effects of fine airborne particulate matter (PM) on cardiovascular disease (e.g., Dockery 2001; Donaldson et al. 2001; Pope et al. 2004). Figure 6 depicts NIEHS-funded research (A1) documenting the health effects of fine PM (B1). For example, NIEHS-funded researchers at Johns Hopkins University published results (B4) of mortality from fine PM in major U.S. cities (Samet et al. 2000) and on hospital admissions related to fine PM (Dominici et al. 2006). Subsequently, research results from these studies and others were disseminated (A4) by NIEHS and the institutions themselves through press releases (e.g., Johns Hopkins University 2000; NIEHS 2006a).

During the last decade, the U.S. EPA shifted its monitoring network to measure finer PM (A7), specifically, PM with diameter less than 2.5 μm (PM_2.5). The U.S. EPA revised its regulations to include an annual ambient standard for PM_2.5 (A8), conducting multiple stages of staff and public review of the new standard from the mid-1990s through 2006 (e.g., U.S. EPA 2004). NIEHS-funded research was cited in the regulatory docket (www.regulations.gov) of the later revisions as key evidence for the health effects of PM_2.5 (e.g., McConnell et al. 1999; Raizenne et al. 1996; Schwartz et al. 1999). States are required to submit implementation plans to achieve compliance with the new ambient standards; as part of these plans, state and local governments pass rules and regulations requiring industry and consumers to change their operations (C6) and reduce emissions (C7). Reduced emissions required by the state implementation plans will improve air quality to the new U.S. EPA standard by 2010 (A11).

In response to research documenting cardiovascular and other health effects, the U.S. EPA added fine PM to its air quality index reporting (U.S. EPA 1999) and specifically included cardiovascular effects in its public health messages (D4) (U.S. EPA 2003). Better knowledge of daily air pollution levels and the fact that those with heart disease are also at risk results in behavior modification by the public to reduce activity during pollution events (Bresnahan et al. 1997) and to advocate to reduce local emissions (D6), thus resulting in reduced human exposure and mortality on high-pollutant days (A12). Multiple studies (including some funded by NIEHS) over the last few decades contributed to and were cited by the EPA when setting and modifying the PM_2.5 standards.

As the influence is traced through the logic model, it becomes more diffuse and suffers from time discontinuities and lack of documentation. This example illustrates that, with a full evaluation and expert elicitation, it is possible to more specifically identify and semi-quantify the impact of NIEHS research, starting with this overview of potential influence.

The case of lowered blood lead levels through phase-out of leaded gasoline and other lead-containing products demonstrates the influence of a pathway through the logic model related to the impact of government policy changes. Figure 7 shows how activities of grant awarding (A1) and use of grant funds (B1) can lead to dissemination of agency-funded research results (A4). This sets the stage for policy assessment (A6), influencing changes in laws, regulations, and standards changes (A9), which in turn leads to improved environment (A11), reduced emissions from industry (C7), and reduced human exposure (A12), thus leading to ultimate improvements in human health status. The NIEHS has sponsored research on the health effects of lead for more than 20 years (Department of Health and Human Services 2002). Beginning in the 1970s, research funds from federal sources (vs. industry-sponsored studies) were allocated to the study of such health effects particularly in children [reviewed by Needleman (2000)]; NIEHS was a large supporter of these studies. Early studies showed that exposure to low levels of lead during early childhood can lead to delays in cognitive and behavioral development, such as lower IQ levels. Dissemination of these results was accomplished through early conferences and publications on low-lead toxicity; for example, an NIEHS-sponsored conference in 1974, the proceedings of which were published that same year in Environmental Health Perspectives. Information from studies like these added to the justification of the need to remove lead from gasoline starting in the 1970s. A criteria document, Air Quality Criteria for Lead (U.S. EPA 1977), assessed the scientific basis for regulation, and a standard of 1.5 μg/m³ (maximum quarterly calendar average) lead was set in 1978 (U.S. EPA 1978). These air pollution regulations have removed significant amounts of lead from the environment (U.S. EPA 1996, 2002). Data from the second and third National Health and Nutrition Examination Studies show that between 1976 and 1980, there was an average drop in blood lead levels of 30%, concurrent with a 50% reduction in use of leaded gasoline (Annest et al. 1983). These trends have continued with an 86% drop in lead poisoning of children in the United States since the late 1970s and other improvements in health (Meyer et al. 2003; NIEHS 2003). In this example, the NIEHS-funded research on blood lead levels supported ongoing regulations and contributed to the documentation of positive health effects.