NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Russell R, Chung M, Balk EM, et al. Issues and Challenges in Conducting Systematic Reviews to Support Development of Nutrient Reference Values: Workshop Summary: Nutrition Research Series, Vol. 2. Rockville (MD): Agency for Healthcare Research and Quality (US); 2009 Mar. (Technical Reviews, No. 17.2.)

Cover of Issues and Challenges in Conducting Systematic Reviews to Support Development of Nutrient Reference Values: Workshop Summary

Issues and Challenges in Conducting Systematic Reviews to Support Development of Nutrient Reference Values: Workshop Summary: Nutrition Research Series, Vol. 2.

Show details

4Workgroup Exercise

The purpose of this project was to conduct an exercise to explore the feasibility of integrating systematic reviews into decision-making processes for deriving nutrient reference values, as well as to identify unanticipated issues and challenges. The EPC led an interactive process with a workgroup comprised of: 1) a panel of scientific experts that include scientists knowledgeable about establishing nutrient reference values such as those used to establish DRIs, 2) ODS-designated liaisons (including representatives from AHRQ, DHHS, FDA, FNB/IOM, ODS, and USDA/CNPP), and 3) EPC staff. Specifically, the aims of this project were to:

  • Identify the types of sequential questions that need to be informed by science in the process of making decisions relative to derivation of nutrient reference intakes.
  • Identify how these questions interact with the decision-making process and which questions lend themselves to systematic review approaches.
  • Identify options, with appropriate modifications, for incorporating systematic reviews and evidence-based methods into the nutrient reference intake decision-making process.
  • Describe pros and cons and appropriate and inappropriate applications of systematic reviews for nutrient reference intake decision-making processes.

The activities for the workgroup consisted of teleconferences, e-mail discussions, and two 1-day meetings. ODS-designated liaisons helped plan the activities and participated as observers and resources to the scientific panel meetings. The EPC staff and ODS-designated liaisons held teleconferences to select the case study nutrient, identified scientific experts to serve on the workgroup, and formulated the agenda for the meetings. Vitamin A was selected as the case study nutrient by ODS with input from the EPC and other ODS-designated liaisons prior to selection of scientific experts. Vitamin A was selected because it encompasses many of the challenges that are anticipated to arise in the process for revising the current DRIs.

Eight scientific experts from various academic institutions were invited to participate in the workgroup. They included vitamin A researchers, experts with a good understanding of the issues in setting nutrient reference values, and people with expertise in other nutrients in order to consider issues that may be encountered in nutrition topics other than vitamin A. Due to scheduling conflicts, two individuals who initially accepted the invitation were unable to participate in the workgroup teleconferences and meetings.

The workgroup exercise focused on the process of identifying and reviewing evidence to address specific questions. The workgroup did not make specific recommendations on nutrient values or how an expert panel should integrate the evidence into its decision-making process because these topics were outside the scope of the current project. While the workgroup carried out the exercise of formulating research questions for systematic reviews, building analytic frameworks, and screening a sample of potentially relevant abstracts, actual systematic reviews of the questions were not conducted.

Activities Surrounding the First Meeting

The first workgroup meeting was held on June 20, 2007. It focused on the nature of the questions related to systematic reviews and the process for interfacing with the decision-making paradigms associated with establishing nutrient reference values (the meeting agenda is presented in Appendix A). Prior to the meeting, ODS in conjunction with its designated liaisons and the EPC staff, drafted potential science-based generic questions that were thought likely to arise during deliberations on establishing nutrient reference values. The EPC then convened two teleconferences with the entire workgroup to review and revise the generic questions. The agreed upon preliminary questions guided the EPC staff to prepare a draft analytic framework and to conduct preliminary literature searches in preparation for the first meeting.

The meeting began with a discussion of the potentials and limitations of evidence-based methods, and the experiences of several groups (including AHRQ and the USPSTF) that have applied these tools. Workgroup members involved in the last vitamin A DRI panel also presented their experience. The EPC staff presented a draft analytic framework and preliminary questions derived from it. The workgroup discussed issues related to identifying indicators or outcomes that would be needed to determine nutrient reference values and how the standard systematic review process might usefully interface with the decisions-making process for establishing these values.

The EPC staff presented methodological approaches for selecting studies, critically appraising them, and summarizing evidence for systematic reviews. This was followed by a workgroup discussion of the relevance of these issues to the process of establishing nutrient reference values. The workgroup completed the session by specifying the PICO criteria for agreed upon key questions. An example of a key question that was formulated as a result of the discussion and upon which an evidence map was produced was “What level of plasma retinol and/or beta carotene is associated with greater risk of morbidity, mortality, xerophthalmia, immune dysfunction or compromise, and poor growth status in children?”

Following the workgroup meeting, the EPC incorporated workgroup suggestions and:

  • Refined the analytic framework (see Figure 2)
  • Refined the research questions for systematic reviews
  • Refined the review criteria (PICO)
  • Refined and implemented the MEDLINE® search strategies (Appendix B)
  • Screened abstracts obtained from the literature search and estimated the amount of literature available to address questions
  • Developed a data extraction form and performed data extraction on a few full-text articles as prototypes
  • Developed generic evidence and summary tables (Appendix C)
  • Generated evidence maps for questions to be addressed by systematic review

Activities Surrounding the Second Meeting

The second workgroup meeting was held on August 13, 2007. Workgroup members reviewed the refined analytic framework and an evidence map on one of the key questions, discussed an analytic framework for deriving upper intake limits, and provided feedback on the usefulness of integrating evidence-based methods to the process of developing nutrient reference values.

An important activity of this meeting was an abstract evaluation exercise led by the EPC staff (Appendix D). The purpose of this exercise was to provide workgroup members with the experience of evaluating abstracts obtained from literature search guided by the key questions. The goal was to gain an appreciation of the issues and challenges of an unbiased review of the literature. Prior to the second meeting, EPC staff selected 20 abstracts representative of different issues in applying eligibility criteria. At the meeting, workgroup members discussed the relevance of the article for the question of interest. The exercise highlighted disagreements among experts in evaluating potential evidence and revealed some of the fundamental problems in expert workgroups convening without an explicit analytic framework for assessing evidence. This exercise was instructive and much appreciated by the workgroup members. The success of this abstract exercise was reflected by the voluntary request from workgroup members for a similar exercise via a teleconference on questions related to upper tolerable limits. This was conducted on October 1, 2007, on 20 abstracts selected from a literature search on potential harms of vitamin A (Appendix E).

Analytic Framework for Vitamin A Reference Value

Discussed in this section are two specific analytic frameworks resulting from the workgroup meetings. These frameworks could be used to guide systematic reviews of the key questions.

Specific Example - Vitamin A Requirement

Figure 3 shows an example of a general analytic framework to establish adequate levels of vitamin A intake (e.g., when identifying means such as an EAR). Here, clinical outcomes considered were death, xerophthalmia, night blindness and immune dysfunction. These clinical outcomes need be more clearly defined in an actual systematic review, as discussed earlier in the Systematic Review Methods section.

Figure 3. Analytic Framework for adequacy of Vitamin A intake. Note that in the figure “Immune dysfunction/compromise” serves as a placeholder for a clinical outcome that has not yet been defined by the workgroup.


Figure 3. Analytic Framework for adequacy of Vitamin A intake. Note that in the figure “Immune dysfunction/compromise” serves as a placeholder for a clinical outcome that has not yet been defined by the workgroup. (more...)

Because of the buffering effect of liver stores of vitamin A on plasma concentrations the relationship between plasma retinol and dietary vitamin A intake reaches a plateau after a certain level of intake has been achieved (excess is stored in the liver). However, when liver vitamin A concentrations drop below a critical level (e.g., 20 μg/mg) serum vitamin A levels also decrease, resulting in a number of clinical and surrogate outcomes.

There are no validated surrogate outcomes for each one of the aforementioned clinical endpoints.

Specific Example - Vitamin A Excessive Intake

Figure 4 describes a general analytic framework that could be considered for the estimation of upper level of intake (e.g., when identifying a UL). The clinical endpoints are different from those considered for establishing requirements or adequate intake levels. Again, a separate analytic framework would be considered for each outcome.

Figure 4. Analytic Framework for Vitamin A Excessive Intake.


Figure 4. Analytic Framework for Vitamin A Excessive Intake.

Note that because bone mineral density is a valid surrogate for fractures we opted to consider it as a surrogate endpoint. A different biomarker of intake (plasma retinyl ester rather than serum retinol) is considered when focusing on excess intake. Circulating retinyl esters levels rise once the hepatic capacity to store vitamin A has been exceeded.

Evidence Map for Key Questions on Vitamin A Reference Values

The aim of the evidence map presented here is to provide an indication of the potential number of studies that might be available to address the relationships between vitamin A and/or beta carotene and the risk of morbidity and mortality, xerophthalmia, immune dysfunction or compromise in adults, and the risk of poor growth in children. In practice, only a fraction (5 to 30 percent, dependent on the topic) of abstracts identified as potentially relevant will be found acceptable upon full text evaluation.

After extensive consultation with workgroup members, the EPC staff searched MEDLINE® from 1950 to July 2007 using key words for serum or plasma vitamin A, retinol, retinyl ester, pro-vitamin A, and beta carotene, and exploring various MSH heading for outcomes of interests (e.g., mortality, xerophthalmia, dark adaptation, infection, failure to thrive). We limited the abstracts to English language and human studies. We excluded case reports and review articles. The completed search strategy is described in Appendix B.

The MEDLINE® search yielded a total of 3,170 abstracts. We screened all of these abstracts using the following eligibility criteria. These criteria were developed with input of the workgroup members based on the key question.

  • Inclusion criteria
    • Population: all human studies
      • For growth outcome - children only
    • Exposure: blood vitamin A measurements (e.g., plasma retinol, retinoic acids, retinal, retinyl ester); blood beta carotene, a vitamin A, or beta-carotene intervention, or the presence of “vitamin A deficiency”, or “vitamin A status” was measured
    • Comparator: different levels or concentrations in individual or population
    • Outcomes: as mentioned above according to the exposure
    • Study designs: any design
    • For mother-and-infant pair study - the exposure and outcome can be measured in either mother or infant, or both
  • Exclusion criteria:
    • Non-human
    • Study designs: in vitro1, ex vivo2 (or cell level outcomes only)
    • For outcomes of immune dysfunction or compromise outcome (including cancer outcomes) - cross-sectional design
    • Exposure: All-trans retinoic acid therapy
    • Diseases or conditions at baseline (in the whole study population):
      • Kidney diseases, liver disease, kidney, or liver transplantation, HIV-AIDS, diseases or conditions requiring medications that may interfere with lipid or protein metabolism, acute infectious diseases (e.g., malaria, meningococcal meningitis, trichuriasis)
      • For plasma retinyl ester only - hyperlipidemia

A total of 363 (11 percent) out of 3,170 abstracts were screened in by applying the above eligibility criteria. Table 1 shows the breakdown of the potentially qualifying abstracts per exposure and outcome category. It should be noted that the purpose of this project was not to actually review potentially relevant articles identified through the literature search.

Table 1. Number of potentially qualifying studies per exposure and outcome category among 363 abstracts meeting criteria †.

Table 1

Number of potentially qualifying studies per exposure and outcome category among 363 abstracts meeting criteria †.

Operationalizing the Model of Using Systematic Reviews to Inform the Development of Nutrient Reference Values

The model posits the formation of a workgroup composed of experts in the field of the nutrient of interest (domain experts) who are charged with drafting the recommendations concerning that nutrient, supported by a team of methodologists who are charged with reviewing (or assisting the review of) the evidence. Members may overlap between the two groups depending on the individual expertise and interest. Regardless, it is important for workgroup members to be familiar with the evidence-based review process so that the findings and limitations of the review are appropriately understood and utilized. Similarly, it is important for methodologists to understand the nuances and limitations of the scientific evidence base concerning the specific nutrient under review.

For those experts in nutrition who are unfamiliar with the process of systematic review, training in evidence-based process could be introduced at the beginning of the deliberations or on an ad hoc basis, as needed. Similarly, workgroup members who are experts in the field of nutrition could also serve as consultants to the methodologists, explaining the science concerning the specific nutrients.

The following are the potential responsibilities of the methodologists:

  • Conduct preliminary literature searches
  • Facilitate the first workgroup meeting where domain experts will be familiarized with the systematic review process, including formulation of the analytic framework and the key questions; study eligibility criteria; quality and applicability grading, and grading strength of evidence; how study results will be synthesized and presented
  • Continue workgroup training by either teleconferences and/or email
  • Perform literature review with further input from the workgroup
  • Create evidence tables for publication and use by the workgroup
  • Facilitate additional workgroup meetings as needed

The following are the potential responsibilities of the domain expert workgroup members:

  • Become familiar with the evidence-review process
  • Be available to the methodologists as domain experts in the evidence-review process
  • Help formulate literature search strategy
  • Help determine what data need to be extracted from studies and summarized in evidence tables
  • Suggest criteria for critical appraisal (quality assessment) of the primary studies
  • Review the content of evidence tables for accuracy and completeness
  • Discuss with the methodologists on how the systematically-gathered evidence may or may not be helpful in crafting the recommendations; and how the review process could be refined
  • Identify missing key studies

Numerous sequential and overlapping methods could be used for training of domain expert members of the workgroup. The entire workgroup can be introduced to the concepts and process of systematic review at an introductory face-to-face meeting. This process includes the important step of ensuring acceptance of the methodologies by the workgroup members, along with individualization of the process to the needs of both the specific nutrient of interest and the workgroup members. At this meeting, discussions would center on the analytic framework, formulation of the key questions, the rationale for specific inclusion and exclusion criteria, important factors for data extraction, and quality and applicability assessment of the evidence. Didactic presentations, group discussions, and hands-on exercises could be used at these workgroup meetings. Training in evidence-based methods could continue, in an iterative fashion, through e-mail, multiple teleconferences, subsequent written materials, individualized consultations regarding specific issues, and at subsequent workgroup meetings.



In vitro experimentation means manipulating the living tissues directly (e.g., incubate the pigs' hepatocytes in vitamin A enriched solution, then measure the enzymes of interest).


Ex vivo experimentation means isolating the living tissues of interest after manipulating the whole organism in some experimental way (e.g., feed the pigs extra loads of vitamin A, then isolate the pigs' hepatocytes for analysis)


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (274K)

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...