NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

National Research Council (US) Committee to Examine the Methodology for the Assessment of Research-Doctorate Programs; Ostriker JP, Kuh CV, editors. Assessing Research-Doctorate Programs: A Methodology Study. Washington (DC): National Academies Press (US); 2003.

Cover of Assessing Research-Doctorate Programs

Assessing Research-Doctorate Programs: A Methodology Study.

Show details

4Quantitative Measures

This chapter proposes and describes the quantitative measures relevant to the assessment of research-doctorate programs. These measures are valuable because they

  • Permit comparisons across programs,
  • Allow analyses of the correlates of the qualitative reputational measure,
  • Provide potential students with a variety of dimensions along which to compare program characteristics, and
  • Are easily updateable so that, even if assessing reputation is an expensive and time-intensive process, updated quantitative measures will allow current comparisons of programs.

Of course, quantitative measures can be subject to distortion just as reputational measures can be. An example would be a high citation count generated by a faulty result, but these distortions are different from and may be more easily identified and corrected than those involving reputational measures. Each quantitative measure reflects a dimension of the quality of a program, while reputational measures are more holistic and reflect the weighting of a variety of factors depending on rater preferences.

The Panel on Quantitative Measures recommended to the Committee several new data-collection approaches to address concerns about the 1995 Study. Evidence from individuals and organizations that corresponded with the Committee and the reactions to the previous study both show that the proposed study needs to provide information to potential students concerning the credentials required for admission to programs and the context within which graduate education occurs at each institution. It is important to present evidence on educational conditions for students as well as data on faculty quality. Data on post-Ph.D. plans are collected by the National Science Foundation and, although inadequate for those biological sciences in which postdoctoral study is expected to follow the receipt of a degree, they do differentiate among programs in other fields and should be reported in this context. It is also important to collect data to provide a quantitative basis for the assessment of scholarly work in the graduate programs.

With these purposes in mind, the Panel focused on quantitative data that could be obtained from four different groups of respondents in universities that are involved in doctoral education:

University-wide. These data reflect resources available to, and characteristics of, doctoral education at the university level. Examples include: library resources, health care, child care, on-campus housing, laboratory space (by program), and interdisciplinary centers.

Program-specific. These data describe the characteristics of program faculty and students. Examples include: characteristics of students offered admission, information on program selectivity, support available to students, completion rates, time to degree, and demographic characteristics of faculty.

Faculty-related. These data cover the disciplinary subfield, doctoral program connections, Ph.D. institution, and prior employment for each faculty member as well as tenure status and rank.

Currently enrolled students. These data cover professional development, career plans and guidance, research productivity, research infrastructure, and demographic characteristics for students who have been admitted to candidacy in selected fields.

In addition to these data, which would be collected through surveys, data on research funding, citations, publications, and awards would be gathered from awarding agencies and the Institute for Scientific Information (ISI), as was done in the 1995 Study.

The mechanics of collecting these data have been greatly simplified since 1993 by the development of questionnaires and datasets that can be made available on the Web as well as software that permits easy analysis of large datasets. This technology makes it possible to expand the pool of potential raters of doctoral programs.


The 1995 Study presented data on 17 characteristics of doctoral programs and their students beyond reputational measures. These are shown in Table 4–1. Although these measures are interesting and useful, it is now possible to gather data that will paint a far more nuanced picture of doctoral programs. Indicators of what data would be especially useful have been pointed out in a number of recent discussions and surveys of doctoral education.

TABLE 4–1. Data Recommended for Inclusion in the Next Assessment of Research-Doctorate Programs. Bolded Elements Were Not Collected for the 1995 Study.


Data Recommended for Inclusion in the Next Assessment of Research-Doctorate Programs. Bolded Elements Were Not Collected for the 1995 Study.

Institutional Variables

In the 1995 Study, data were presented on size, type of control, level of research and development funding, size of the graduate school, and library characteristics (total volumes and serials). These variables paint a general picture of the environment in which a doctoral program exists. Does it reside in a big research university? Does the graduate school loom large in its overall educational mission? The Committee added to these measures that were specifically related to doctoral education. Does the institution contribute to health care for doctoral students and their families? Does it provide graduate student housing? Are day care facilities provided on campus? All these variables are relevant to the quality of life of the doctoral student, who is often married and subsisting on a limited stipend.

The Committee took an especially hard look at the quantitative measures of library resources. The number of books and serials is not an adequate measure in the electronic age. Many universities participate in library consortia and digital material is a growing portion of their acquisitions. The Committee revised the library measures by asking for budget data on print serials, electronic serials, and other electronic media as well as for the size of library staff.

An addition to the institutional data collection effort is the question about laboratory space. Although this is a program characteristic, information about laboratory space is provided to the National Science Foundation and to government auditors at the institutional level. This is a measure of considerable interest for the laboratory sciences and engineering, and the Committee agreed that it should be collected as a possible correlate of quality.

Program Characteristics

The 1995 Study included data about faculty, students, and graduates gathered through institutional coordinators, Institute for Scientific Information (ISI) and the NSF Doctorate Records File (DRF). For the humanities, it gathered data on honors and awards from the granting organizations. Most of the institutional coordinators did a conscientious and thorough job, but the Committee believes that it would be helpful to pursue a more complex data-collection strategy that would include a program data collector (usually the director of graduate studies) in addition to the key institutional coordinator, a questionnaire to faculty, and questionnaires to students in selected programs. This approach was tested with the help of the pilot institutions. The institutional coordinator sent the NRC e-mail addresses of respondents for each program. The NRC then provided the respondent a password and the Web address of the program questionnaire. A similar procedure was followed for faculty whose names were provided by the program respondents. Copies of the questionnaires may be found in Appendix D.

In 1995, programs were asked for the number of faculty engaged in doctoral education and the percentage of faculty who were full professors. They were also asked for the numbers of Ph.D.s granted in the previous 3 years, their graduate enrollment both full-time and part-time, and the percentage of females in their total enrollment. Data on doctoral recipients, such as time to degree and demographic characteristics, came entirely from the DRF and represented only those who had completed their degrees.

The Committee believed that more informative data could be collected directly from the program respondents. Following the 1995 Study, a number of questions had been raised about the DRF data on time to degree. More generally, the Committee observed that data on graduates alone gave a possibly biased picture of the composition and funding of students enrolled in the program. The program questionnaire contains questions that are directly relevant to these concerns.

In the area of faculty characteristics, the program questionnaire requests the name, e-mail address, rank, tenure status, and demographic characteristics (gender, race/ ethnicity, and citizenship status) of each faculty member associated with the program. Student data requested include characteristics of students offered admission, information on program selectivity, support available to students, completion rates, and time to degree. It also asks whether the program requires a master's degree prior to admission to the doctoral program, since this is a crucial consideration affecting the measurement of time to degree. The questionnaire also permits construction of a detailed profile of the percentage of students receiving financial aid and the nature of that aid. Finally, the questionnaire asks a variety of questions related to program support of doctoral education: whether student teaching is mentored, whether students are provided with their own workspaces, whether professional development is encouraged through travel grants, and whether excellence in the mentoring of graduate students by faculty is rewarded. These are all “yes/no” questions that impose little respondent burden.

Faculty Characteristics

In the 1995 Study, a brief faculty questionnaire was administered to the raters who produced the reputational rankings. These raters were drawn from a sample of faculty nominated by their institutional coordinators. The sample size reflected the number of programs in each field. The brief questionnaire asked raters the year, institution, and date of their highest degree as well as their current field of specialization. The Committee believes that the faculty questionnaire should be modified to collect certain other data. For example, the university origins of current faculty are a direct measure of which graduate programs are training Ph.D.s who become faculty at research universities. Data on date of degree would also permit a comparison of origins of recently hired faculty as compared to faculty hired, for example, more than 20 years ago. Although subfield data were collected for the 1995 Study, they were not used. They could be useful in improving program descriptions for potential graduate students and for assuring that specialist programs are rated by knowledgeable peers in the same specialty.

The Committee also believes that additional questions asked of faculty could permit a richer description of interdisciplinarity. For example, faculty could list all programs in which they have participated, either by teaching or serving on dissertation committees. Many faculty would be listed as members of more than one graduate program, and for the purposes of the reputational survey, the Committee recommends that they be listed as program faculty for all programs with which they are associated. To avoid the possibility of double counting the output of productive faculty, objective measures should be attributed pro rata among the various programs in which they are listed. The decision as to how to prorate an effort should be made by the faculty member with guidance that they should try to describe how time devoted to doctoral education (teaching and student mentoring) has been allocated among the programs for the past 3-year period.

The Committee was concerned that programs might want to associate a well-known faculty member with as many programs as possible in order to boost its rating, even if he or she were not involved with the program. Allocation of publications should serve to discourage this behavior.

Student Characteristics and Views

Student observations have not been a part of past assessments of research-doctorate programs. Past studies have included data about demographic characteristics and about sources of financial support of Ph.D. recipients drawn from the DRF and about graduate student enrollment collected from the doctoral institutions. Another student measure was “educational effectiveness of the doctoral program,” and for reasons discussed in Chapter 6, the Committee is recommending the elimination of this measure. The approach for measuring student processes and outcomes is discussed in Chapter 5.


The pilot trials were conducted over a 3-month period. The most important finding was that 3 months was barely sufficient for dealing with the study questionnaires. The full study should probably allow at least 4–6 months for data submission. The answers to many of the questions are prepared for other data collection efforts, but additional time is needed to customize answers to fit the taxonomy and to permit time for follow-up with nonrespondents.

All institutions carried out the trial through a single point of contact for the campus. This single point of contact worked with institutional research offices and program contacts to answer questions as well as interacted with NRC staff to assure that data definitions were uniform.

Electronic data collection worked well for institutions, programs, and faculty. We learned that it was better not to provide a hard copy alternative (as contrasted to Web response), since hard copy data simply had to be re-entered in databases once it was received by the NRC. All the pilot institutions store and access institutional and program data electronically. E-mail is the standard mode of communication with faculty and the rates of faculty response (60 percent) were high for a one-wave administration.

The Committee also learned that more precise definitions are needed to guide respondents. For example, when asking for data about “first-year doctoral students” a distinction may be needed about whether the students have a master's in the field. Care needs to be taken not to include terminal master's students, and precise definitions of “full-time” and “part-time” should be included.

The Committee learned the following from the questionnaire responses:

Institutional Questionnaire

  • Library expenditures: Not all institutions separate e-media expenditures from print expenditures.
  • Space: The questionnaire needs to provide guidance about how to allocate shared space. Answers to space questions also depend on how well the institution's programs fit the taxonomy. If the fit is poor, the allocation of space is arbitrary.
  • Graduate student awards and support are more appropriately queried at the program level.

Program questionnaire

  • Programs had difficulty filling out the inception cohort matrix but believed they could have done it if they had had more lead time.
  • Programs knew who their competitors were for doctoral students.
  • Programs that required GREs knew the averages and minima. For programs that do not require GREs, it would be helpful to ask what percentage of applicants submit GRE scores as well as report averages and minima only for those programs that are above a certain level (e.g., 80 percent).
  • Requests for faculty lists and faculty data should be separate from requests for other program data.

Faculty questionnaire

  • E-mail notifications must have a sufficiently informative subject heading so that they are not mistaken for spam.
  • Questionnaires should contain a due date.
  • Faculty associated with more than one program should be asked to fill out only one questionnaire. The NRC needs to develop procedures to duplicate information for the other programs with which a faculty member is associated.
  • Some faculty identified their program by a name other than that of the program that submitted their name. A procedure must be developed to resolve this problem.

Each pilot institution was asked to provide comments on the questionnaires. These comments, some of which are reported above, will be used as background material for the committee that conducts the full study. Draft questionnaires for the full study should be reviewed by a number of institutional researchers from a diverse set of institutions as well as by survey researchers.

Data Collected from Other Sources

The Committee recommends that most of the quantitative data presented in the 1995 Study from other sources be collected again. These include: publication and citation data from ISI, data on research grants from government agencies and large private foundations, data on books from the Arts and Humanities Citation Index, and data on awards and honors from a large set of foundations and professional societies. Student data from the Doctorate Record File should be considered for inclusion but checked for inconsistencies against institutional and program records. In the case of inconsistencies, a validation process should be designed.


The Committee recommends that the data listed in Bold type in Table 4–1 be added to the quantitative measures that were collected for the 1995 Study.

Copyright © 2003, National Academy of Sciences.
Bookshelf ID: NBK43472


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (7.5M)

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...