NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

National Research Council (US) Committee on an Assessment of Research Doctorate Programs; Ostriker JP, Kuh CV, Voytuk JA, editors. A Data-Based Assessment of Research-Doctorate Programs in the United States. Washington (DC): National Academies Press (US); 2011.

Cover of A Data-Based Assessment of Research-Doctorate Programs in the United States

A Data-Based Assessment of Research-Doctorate Programs in the United States.

Show details

5Faculty Values as Reflected in the Two Illustrative Rankings

This study is valuable for both the comparative data it makes available and the importance it attaches to some of the collected data by conducting a survey of faculty and relating program ratings to measured characteristics. The values used throughout this report, for the two overall rankings and, taken separately, for the dimensional rankings, derive in part from faculty members’ answers to questions designed to measure faculty perceptions of the relative importance of program characteristics to the quality of doctoral programs.

The 21 characteristics identified by the committee and in the literature as important were divided into three categories are shown in Box 5-1.

Box Icon

BOX 5-1

Characteristics Included in the Faculty Weighting Process.

Faculty respondents were asked to choose up to four characteristics in each category that they thought were important. They were then asked to indicate which one or two of the four they found the most important. The final task was to assign an importance score of from 0–100 to each category; the sum of the importance scores over all categories was to equal 100.1

The five characteristics given the highest rating on each measure are shown in Table 5-1. Specifically, it shows the average ranking of the characteristic in the field across all 20 measures.2 This table makes the differences in the two rating methods clear. On the general survey (S measure) in all fields, the publication measure was very important. It was less important in the regression-based R measure, where for all fields size, as measured by the average number of Ph.D.’s was important. The percentage of faculty with grants was highly ranked on the S measure in all fields but the humanities. Awards per allocated faculty, a measure that may reflect reputation, was important in all fields but the agricultural sciences, and it was highly ranked for both the R and S measures in three of the five broad fields. None of the diversity measures appeared to be important in either methodology. GRE scores were important for R measures, but not for S measures, while placement of students in academic positions was important for S measures, but not for R measures.

TABLE 5-1. Most Highly Rated Characteristics of Doctoral Programs on R and S Measures.

TABLE 5-1

Most Highly Rated Characteristics of Doctoral Programs on R and S Measures.

Faculty values are also reflected in the relative importance of each category measured on the faculty questionnaire. For all fields the importance score for the faculty productivity variables was highest, followed by the student support and outcomes category, with program demographic characteristics coming in last. These category importance values are shown in Table 5-2. One interesting observation is that although these weights are different from one another in a statistical sense, they are remarkably similar regardless of the field of the respondents.

TABLE 5-2. Faculty Importance Weights by Broad Field.

TABLE 5-2

Faculty Importance Weights by Broad Field.

DIMENSIONAL MEASURES

Despite the relatively moderate importance that faculty placed on the student treatment and program diversity dimensions of doctoral education, the committee felt it was very important to measure and discuss these dimensions, in part because they have figured prominently in national discussions of doctoral education.3 The dimensional measures were obtained by means of the faculty responses to Section G (see Table 5-1).4 These measures take a subset of all the characteristics and recalculate the weights so that the total of the weights for the subset adds up to 1. The dimensional measures used in this study—research activity, student support and outcomes, and diversity of the academic environment—are described in the sections that follow.

Research Activity

This dimensional measure relates to the various ways in which to gauge the contribution of research: publications, citations (except for the humanities), the percentage of the faculty holding research grants, and recognition of scholarship as evidenced by honors and awards. The importance weights are shown in Table 5-2a. Specifically, the components of the research activity dimensional measure are average publications per allocated faculty member,5 average citations per publication, percentage of core and new doctoral faculty respondents holding grants, and awards per allocated faculty member.6 Publishing patterns and the availability of research funding and awards for scholarship vary by field, but the weight placed on publications per faculty member is remarkably consistent—about 30 percent—across fields. Research activity is the dimensional measure that most closely tracks the overall measure of program quality, because in all fields both the S measure—based on abstract faculty preferences—and the R measure place high weights on these characteristics.

TABLE 5-2A. Average Faculty Importance Weights on Components of Research Activity Dimensional Measure.

TABLE 5-2A

Average Faculty Importance Weights on Components of Research Activity Dimensional Measure.

For the research activity measures, faculty in the sciences and engineering place the greatest weight on grants per faculty member. In some fields research funding is common, and grants are an important source of support for faculty and doctoral students. In the social and behavioral sciences and the humanities, the greatest weight is placed on publications. In one social science discipline, economics, the weight placed on citations is almost equal to that placed on publications, but in all other fields publication is more highly valued. Grants are a less important source of funding in the humanities, and for those fields publications and awards are the most important visible signs of research activity. The values evinced in the broad fields are, with the exception of the humanities, very similar.

Student Support and Outcomes

This measure combines data on the percentage of students fully funded in the first year, the percentage of students completing their degrees in a given time period, time to degree, and placement in academic positions (including academic postdoctoral positions). The committee found that faculty typically placed a larger weight on student support and completion rates than on median time to degree or academic placement.7

Surprising uniformity appears across broad fields on the weights, which are shown in Table 5-2B.

TABLE 5-2B. Average Faculty Importance Weights on Components of the Student Support and Outcomes Dimensional Measure.

TABLE 5-2B

Average Faculty Importance Weights on Components of the Student Support and Outcomes Dimensional Measure.

The percentage of graduates obtaining academic positions dominates these measures, and, interestingly, the weight given to this variable (0.34–0.37) is essentially the same in all of the broad academic fields. The negative sign on time to degree indicates that the shorter the time to degree, the better. Student support in the first year is also an important variable in all fields. Percentage of completion and time to degree are less important, and although these variables have been discussed within the community of graduate deans, they are not variables that faculty feel are important in determining the quality of a doctoral program.

Diversity of the Academic Environment

The diversity measures—percentage of faculty and of students from underrepresented minority groups, percentage of faculty and of students who are female, and percentage of students who are international (that is, in the United States on a temporary visa)—did not appear to be major factors in determining the overall perceived quality of doctoral programs.8 When these measures are taken separately, definite patterns emerge for variables that faculty thought were more important, and these patterns vary by field. Most fields place the highest weight on the percentage of students from underrepresented minority groups. In the biological and health sciences, social and behavioral sciences, and humanities, relatively high weights are also placed on the percentage of faculty who are underrepresented minorities. The percentage of international students was not highly weighted relative to the other diversity weights, except for the physical and mathematical sciences. These weights, by broad field, are shown in Table 5-2C.

TABLE 5-2C. Average Faculty Importance Weights on Components of the Diversity Dimensional Measure.

TABLE 5-2C

Average Faculty Importance Weights on Components of the Diversity Dimensional Measure.

The preferences of faculty in the broad fields are very similar across fields. The physical and mathematical sciences place a greater weight on the percentage of students who are female than the percentage of students who from a underrepresented minority. This weighting is reversed for the other fields. None of the fields places a large weight on faculty diversity, although generally a slightly higher weight is placed on the percentage of faculty who are female. The physical sciences and engineering and, to some extent, the social sciences faculty indicate that a higher percentage of international students is beneficial and important to program quality. The relatively high weight for this measure for the humanities reflects high weighting in the foreign language fields and comparative literature.

SUMMARY OF THE FINDINGS

The findings of the committee fall into three areas:

  1. Indicators of research activity are of the greatest importance to faculty in determining program quality by means of the S measures, which are based on the program characteristics that faculty say explicitly are important. In many cases program size is very important when quality is measured by the regression-based, or R measures.
  2. Of the student support and outcome characteristics, placement in an academic position and support in the first year are highly weighted. Completion rates and time to degree are not.
  3. Faculty view student diversity as important, when considered with other diversity measures, but not as a direct predictor of overall program quality.

Footnotes

1

This was a forced choice. Faculty could not enter a characteristic beyond the ones given.

2

To calculate the ranking of the importance weight of a characteristic, the rank order (1–20, with 1 being the highest) of the median weight was calculated for each characteristic, and this rank was averaged across all the fields in the each broad field.

3

See, for example, Ronald G. Ehrenberg, Harriet Zuckerman, Jeffrey A. Groen, and Sharon M. Brucker, Educating Scholars: Doctoral Education in the Humanities (Princeton, N.J.: Princeton University Press, 2010).

4

These dimensional weights are different from the S weights, which take all 20 variables into account.

5

Because many faculty members supervise dissertations in more than one program, faculty members were allocated across these programs so that the total, taken across all programs, equaled 1 or less (when the faculty member was in a professional school).

6

In constructing this measure, a distinction was made between “highly prestigious” and “prestigious” awards, with the former given a weight of 5 and the latter given a weight of 1. The committee reviewed 1,393 awards and honors from various scholarly organizations. Highly prestigious awards were identified by the committee.

7

Ideally, the committee would have used a measure such as employment in one’s field five years after receipt of a Ph.D., but many programs did not collect such data. The committee hoped that including this measure would encourage more programs to pay attention to postdegree outcomes for their graduates.

8

In other words, the weights on these characteristics were small relative to other characteristics in the R and S measures.

Copyright © 2011, National Academy of Sciences.
Bookshelf ID: NBK83394

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (4.5M)

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...