• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jmlaJournal informationSubscribeSubmissions on the Publisher web siteCurrent issue of JMLA in PMCAlso see BMLA journal in PMC
J Med Libr Assoc. Jan 2007; 95(1): 70–76.
PMCID: PMC1773051

Peer assessment of journal quality in clinical neurology

Abstract

Objective: To explore journal quality as perceived by clinicians and researchers in clinical neurology.

Methods: A survey was conducted from August 2003 to January 2004. Ratings for 41 selected clinical neurology journals were obtained from 254 members of the World Federation of Neurology (1,500 solicited; response rate 17%). Participants provided demographic information and rated each journal on a 5-point Likert scale. Average ratings for all journals were compared with the ISI's journal impact factors. Ratings for each journal were also compared across geographic regions and respondent publication productivity.

Results: The top 5 journals were rated much more highly than the others, with mean ratings greater than 4. Mean journal ratings were highly correlated with journal impact factors (r = 0.67). Most of the top 10 journal ratings were consistent across the subgroups of geographic regions and journal paper productivity. However, significant differences among the different geographical regions and respondent productivity groups were also found for a few journals.

Conclusions: The results provide valuable insight on how neurological experts perceive journals in clinical neurology. These results will likely aid researchers and clinicians in identifying potentially desirable research outlets and indicate journal status for editors. Likewise, biomedical librarians may use these results for serials collection development.

Highlights

  • The top five journals (Neurology, Brain, Annals of Neurology, Journal of Neurotrauma, and Stroke) were rated much more highly than the others.
  • Clinical and research neurologists' ratings of journal quality correlated well with journal impact factors.
  • Most of the top ten journal ratings were consistent across the subgroups of geographic regions and journal paper productivity, though some variability in perception of individual journal quality was found for a few journals in these two comparisons.

Implications

  • Peer assessment is a viable technique for assessing perceptions of journal quality in the health sciences.
  • Peer assessment data regarding perceived journal quality in a given field will likely provide a valuable tool for collection development decision-making by health sciences librarians.
  • High correlation between journal ratings and the journal impact factors suggests that the journal impact factor could be used to assess journal performance in the field of clinical neurology.

INTRODUCTION

Due to the primacy of journals in scholarly communication, the value of journals in an academic discipline has become increasingly significant [1, 2]. Journal quality may facilitate appraisal of academic publication records, especially with respect to tenure and promotion decisions, since publications in academic journals are judged as major indicators of a researcher's performance in academia [3, 4]; thus, journal quality serves as a surrogate measure of the quality of an individual's scholarly output [5, 6]. Furthermore, journal quality not only influences the prestige of the author but also that of the author's department and the associated university [710]. Additionally, journal evaluation provides academics with a better understanding of the publications most suitable for their research [11]. From the editors' perspective, journal evaluation is also important for the maintenance and improvement of their publications [12]. Finally, journal evaluation is important in assisting information professionals with the management of journal collections [13].

In some fields, the “image” of journals has been determined by surveying domain experts regarding their perceptions of the quality of journals as potential outlets for research. Through surveys, peer evaluation allows experts within the discipline to assess journal quality on the basis of their knowledge and experience. Peer assessment attempts to assign measures of value to journals based on the collective perceptions of those reasonably familiar with the journals [1, 14]. Despite possible weakness due to subjectivity, peer assessment is widely used in assessing journal quality in a variety of academic disciplines [1, 12, 1415].

In addition to peer assessment, the Institute for Scientific Information (ISI) journal impact factor has also been used to measure journal influence or importance in a given field [1617]. The impact factors of scientific journals have been utilized to rank and evaluate journals in the process of academic publication evaluation as an approximation of the prestige or importance of journals within specific disciplines [13, 14, 17].

In the health sciences, journal impact factors may be more commonly used than peer assessment for evaluating journal quality [18]; published studies employing the method of peer assessment have been conducted only in internal medicine [19] and health-care management [15, 20].

The main objective of this study was to gather data on journal quality as perceived by clinicians and researchers in clinical neurology. Journal ratings from the survey were also compared with the ISI journal impact factors to compare the two journal evaluation methods. To the authors' knowledge, this survey is the first attempt to identify and evaluate perceived quality of journals in clinical neurology.

METHODS

Journal selection

Journals selected for this survey project were among the 137 clinical neurology journals included in ISI's Journal Citation Reports (JCR) for 2000; this JCR category includes journals in neurosciences (such as neuropathology and neuroimmunology), psychology, neurosurgery, psychiatry, pediatrics, palliative care, and peripheral vascular diseases. This study defined clinical neurology more narrowly as a branch of medicine related to the structure, functions, and diseases of the nervous system and involving or concerned with the direct observation and treatment of living patients. To select major journals in clinical neurology using the study's narrower definition, one co-author (FB), a clinical neurologist, identified the ISI journals in clinical neurology used by specialists like himself. Overall, 40 journals from the ISI clinical neurology category were selected as those largely publishing articles relevant to clinical neurology rather than other closely related disciplines covered in the broad ISI subject listing. The co-author also selected the Journal of the Neurological Sciences in the ISI JCR journal subject category of Neurosciences and Neuroimaging as one of the most important journals in clinical neurology. Hence, 41 journals were used for this survey project.

Sample

The target participant sample was derived from members of the World Federation of Neurology (WFN), which has approximately 25,000 members worldwide. An invitation was sent via e-mail to the administration officer of WFN, who then distributed the message to WFN members. During the study period, there were approximately 1,500 members on the WFN e-mail list; hence, the sample was limited to these members. Due to technical constraints, the process of sending the invitation to research members in various countries was manually operated and required several months. Because of this lengthy process, there were no follow-up reminders.

Survey instrument

In the survey (see Appendix; online only), respondents were asked to provide demographic information such as research areas, current positions, number of publications in a peer-reviewed journal between 1998 and 2003, professional affiliations, and age within five categories. When respondents indicated two countries for their institutional affiliations, the first country was counted. Respondents were asked to rate each of the forty-one journals on a five-point Likert scale that ranged from “Excellent” to “Poor.” A “Cannot Rate” option was available for cases in which respondents were unfamiliar with or otherwise unable to rate a journal. Several open-ended questions allowed respondents to comment or to add and rate journals that they thought important in clinical neurology but not included in the survey list. Although respondents were encouraged to use the Web to complete and submit the survey, other options, such as completing a Word version of the survey and returning the result via email, fax, or postal mail, were also provided.

Statistical analysis

“Cannot Rate” was treated as missing data while “Excellent,” “Very Good,” “Good,” “Fair” and “Poor” were assigned codes of 5, 4 3, 2, and 1, respectively. SPSS (v. 13) was used for the analyses. Average values of the ratings for all journals were calculated. To compare the survey result with the ISI's journal impact factor, Pearson's correlations were calculated between the mean journal ratings and the journal impact factors. Due to the non-normal distribution of journal ratings and the data attributes (ordinal data), the Kruskal-Wallis test was employed to compare journal ratings across different regional groups and different groups of respondents' publication productivity for the top 10 journals.

RESULTS

Response rate and geographical distribution of respondents

The survey was conducted from August 2003 through January 2004. Over the 6-month period, 260 responses were received; 5 duplicates and 1 blank questionnaire were excluded. Thus, 254 validated and unique responses were collected (response rate of approximately 17%). Most (n = 240) of the responses were received via the Web, 9 via e-mail as Word attachments, 4 via fax, and 1 via postal mail. The institutional addresses of the respondents included 48 different countries, excluding 2 indecipherable and incomplete country names. The United States (n = 48) and Argentina (n = 33) ranked 1st and 2nd in the overall response rate.

The 48 countries were grouped into 7 major geographical areas (Table 1). Respondents from Europe constituted the largest proportion (n = 99, 39.0%) of respondents. The next 2 largest groups of respondents were from Latin America (n = 56, 22.0%) and North America (n = 49, 19.3%).

Table thumbnail
Table 1 Distribution of responses by geographic regions

Characteristics of respondents

Over three-quarters of the respondents (n = 196) described themselves as academics engaged in teaching, research, and clinical practice. Only a few selected the “other” option (n = 2) or did not respond (n = 9). Approximately one-third of the respondents (n = 81) regarded themselves as clinicians.

Respondents were able to nominate as many research areas as were applicable. As expected, most respondents (n = 231, 90.9%) indicated that their principal focus of research was in clinical neurology; however, some were also engaged in other research areas such as neurophysiology (n = 116, 45.7%), behavioral neurology or neuropsychology (n = 79, 31.1%), neurobiology (n = 64, 25.2%) and neuroimaging (n = 62, 24.4%).

Approximately 40% of respondents were in the 46– 55 age group (n = 107). The majority of respondents (n = 225, 89%) held at least one doctoral degree (e.g., MD or PhD); 84% (n = 213) had MD degrees, and of these, about two-fifths (n = 81) also indicated PhD degrees.

Of the 197 respondents (77.6%) who published at least 1 paper in a peer-reviewed journal between 1998 and 2003, 42 (16.5%) indicated that they had published more than 20 papers over this period. In contrast, about one-fifth of respondents (n = 52) did not report any peer-reviewed journal publication in the same period.

Perceived journal quality

The complete summary of respondents' journal ratings is shown in Table 2, listed in decreasing order of average mean rating. The first five journals (i.e., Neurology, Brain, Annals of Neurology, Journal of Neurotrauma, and Stroke) were rated more highly than the others, with mean ratings greater than four (“Very good”).

Table thumbnail
Table 2 Journals in decreasing order of mean rating

In addition, as response rates could be regarded as a surrogate for respondents' degree of familiarity with a given title, the first 5 journals were also those with which respondents were more familiar as evidenced by their high response rates. Not surprisingly, Neurology, the official journal of the American Academy of Neurology, was ranked 1st with 241 responses. Among the top 5 ranked journals, ratings for Stroke were more variable than for the other 4 journals, as evidenced by its somewhat higher standard deviation. Fewer than 30% of journals fell below a rating of “Good,” but their mean ratings did not fall below “Fair” (ratings between 2.0–2.5). Movement Disorders was ranked lowest, with a mean rating of 2.38. The distribution of the range of mean ratings is shown in Figure 1; both Figure 1 and Table 2 indicate that most journals were rated around 3.0 (“Good”).

Figure 1
Distribution of peer ratings of journal quality (Note: numbers above each range indicate the number of journals)

In terms of ISI impact factors, the top ranked journals in our study, Neurology, Brain, and Annals of Neurology, were ranked eighth, second and first, respectively, in the JCR subject category of clinical neurology for 2000.

In this study, investigators calculated Pearson's correlation between journal impact factors and mean journal ratings to test their potentially correlated relationship. The correlation coefficient was 0.67, indicating significant correlation between impact factors and mean journal ratings (P < 0.01).

Comparisons among regional groups

Ratings for the top 10 journals from the overall ranking were examined across geographic regions (Table 3). As there were only a few respondents from Africa (n = 3), Oceania (n = 7), the Middle East (n = 8), and Asia (n = 17), responses from these 4 regions were categorized (or grouped) as “Other” and were compared to responses from Europe (n = 99), Latin America (n = 56), and North America (n = 49). The statistical analysis indicated that there were significant differences among the different geographic regions for Journal of Neurotrauma and Journal of the Neurological Sciences (P < 0.05). Respondents from Latin America rated both Journal of Neurotrauma and Journal of the Neurological Sciences higher than those from Europe, North America, and other regions, while respondents from other regions rated Journal of the Neurological Sciences lower than did respondents from Europe, North America, and Latin America. The remaining titles among the top 10 journals were rated consistently among the regions.

Table thumbnail
Table 3 Average journal ratings among regional groups of respondents for the top ten journals

Comparison among groups in journal article productivity

The survey results for the top ten journals were also examined to identify whether there were any significant differences in ratings based on the publication productivity of respondents. Based on the number of articles respondents published in peer-reviewed journals between 1998 and 2003, 6 groups of publication productivity were identified (Table 4). Overall, most of the top 10 journal ratings were consistent across the subgroups of journal paper productivity. However, the Kruskal-Wallis analysis indicated significant differences in the ratings for Annals of Neurology and Neurological Research among the levels of publication productivity (P < 0.05). For example, respondents who had published more than 20 peer-reviewed journal articles from 1998 to 2003 rated Annals of Neurology much higher than did those who published no or less than or equal to 20 peer-reviewed journal publications for the same period. In contrast, respondents who had published more journal articles (>15 articles) rated Neurological Research much lower than did those who published no or fewer (≤15 articles) peer-reviewed journal publications.

Table thumbnail
Table 4 Average journal ratings across various levels of respondent publication productivity for the top ten journals

Comments from respondents

In the survey, respondents were encouraged to add journals that they thought were important but were not included in the survey list. Respondents nominated a number of journals, including Current Opinion in Neurology, Lancet Neurology, Muscle & Nerve, and Practical Neurology. In addition to journals in the discipline of clinical neurology, several general medicine titles were also regarded as important: Lancet, the New England Journal of Medicine (NEJM), the Journal of the American Medical Association (JAMA), the Nature titles, and the Proceedings of the National Academy of Sciences. One respondent noted, “… the general journals are also of great importance to the neurology field although not every issue [has] specific neurological topics.”

DISCUSSION

Journals are not only the means by which scientific communities contribute to the knowledge base, but also the means by which scholars compete for prestige and recognition [21]. Accordingly, journal quality is important in assessing research performance of both institutions and individual researchers. Although journals can be assessed through citation indicators, this study used a survey to investigate perceived journal quality among researchers and clinicians in the clinical neurology specialty. Global survey results indicate that most of the top ten journals were rated consistently, regardless of the differences in regional groups or the publication productivity of respondents. Among the top five journals (i.e., Neurology, Brain, Annals of Neurology, Journal of Neurotrauma, and Stroke), the number of respondents and ratings varied only slightly, indicating that these five journals are both well recognized and valued by respondents in this survey. This study also found that mean journal ratings were highly correlated with journal impact factors, confirming that the ISI journal impact factor could be used to assess journal performance; this is consistent with other investigators' findings in internal medicine [19].

The international nature of the survey population also raises the issue of varying journal needs in different countries, as two significant differences in journal ratings were observed from the top ten journals in the regional comparison; these differences may indicate regional variability in perception of individual journal quality. Moreover, the observed differences in journal ratings across publication productivity also imply that productive researchers might have different perceptions of some journals as compared with those who had published few articles. In addition to the quantitative results of general perceptions of journal quality, many useful comments regarding publication outlets in clinical neurology were received. One respondent's comments noted that these results may assist the WFN and other groups such as the HINARI program (http://www.healthinternetwork.org) in identifying key titles for provision online free or at a reduced rate to developing country institutions.

Since a large majority of the respondents defined themselves primarily as academics rather than clinicians, these findings might be skewed toward higher impact factor journals used in research. In hospital libraries frequented by clinicians, features of journals (other than research quality) might be rated higher; hence, these results may have limited utility for information professionals in medical libraries catering primarily to clinicians. Further research of this type is needed in other medical specialties and with an equal proportion of clinicians involved in evaluating journals.

Due to the small sample size as compared to the total membership worldwide and the moderately low response rate, this study should be seen as exploratory. The survey technique in this study suffers from the general limitations of any survey. For example, the survey itself has limitations because it is a subjective instrument; poor memory or misunderstanding of the questions can lead to ambiguous responses and measurement errors. Moreover, as structured instruments, questionnaires allow little flexibility in response format. In addition, email distribution of a survey has its own inherent limitations, including undeliverable email addresses. Furthermore, journals in this study were selected by one professional, which might lead to more subjectivity in the survey design. Nevertheless, by answering the open-ended questions, most of the respondents validated the selected list of journals in this study.

Finally, the use of a Web-based survey distribution has its own limitations; the interface design or even the size and color of the Web page may influence the response rate. By providing options for open-ended answers and comments, and by making the interface clear, concise, and compatible with different Internet browsers, the authors attempted to overcome some of these limitations. It should also be noted that there is an extensive literature critiquing the use of the journal impact factor in peer review [1]; however, this study merely correlated findings with those of the journal impact factors.

CONCLUSIONS AND FUTURE DIRECTIONS

To the authors' knowledge, this survey project was the first to investigate perceptions of journal quality in the area of clinical neurology. The survey results enhance the knowledge of the status of journals in clinical neurology and provide valuable insight into neurological experts' perceptions of journal outlets in the field. These results are not only helpful for researchers, academics, and clinicians in locating potentially desirable research outlets in clinical neurology but also for journal editors to evaluate their journals' status. Likewise, biomedical librarians will also be interested in this study, as the survey results to some extent reflect both user perception of journal quality and user needs in the field of clinical neurology. Thus, librarians could use the survey outcome as a collection development aid to make decisions on serial collections.

The survey instrument used for data collection was also part of a larger project for developing and testing an integrated conceptual model of journal citation impact. The illustrative model for researchers and information professionals will enhance their understanding of the evaluative role of citation analysis and their ability to assess journal quality more objectively and comprehensively. The output of the project contributes to the overall development of improved programs for evaluating research performance in medicine and is beneficial for all who work on assessment of journal performance.

Supplementary Material

Appendix:

Acknowledgments

The authors would like to thank the World Federation of Neurology and its members for their support and cooperation, particularly WFN staff, Mr. Newton and Ms. Bilger, for their great help. We also thank Dr. Kimura for authorizing this study.

REFERENCES

  • Yue WP. Predicting the citation impact of clinical neurology journals using structural equation modeling with partial least squares [dissertation]. Sydney, Australia: University of New South Wales, 2004.
  • Bence V, Oppenheim C. The role of academic journal publications in the UK research assessment exercise. Learned Publishing. 2004.  Apr; 17(1):53–68.
  • Diamond AM. What is a citation worth? J Hum Resour. 1986.  Winter; 21(2):200–215.
  • Lindsey D.. Distinction, achievement, and editorial board membership. Am Psychologist. 1976;31(11):799–804.
  • Miller T, Dodge HR.. Ratings of professional journals by teachers of management. Improving College & Univ Teach. 1979;27(3):102–103.
  • Young ST. POM research productivity in US business schools. J Oper Manage. 1996.  Mar; 14(1):41–53.
  • Andrews WT, Mckenzie PB. Leading accounting departments revisited. Account Rev. 1978.  Jan; 53(1):135–138.
  • Coe RK, Weinstock I. Evaluating the management journals: a second look. Acad Manage J. 1984.  Sep; 27(3):660–666. [PubMed]
  • Stahl MJ, Leap TL, Wei ZZ.. Publication in leading management journals as a measure of institutional research productivity. Acad Manage J. 1988;31(3):707–20.
  • Weber RP, Stevenson WC. Evaluations of accounting journal and department quality. Account Rev. 1981.  Jul; 56(3):596–612.
  • Dióspatonyi I, Horvai G, and Braun T. Publication speed in analytical chemistry journals. J Chem Inf Comp Sci. 2001.  Nov-Dec; 41(6):1450–1456. [PubMed]
  • Forrester JP, Watson SS. An assessment of public administration journals: the perspective of editors and editorial board members. Public Admin Rev. 1994.  Sep-Oct; 54(5):474–482.
  • Nisonger TE.. Management of serials in libraries. Englewood, CO: Libraries Unlimited, 1998
  • Yue WP, Wilson CS.. An integrated approach for the analysis of factors affecting journal citation impact in clinical neurology. Proceedings of the 67th ASIS&T Annual Meeting; 2004 November 12–17; Providence, RI: ASIS&T. 2004;41:527–536.
  • Brooks CH, Walker LR, and Szorady R. Rating journals in health care administration—the perceptions of program chairpersons. Med Care. 1991.  Oct; 29(8):755–765. [PubMed]
  • Garfield E, Sher IH. New factors in the evaluation of scientific literature through citation indexing. Am Documentation. 1963.  Jul; 14(3):195–201.
  • Garfield E. Journal impact factor: a brief review. Can Med Assoc J. 1999.  Oct 19; 161(8):979–80. [PMC free article] [PubMed]
  • Fassoulaki A, Sarantopoulos C, Papilas K, Patris K, and Melemeni A. Academic anesthesiologists' views on the importance of the impact factor of scientific journals: a North American and European survey. Can J Anaesth. 2001.  Nov; 48(10):953–7. [PubMed]
  • Saha S, Saint S, and Christakis DA. Impact factor: a valid measure of journal quality? J Med Libr Assoc. 2003.  Jan; 91(1):42–46. [PMC free article] [PubMed]
  • McCracken MJ, Coffey BS. An empirical-assessment of health-care management journals—a business perspective. Med Care Res Rev. 1996.  Mar; 53(1):48–70. [PubMed]
  • Merton RK. Social theory and social structure. Glencoe, IL: Free Press, 1957.

Articles from Journal of the Medical Library Association : JMLA are provided here courtesy of Medical Library Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...