• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of canjsurgCanandian Journal of SurgerySubmit a ManuscriptEmail AlertsAbout CJS
Can J Surg. Apr 2007; 50(2): 119–123.
PMCID: PMC2384258

Language: English | French

Factors associated with citation rates in the orthopedic literature

Abstract

Introduction

Investigators aim to publish their work in top journals in an effort to achieve the greatest possible impact. One measure of impact is the number of times a paper is cited after its publication in a journal. We conducted a review of the highest impact clinical orthopedic journal (Journal of Bone and Joint Surgery, American volume [J Bone Joint Surg Am]) to determine factors associated with subsequent citations within 3 years of publication.

Methods

We conducted citation counts for all original articles published in J Bone Joint Surg Am 2000 (12 issues). We used regression analysis to identify factors associated with citation counts.

Results

We identified 137 original articles in the J Bone Joint Surg Am. There were 749 subsequent citations within 3 years of publication of these articles. Study design was the only variable associated with subsequent citation rate. Meta-analyses, randomized trials and basic science papers received significantly more citations (mean 15.5, 9.3 and 7.6, respectively) than did observational studies (mean retrospective 5.3, prospective 4.2) and case reports (mean 1.5) (p = 0.01). These study designs were also significantly more likely to be cited in the general medical literature (p = 0.02).

Conclusion

Our results suggest that basic science articles and clinical articles with greater methodological safeguards against bias (randomized controlled trials and meta-analyses) are cited more frequently than are clinical studies with less rigorous study designs (observational studies and case reports).

Résumé

Introduction

Les chercheurs veulent publier leur travail dans des journaux de premier plan afin d'avoir le plus d'impact possible. Le nombre de citations qui suivent la publication d'une communication dans un journal est un moyen d'en mesurer l'impact. Nous avons procédé à une étude du journal d'orthopédie clinique qui a le plus grand impact (Journal of Bone and Joint Surgery, American volume [J Bone Joint Surg Am]) afin de déterminer les facteurs associés aux citations qui ont suivi dans les trois années de la publication.

Méthodes

Nous avons dénombré les citations de tous les articles originaux publiés dans J Bone Joint Surg Am 2000 (12 numéros). Nous avons utilisé une analyse de régression pour déterminer les facteurs associés au nombre de citations.

Résultats

Nous avons trouvé 137 articles originaux dans le J Bone Joint Surg Am. Nous avons dénombré 749 citations subséquentes au cours des trois années qui ont suivi la publication de ces articles. La conception de l'étude a été la seule variable associée au taux de citations subséquentes. Les méta-analyses, les essais randomisés et les articles de science fondamentale ont été cités beaucoup plus souvent (en moyenne 15,5, 9,3 et 7,6 fois respectivement) que les études par observation (rétrospectives, moyenne de 5,3 fois; prospectives, moyenne de 4,2 fois) et les rapports de cas (moyenne de 1,5 fois) (p = 0,01). Les concepts de ces études étaient aussi beaucoup plus susceptibles d'être cités dans des publications médicales générales (p = 0,02).

Conclusion

Nos résultats indiquent que les articles de science fondamentale et les articles cliniques comportant la plus importante protection méthodologique contre la partialité (essais contrôlés randomisés et méta-analyses) sont cités plus souvent que les études cliniques dont le concept est moins rigoureux (études par observation et rapports de cas).

Investigators conducting surgical research aim to disseminate their findings to the widest possible audience. Citation rate of a study after publication is one measure of the study's exposure. Having a research paper cited by multiple sources after its original publication can facilitate the translation of its findings to patient care. That is, the quicker the information about important surgical research is spread to the surgical community, the more likely it is to be applied to patient care. Although the application of new techniques and research findings among surgeons is more complex than simply disseminating information to a vast audience, identifying factors that are associated with increased citation rates may assist authors in improving the impact of their work.

We focused on the orthopedic surgery literature, with the understanding that our results would likely be relevant to other areas of surgery. The Journal of Bone and Joint Surgery, American volume (J Bone Joint Surg Am) is currently the most frequently cited clinical orthopedic journal, with a reported impact factor of 2.14. The purpose of our study was to evaluate the factors associated with increased citation rates of individual studies. We hypothesized that study designs, such as randomized trials, would achieve higher citations than would study designs of lower levels of evidence, such as case series.

Methods

We included all original scientific articles published in the J Bone Joint Surg Am in 2000. Two investigators conducted hand searches of the journal, from January through December 2000. From each eligible article, we collected the following baseline information:

  • study design (retrospective observational study, prospective observational study, randomized trial, meta-analysis, and basic science);
  • sample size;
  • continent where the study was conducted; and
  • area of orthopedic specialty (spine, hip and knee reconstruction, pediatrics, trauma, sports medicine, upper extremity and oncology).

Citation counts

Using the first author's name, 2 of us queried the Institute for Scientific Information (ISI) Web of Science database (http://isiknowledge.com) to ascertain, as of March 31, 2003, the number of subsequent citations for each article after publication. If entering the first author's name failed to yield any citations for an article, we searched for the second and last author to limit misclassification of an article as having zero subsequent citations. We chose a 3-year period after publication (2000–2003) to assess citations, on the basis of previous reports (see Fassoulaki and colleagues1).

For each article identified in the ISI database, we collected the number of citations in orthopedic journals, the number of citations in nonorthopedic journals and the number of subsequent self-citations by one or more authors of the original published paper. A journal was considered orthopedic if the publication routinely included articles pertinent to orthopedic surgeons (i.e., those characterizing studies involving patients with musculoskeletal problems relevant to orthopedic surgical practice). We subdivided the number of citations in nonorthopedic journals into those published in the 5 highest impact medical journals (Journal of the American Medical Association impact factor = 17.6, New England Journal of Medicine impact factor = 29.1, British Medical Journal impact factor = 6.6, Annals of Internal Medicine impact factor = 11.1 and Lancet impact factor = 13.3). Gami and others2 suggested that bias may result from author self-citation. To determine whether a subsequent citation was a self-citation, we compared the complete authorship of the original journal articles with the complete authorship of all citing articles.

Data analysis

We used analysis of variance to compare the number of citations among journals, across different years, and between orthopedic and nonorthopedic journals. We used t tests to conduct post hoc comparisons. To account for multiple comparison tests. We corrected the alpha level, using the least squares difference technique.3 We also conducted multivariable regression to examine whether study design, sample size, geographical location of study or study topic predicted the number of subsequent citations. Residuals from the regression analysis were plotted to ensure that assumptions of normality were maintained.

Results

Study characteristics

We identified 137 original articles in J Bone Joint Surg Am published in 2000. Two were meta-analyses (level 1 evidence), 3 were randomized trials (level 1 evidence), 22 were prospective observational studies (level 2 evidence), 79 were retrospective observational studies (level 4 evidence), 20 were basic science reports and 11 were case reports. Sample sizes ranged from 1 to 2367 patients (mean 109). Most articles focused on joint arthroplasty and upper extremity disorders (70/137, 51%) and were conducted in North America (109/137, 80%) (Table 1).

Table thumbnail
Table 1

Subsequent citations

We identified 749 citations of the 137 original articles. The number of citations after publication ranged from 0 to 38 (mean 5.8); of these, 24 articles (17.5%) had received no citations up to March 31, 2003.

In our regression analysis adjusted for study design, sample size, self-citation, geographical location of study and study topic, study design was the only variable significantly associated with the number of citations after publication (p = 0.045). Meta-analyses, randomized trials and basic science papers received significantly more citations (mean 15.5, 9.3 and 7.6, respectively) than did observational studies (mean retrospective 5.3, prospective 4.2) and case reports (mean 1.5) (analysis of variance, p = 0.01). (Fig. 1) The overall rate of self-citations among the 749 citations was 12.2% (92/749). Study design was not associated with self-citations.

figure 10FF1
FIG. 1. Mean number of citations by study design. Randomized trials, meta-analyses and basic science papers received significantly more citations than did other study designs. *p < 0.05 when compared with meta-analyses.

Citations within orthopedic versus nonorthopedic journals

Of 749 citations, 225 (30%) appeared in nonorthopedic journals. Study design was significantly associated with citations in nonorthopedic journals (p = 0.02). Meta-analyses, randomized trials and basic science reports yielded significantly greater citations than did other study designs (p < 0.05) (Fig. 2). Nine articles in the J Bone Joint Surg Am were cited in 1 of the 5 highest impact general medical journals. Of these, 4 were prospective case series, 1 was a randomized trial and 4 were retrospective case series.

figure 10FF2
FIG. 2. Mean number of citations in orthopedic and nonorthopedic journals. Meta-analyses, randomized trials and basic science papers received significantly more citations in nonorthopedic journals than did other study designs (*p < 0.05 when compared ...

Discussion

In a review of 137 scientific publications in J Bone Joint Surg Am, we found that authors can expect, on average, to have 5 citations within 3 years of publication; we also found that meta-analyses, randomized trials and basic science reports are significantly more likely to be cited, and cited in nonorthopedic journals, than are other study designs. Further, although self-citation rate was substantial, it was not associated with overall citation rates.

The impact factor for a journal is the number of citations to articles published in the journal in the past 2 years divided by the number of articles published during that period. Impact factor is the frequency with which an average article in a journal has been cited in a particular year. ISI also reports the number of citations that individual articles receive. The use of citations to measure relative impact has been criticized for the following reasons: 1) the 2-year period set by the ISI for citations is arbitrary, 2) the ISI's database doesn't contain all available journals, and 3) the impact factor does not take into account self-citations. Although the use of the impact factor is recognized to have limitations, it is presently the preferred measure of an article's exposure.4,5

Strengths and limitations

One strength of our study is that we evaluated citations from all scientific articles over a complete year. In addition, we assessed citation counts for each article in duplicate. Our findings provide some vindication for using levels of evidence as a general measure of the hierarchy of study design. Papers with higher levels of evidence were more likely to be cited after publication than were lower-evidence publications (level 4). Our findings are limited by the fact that we did not assess the quality of the citation or whether it supported the index paper, nor did we point out flaws in the index paper.

Relevant literature

An important finding of our study is the 3.6-fold and 2.2-fold increases in citations with meta-analyses and randomized trials, respectively, when compared with observational studies. Our findings are supported by Weale and Lear6 who identified 139 surgical journals and 112 medical journals in the ISI database in 2001. These authors searched Medline to identify any randomized trials published in 1999 and 2000 in these 251 journals. Randomized trials accounted for a median of 2% of all published studies.6 The number of randomized trials published in a journal over the study period was significantly correlated with impact factor (r = 0.38, p = 0.01).

Our results are discrepant with those of Callaham and colleagues,7 who used the Science Citations Index database for all citations of published articles originally submitted to a 1991 emergency medicine specialty meeting. Of 204 included articles, the mean citations per year was 2.04 (95% confidence interval 1.6–2.4; range 0–20.9). The strongest predictors of citations per year was the impact factor of the original publishing journal and the study sample size. However, features of the study design (i.e., randomization, blinding, retrospective vs. prospective) had little influence on the number of citations. In another study, West and McIllwaine8 identified 417 citations in papers published in 1995–1997 in an addiction journal. No association between the number of citations and quality rating of the index paper was identified (r = -0.01, p > 0.05).

Studies evaluating the influence of self-citation rates on journal impact factors have reported conflicting results.1,610 This finding may be due to the differences in self-citation rates among journals. Fassoulaki and colleagues1 examined the influence of self-citation rates on journal impact factor in 1995 and 1996 issues of 6 different anesthesia journals. Self-citation rates ranged from 17%–35% in these journals and were positively correlated with journal impact factor (r = 0.90, p = 0.015).1 Motamed and colleagues9 identified self-citation rates of 4%–11.9% across 6 otolaryngology journals in 1997 and 1998; however, these investigators reported no significant correlation with journal impact factor (r = –0.31, p = 0.56). The self-citation rate (12.2%) in J Bone Joint Surg Am was similar to that reported by Motamed and colleagues.9

Conclusions

The publication of randomized trials, meta-analyses and basic science reports in J Bone Joint Surg Am led to significantly more citations and more citations in nonorthopedic journals than other study designs. Authors involved in orthopedic research may improve the impact of their work by pursuing basic science research or by designing clinical studies with greater methodological safeguards against bias. Increased dissemination of research findings may lead to quicker adoption of clinically important research findings to patient care.

Notes

Dr. Bhandari is funded, in part, by a Clinical Scientist Fellowship, Department of Clinical Epidemiology and Biostatistics, McMaster University. Dr. Busse is funded by a Canadian Institutes of Health Research Fellowship Award. Dr. Devereaux is funded by a Heart and Stroke Foundation of Canada/Canadian Institutes of Health Research Fellowship Award. No funding was received in preparation of this manuscript.

Accepted for publication Jan. 27, 2006.

Competing interests: None declared.

Dr. Mohit Bhandari, Department of Clinical Epidemiology and Biostatistics, McMaster University Medical Centre, 1200 Main St. W., Rm 2C9, Hamilton ON L89 3Z5; fax 905 524-3841

References

1. Fassoulaki A, Paraskeva A, Papilas K, et al. Self-citations in six anaesthesia journals and their significance in determining the impact factor. Br J Anaesth 2000;84:266-9. [PubMed]
2. Gami AS, Montori VM, Wilczynski NL, et al. Author self-citation in the diabetes literature. CMAJ 2004;170:1925-7. [PMC free article] [PubMed]
3. Hochberg Y, Tamhane AC. Multiple Comparison Procedures. New York: John Wiley and Sons; 1987.
4. Lankhorst GJ, Franchignoni F. The ‚impact factor'–an explanation and its application to rehabilitation journals. Clin Rehabil 2001;15:115-8. [PubMed]
5. Lundberg G. The “omnipotent” Science Citation Index impact factor. Med J Aust 2003;178:253-4. [PubMed]
6. Weale AR, Lear PA. Randomised controlled trials and quality of journals. Lancet 2003;361:1749-50. [PubMed]
7. Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA 2002;287:2847-50. [PubMed]
8. West R, McIllwaine A. What do citation counts count for in the field of addiction? An empirical evaluation of citation counts and their link with peer ratings of quality. Addiction 2002;97:501-4. [PubMed]
9. Motamed M, Mehta D, Basavaraj S, et al. Self citations and impact factors in otolaryngology journals. Clin Otolaryngol 2002;27:318-20. [PubMed]
10. Hecht F, Hecht B, Sandberg A. The journal impact factor: a misnamed, misleading and misused measure. Cancer Genet Cytogenet 1998;104:77-81. [PubMed]

Articles from Canadian Journal of Surgery are provided here courtesy of Canadian Medical Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...