LIMITATIONS

There are 3 main limitations to this evidence map. The first, common to all systematic reviews, is that we may have not identified all the potentially eligible evidence. If a systematic review was published in a journal not indexed in any of the 4 databases we searched, and we did not identify it as part of our search of references of included publications, then we would have missed it. Nevertheless, our search strategy did identify 370 systematic reviews, so we did not suffer from any lack of potential reviews to evaluate. An extension to this limitation is the included systematic reviews may themselves have missed some original research studies eligible for their review. The total number of studies included, across all the reviews that entered into our map, is more than 900 original research studies. As with reviews, therefore, the map does not suffer from a lack of original research studies.

The second limitation of evidence maps is that we did not independently evaluate the source evidence; in other words, we took the conclusions of the authors of the systematic review “at face value.” That is the nature of an evidence map. Particular to this application of the mapping process, we did not map all the eligible reviews; for health conditions that had more than one eligible review, we only mapped the one we deemed most informative. This necessarily requires judgment, and others could disagree with that judgment. We list in the appendix all the reviews that were excluded from the map for this “overlap” reason, and interested readers can review them and select for themselves the one they judge most informative. As in all evidence-based products, and particularly in one such as this covering a large and complex evidence base, it is possible there are errors of data extraction and compilation. We used dual review to minimize the chance of such errors, but if we are notified of errors we will correct them.

Lastly, a limitation to assessing the effect of acupuncture is the variation (and controversy) with which sham acupuncture is designed. Some studies defined sham as standard needling technique at non-active points, some included shallow needling in both active and non-active points, and more contemporarily, non-penetrating needles used at both active and non-active points. One of the major controversies around the use of sham as an inert comparator is that the unintended physiologic effects beyond placebo have not been considered;⁷⁶ thus, the exact mechanism by which acupuncture is effective is unclear when compared to “sham” acupuncture. The uncertainty around what is considered “sham” acupuncture and the lack of clear understanding of the exact mechanism by which acupuncture is effective compared to sham acupuncture calls into question how we should assess conclusions from studies employing “sham” as a comparator. This uncertainty also renders conclusions about the effect of acupuncture compared to sham more challenging to interpret than, for example, the comparison of a pharmaceutical intervention to placebo, in which case the placebo is confidently assumed to be inert.

FUTURE RESEARCH

The vast majority of the conclusions of the eligible systematic reviews were classified as low or very low certainty of evidence, indicating that the most critical research need is for better evidence to increase certainty of evidence for acupuncture. Studies comparing acupuncture to placebo or sham are probably not the priority; rather the priority should be studies comparing acupuncture to other recommended/accepted/active therapies for the condition. In such studies, the type and schedule of acupuncture treatment needs careful documentation so findings can be applied in other settings. Studies comparing acupuncture to other recommended therapies should also have a sufficiently long follow-up time period to allow any nonspecific effects (eg, of getting something “new”) to dissipate. For example, for studies of chronic painful conditions, this time period has been proposed to be at least 6 months.

For health conditions of priority to the VA that currently do not have at least moderate-certainty evidence supporting use of acupuncture, new studies that address limitations of existing research are needed. More rigorous evidence on acupuncture is likely the best way to expand access to acupuncture among Veterans most likely to benefit from it. The need for more rigorous research also applies to the acupuncture community at large. We note that in the 9 years covered by this update, we identified 370 new systematic reviews of acupuncture. This compares to about 370 new RCTs of acupuncture published in the same time period and included in the systematic reviews in our map. Thus, researchers interested in acupuncture are producing about as many systematic reviews (that generally conclude the certainty of evidence is low or very low) as new RCTs needed to raise the certainty of evidence. This seems to be a mismatch between resources and need. The field of acupuncture would be best advanced with resources devoted to producing more high-quality RCTs and producing fewer new systematic reviews.

CONCLUSIONS

There are many systematic reviews of acupuncture for more than 4 dozen adult health conditions. The number of conclusions about the effectiveness of acupuncture that were judged to have at least moderate certainty of evidence is small relative to the large number of existing RCTs and reviews. Most of these studies compare acupuncture to sham or control acupuncture for painful conditions or pain outcomes. There is no evidence that acupuncture is less safe than usual care for these conditions.