The empirical Bayes (EB) methodology has been applied for over 20 years now in conducting statistically defendable before-after studies of the safety effect of treatments applied to roadway sites. The appeal of the methodology is that it corrects for regression to the mean and traffic volume and other changes not due to the measure. There is, therefore, a natural tendency to put a stamp of approval on any study that uses this methodology, and to assume that the results can then be used in specifying crash modification factors for use in developing treatments for hazardous locations, or in designing new roads using tools such as the interactive highway safety design model (IHSDM). At the other extreme are skeptics who suggest that the increased sophistication and data needs of the EB methodology are not worth the effort since alternative, less complex methods can produce equally valid results. The primary objective of this paper is to capitalize on experience gained from two decades of conducting EB studies around the world to illustrate that the EB methodology, if properly undertaken, produces results that could be substantially different and less biased than those from more conventional types of studies. A secondary objective is to emphasize that caution is needed in assessing the validity of studies undertaken with the EB methodology and in using these results for providing crash modification factors. To this end, a number of issues that are critical to the proper conduct and interpretation of EB evaluations are raised and illustrated based on lessons learned from recent experience with these studies. These include: amalgamating the effects on different crash types; the specification of the reference/comparison groups; and accounting for traffic volume changes. Current and future directions, including the improvements offered by a full Bayes approach, are discussed.