The purpose of this study was to develop a method to define and rate the severity of adverse events (AEs) in emergency medical services (EMS) safety research. They used a modified Delphi technique to develop a consensus definition of an AE. The consensus definition was as follows: "An adverse event in EMS is a harmful or potentially harmful event occurring during the continuum of EMS care that is potentially preventable and thus independent of the progression of the patient's condition." Physicians reviewed 250 charts from 3 EMS agencies for AEs. The authors examined physician agreement using κ, Fleiss's κ, and corresponding 95% confidence intervals (CIs). Overall physician agreement on presence of an AE per chart was fair (κ = 0.24; 95% CI = 0.19, 0.29). These findings should serve as a basis for refining and implementing an AE evaluation instrument.