Send to

Choose Destination
Transl Behav Med. 2019 Nov 20. pii: ibz164. doi: 10.1093/tbm/ibz164. [Epub ahead of print]

Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS).

Author information

Hathaway-Sycamores Child and Family Services, Pasadena, CA, USA.
University of Montana, Missoula, MT, USA.
Department of Global Health, University of Washington, Seattle, WA, USA.
Brown School, Washington University in St. Louis, St. Louis, MO, USA.
Kaiser Permanente Washington Health Research Institute, Seattle, WA, USA.
Departments of Global Health and Health Services, University of Washington, Seattle, WA, USA.
Hospital for Sick Children and University of Toronto, Toronto, Canada.
School of Medicine and Public Health, The University of Newcastle, Newcastle, Australia.
VA Ann Arbor Healthcare System, Center for Clinical Management Research (CCMR), Ann Arbor, MI, USA.


The use of reliable, valid measures in implementation practice will remain limited without pragmatic measures. Previous research identified the need for pragmatic measures, though the characteristic identification used only expert opinion and literature review. Our team completed four studies to develop a stakeholder-driven pragmatic rating criteria for implementation measures. We published Studies 1 (identifying dimensions of the pragmatic construct) and 2 (clarifying the internal structure) that engaged stakeholders-participants in mental health provider and implementation settings-to identify 17 terms/phrases across four categories: Useful, Compatible, Acceptable, and Easy. This paper presents Studies 3 and 4: a Delphi to ascertain stakeholder-prioritized dimensions within a mental health context, and a pilot study applying the rating criteria. Stakeholders (N = 26) participated in a Delphi and rated the relevance of 17 terms/phrases to the pragmatic construct. The investigator team further defined and shortened the list, which were piloted with 60 implementation measures. The Delphi confirmed the importance of all pragmatic criteria, but provided little guidance on relative importance. The investigators removed or combined terms/phrases to obtain 11 criteria. The 6-point rating system assigned to each criterion demonstrated sufficient variability across items. The grey literature did not add critical information. This work produced the first stakeholder-driven rating criteria to assess whether measures are pragmatic. The Psychometric and Pragmatic Evidence Rating Scale (PAPERS) combines the pragmatic criteria with psychometric rating criteria, from previous work. Use of PAPERS can inform development of implementation measures and to assess the quality of existing measures.


Implementation science; Measurement; Pragmatic measure; Psychometric


Supplemental Content

Full text links

Icon for Silverchair Information Systems
Loading ...
Support Center