Appraising quantitative studies

Methods are available for specific study types:

• systematic reviews and meta-analyses (1–7)
• randomized controlled trials (8,9)
• cohort studies (10–14)
• case–control studies (15,16)
• economic evaluations (17,18).

Appraising qualitative studies

Saini & Shlonsky (19) have suggested a 25-item quality appraisal tool (QRQC) to evaluate the credibility, dependability, confirmability, transferability, authenticity and relevance of qualitative studies; this tool is applicable for variety of qualitative research methods. The National Institute for Health and Care Excellence (NICE) also has guidelines (20) as does the Critical Appraisal Skills programme (CASP) (21).

Mixed method appraisal tool

The MMAT Mixed Methods Appraisal Tool (22) is designed for systematic reviews that include qualitative, quantitative and mixed methods studies (23,24). The tool is used worldwide because it allows researchers to overcome the difficulties associated with using different critical appraisal tools for different designs.

Appraising grey literature

Grey literature includes productions in print and electronic formats by governments, academics, business and industry that are not part of the peer-reviewed publications from commercial publishers. It can also include theses or dissertations (reviewed by examiners who are subject specialists), conference papers (often peer reviewed or presented by those with specialist knowledge) and various types of report from specialists. The AACODS checklist (authority, accuracy, coverage, objectivity, date, significance) is designed to enable evaluation and critical appraisal of such grey literature (25).

References

1.

Lavis JN, Oxman AD, Souza NM, Lewin S, Gruen RL, Fretheim A. SUPPORT tools for evidence-informed health policy-making (STP). 9: Assessing the applicability of the findings of a systematic review. Health Res Pol Syst. 2009;7(suppl 1):S9. [PMC free article: PMC3271836] [PubMed: 20018116]

2.

Lewin S, Oxman AD, Lavis JN, Fretheim A. SUPPORT tools for evidence-informed health policy-making (STP). 8: Deciding how much confidence to place in a systematic review. Health Res Pol Syst. 2009;7(suppl 1):S8. [PMC free article: PMC3271835] [PubMed: 20018115]

3.

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10. [PMC free article: PMC1810543] [PubMed: 17302989]

4.

AMSTAR. Checklist [website]. Ottawa: Bruyère Research Institute; 2007. (http://amstar​.ca/Amstar_Checklist.php accessed 29 March 2017).

5.

Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew L, et al. From systematic reviews to clinical recommendations for evidence-based health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J. 2010;4:84–91. [PMC free article: PMC2948145] [PubMed: 21088686]

6.

Appendix B methodology checklist: systematic reviews and meta-analyses. London: National Institute for Health and Care Excellence; 2012. (https://www​.nice.org​.uk/process/pmg6/resources​/the-guidelines-manual-appendices-bi-2549703709​/chapter​/appendix-c-methodology-checklist-randomised-controlled-trials accessed 20 July 2017).

7.

CASP toolkit of critical appraisal: 10 questions to help you make sense of a systematic review. Oxford: Critical Appraisal Skills Programme; 2017. (http://media​.wix.com​/ugd/dded87_7e983a320087439e94533f4697aa109c.pdf accessed 29 March 2017).

8.

Appendix C methodology checklist: randomised controlled trials. London: National Institute for Health and Care Excellence; 2012. (https://www​.nice.org​.uk/process/pmg6/resources​/the-guidelines-manual-appendices-bi-2549703709​/chapter​/appendix-c-methodology-checklist-randomised-controlled-trials accessed 20 July 2017).

9.

CASP toolkit of critical appraisal: 11 questions to help you make sense of a trial. Oxford: Critical Appraisal Skills Programme; 2017. (http://media​.wix.com​/ugd/dded87_4239299b39f647ca9961f30510f52920.pdf accessed 29 March 2017).

10.

Gurwitz JH, Sykora K, Mamdani M, Streiner DL, Garfinkel S, Normand SLT, et al. Reader's guide to critical appraisal of cohort studies: 1. Role and design. BMJ. 2005;330(7496):895–7. [PMC free article: PMC556167] [PubMed: 15831878]

11.

Mamdani M, Sykora K, Li P, Normand SLT, Streiner DL, Austin PC, et al. Reader's guide to critical appraisal of cohort studies: 2. Assessing potential for confounding. BMJ. 2005;330(7497):960–2. [PMC free article: PMC556348] [PubMed: 15845982]

12.

Normand S-LT, Sykora K, Li P, Mamdani M, Rochon PA, Anderson GM. Reader's guide to critical appraisal of cohort studies: 3. Analytical strategies to reduce confounding. BMJ. 330. (7498):1021-3. [PMC free article: PMC557157] [PubMed: 15860831]

13.

Appendix D methodology checklist: cohort studies. London: National Institute for Health and Care Excellence; 2012. (https://www​.nice.org​.uk/process/pmg6/resources​/the-guidelines-manual-appendices-bi-2549703709​/chapter​/appendix-d-methodology-checklist-cohort-studies accessed 20 July 2017).

14.

CASP toolkit of critical appraisal: 12 questions to help you make sense of cohort study. Oxford: Critical Appraisal Skills Programme; 2017. (http://media​.wix.com​/ugd/dded87_5ad0ece77a3f4fc9bcd3665a7d1fa91f.pdf accessed 29 March 2017).

15.

Appendix E methodology checklist: case-control studies. London: National Institute for Health and Care Excellence; 2012. (https://www​.nice.org​.uk/process/pmg6/resources​/the-guidelines-manual-appendices-bi-2549703709​/chapter​/appendix-e-methodology-checklist-casecontrol-studies accessed 20 July 2017).

16.

CASP toolkit of critical appraisal: 11 questions to help you make sense of case control study. Oxford: Critical Appraisal Skills Programme; 2017. (http://media​.wix.com​/ugd/dded87_afbfc99848f64537a53826e1f5b30b5c.pdf accessed 29 March 2017).

17.

Appendix G methodology checklist: economic evaluations. London: National Institute for Health and Care Excellence; 2012. (https://www​.nice.org​.uk/process/pmg6/resources​/the-guidelines-manual-appendices-bi-2549703709​/chapter​/appendix-g-methodology-checklist-economic-evaluations accessed 20 July 2017).

18.

CASP toolkit of critical appraisal: 12 questions to help you make sense of economic evaluations. Oxford: Critical Appraisal Skills Programme; 2017. (http://media​.wix.com​/ugd/dded87_861b48c94b654b82a84250ca684d9186.pdf accessed 29 March 2017).

19.

Saini M, Shlonsky A. Systematic synthesis of qualitative research. Oxford: Oxford University Press; 2012.

20.

Appendix H methodology checklist: qualitative studies. London: National Institute for Health and Care Excellence; 2012. (https://www​.nice.org​.uk/process/pmg6/resources​/the-guidelines-manual-appendices-bi-2549703709​/chapter​/appendix-h-methodology-checklist-qualitative-studies accessed 20 July 2017).

21.

CASP toolkit of critical appraisal: 10 questions to help you make sense of qualitative research. Oxford: Critical Appraisal Skills Programme; 2017. (http://media​.wix.com​/ugd/dded87_25658615020e427da194a325e7773d42.pdf accessed 29 March 2017).

22.

MMAT mixed methods appraisal tool, version 2011. Montreal: Department of Family Medicine, McGill University; 2011. (http:​//mixedmethodsappraisaltoolpublic​.pbworks​.com/w/file/fetch​/84371689/MMAT%202011​%20criteria%20and​%20tutorial%202011-06-29updated2014​.08.21.pdf accessed 29 March 2017).

23.

Pluye P, Robert E, Cargo M, Bartlett G, O'Cathain A, Griffiths F, et al. Proposal: a mixed methods appraisal tool for systematic mixed studies reviews. Montreal: McGill University; 2011. pp. 1–8.

24.

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Pub Health. 2014;35(1):29. [PubMed: 24188053]

25.

Tyndall J. AACODS checklist for appraising grey literature. Adelaide: Flinders University; 2010. (https://dspace​.flinders​.edu.au/jspui/bitstream​/2328/3326/4/AACODS_Checklist.pdf, accessed 29 March 2017).