Skip to Main Content
Banner Image

Systematic Reviews

What is critical appraisal ?

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context (Burls,1999)

We critically appraise the literature in order to:

  • Weigh up the evidence for usefulness
  • Assess benefits and strengths for research against flaws and weaknesses
  • Assess if findings are reliable and useful in our own context for decision-making

 

How to critically appraise studies?

There are several tools and checklist available to critically appraise studies. Here is CASP's freely available checklists for  appraising research papers

Some commonly asked questions include:

1. What is the research question and why was the study needed? 
2. Does the study have new findings? Is the work original and important?
3. Does the research question address the following components? ( PICO)

  • The group or population of patients
  • The intervention or therapy
  • The outcome

4. Did the authors use the right type of study design for the research question? 

  • Therapy questions > Randomized controlled trials
  • Experiences > Qualitative studies

5. Did the chosen study design minimise bias? 

6. Was the study designed in line with the original protocol i.e. stated methods? Eg., were there changes to the inclusion or exclusion criteria?

7. Has the study's hypothesis (expected outcome) been tested?

8. Is the analysis of the data accurate?  Eg., Are the conclusions based on the data and analysis?

9. Does the study contribute to the understanding of the problem being investigated?

10. What are the strengths and limitations of the study?

11. Are the findings of the study useful for clinical practice?

You will probably want to assign a 'level of evidence' and quality score using the tool most appropriate for your topic.

Critical appraisal Websites and tools

Center for Evidence Based Medicine (CEBM)

The Centre for Evidence-Based Medicine in Oxford (UK) provides tools to develop, teach and promote evidence-based health care. Has useful tools and downloads for the critical appraisal of medical evidence. Example appraisal sheets are provided together with several helpful examples.

Critical Appraisal Skills Programme (CASP)

The Critical Appraisal Skills Programme (CASP) helps people to find and interpret the best available evidence from health research.

The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses

This scale was developed to assess the quality of nonrandomised studies with its design, content and ease of use directed to the task of incorporating the quality assessments in the interpretation of meta-analytic results.

 AMSTAR2 (A Measurement Tool to Assess Systematic Reviews)

AMSTAR2 is an updated version of AMSTAR.  It is a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. 

The Revised Cochrane risk-of-bias tool for randomized trials (RoB 2)

RoB 2 is the revised (version 2) Cochrane risk-of-bias tool for randomized trials.  It is the recommended tool to assess the risk of bias in randomized trials included in Cochrane Reviews. 

The ROBINS-I tool (Risk Of Bias In Non-randomized Studies - of Interventions)

The ROBINS-I tool is based on the Cochrane RoB tool for randomized trials.  It is used to evaluate the risk of bias (RoB) in the results of non-randomized studies of interventions (NRSI) that compare the health effects of two or more interventions.  

Mixed Methods Appraisal Tool (MMAT)

A checklist that was designed to provide a quality appraisal tool for quantitative, qualitative and mixed methods studies included in systematic mixed studies reviews

The Joanna Briggs Institute (JBI) Critical Appraisal Tools   

Developed by the JBI and collaborators to assess the methodological quality of different study designs for systematic reviews..

Calculators

Useful calculators for Therapy, Harm, Diagnostic Studies and measuring Kappa-  from JAMAevidence ( login required)

Evidence Based Medicine Calculators The CEBM Statistics Calculator was created for your own personal use and testing purposes. It is to be used as a guide only. Medical decisions should NOT be based solely on the results of this program. Although this program has been tested thoroughly, the accuracy of the information cannot be guaranteed.

Diagnostic Test Calculator This calculator can determine diagnostic test characteristics (sensitivity, specificity, likelihood ratios) and/or determine the post-test probability of disease given given the pre-test probability and test characteristics. Given sample sizes, confidence intervals are also computed.