About Systematic Reviews

Choosing the Best Systematic Review Critical Appraisal Tool

Critical appraisal is an important step in the systematic review methodology. It assesses the quality and validity of the studies to be considered in the research, helping ensure that the review results in accurate, balanced, and quality findings. Similar to how there are data extraction tools for systematic reviews, there are also critical appraisal tools that researchers can use to evaluate studies. With that said, there is no gold standard in it. These tools, which are usually checklists, must be relevant to the specific systematic review they appraise.

What is a Critical Appraisal?

Critical appraisal involves the evaluation of the quality, reliability, and relevance of studies, which is assessed based on quality measures specific to the research question, its related topics, design, methodology, data analysis, and the reporting of different types of systematic reviews.

Planning a critical appraisal starts with identifying or developing checklists. There are several critical appraisal tools that can be used to guide the process, adapting evaluation measures to be relevant to the specific research. It is important to pilot test these checklists and ensure that they are comprehensive enough to tackle all aspects of your systematic review.

What is the Purpose of a Critical Appraisal?

A critical appraisal is an integral part of a systematic review because it helps determine which studies can support the research. Here are some additional reasons why critical appraisals are important.

Assessing Quality

Critical appraisals employ measures specific to the systematic review. Through these, researchers can assess the quality of the studies—their trustworthiness, value, and reliability. This helps weed out substandard reviews, saving researchers’ time that would have been wasted reading full texts.

Determining Relevance

By appraising studies, researchers can determine whether or not they are relevant to the systematic review, such as if they’re connected to the topic or if their results support the research, etc. By doing this, the question “Can you include a systematic review in a scoping review?” can also be answered depending on its relevance to the study.

Identifying Flaws

Critical appraisals aim to identify methodological flaws in the literature, helping researchers and readers make informed decisions about the research evidence. They also help reduce the risk of bias when selecting studies.

What to Consider in a Critical Appraisal

Critical appraisals vary as they are specific to the topic, nature, and methodology of each systematic review. However, they generally have the same goal, trying to answer the following questions about the studies being considered:

  • Is the study relevant to the research question?
  • Is the study valid?
  • Did the study use appropriate methods to address the research question?
  • Does the study support the findings and evidence claims of the review?
  • Are the valid results of the study important?
  • Are the valid results of the study applicable to the research?

Critical Appraisal Tools

There are hundreds of tools and worksheets that can serve as a guide through the critical appraisal process. Here are just some of the most common ones to consider:

  • AMSTAR – to examine the effectiveness of interventions.
  • CASP – to appraise randomized control trials, systematic reviews, cohort studies, case-control studies, qualitative research, economic evaluations, diagnostic tests, and clinical prediction rules.
  • Cochrane Risk of Bias Tool – to assess the risk of bias of randomized control trials (RCTs).
  • GRADE – to grade the quality of evidence in healthcare research and policy.
  • JBI Critical Tools – to assess trustworthiness, relevance, and results of published papers.
  • NOS – to assess the quality of non-randomized studies in meta-analyses.
  • ROBIS – to assess the risk of bias in interventions, diagnosis, prognosis, and etiology.
  • STROBE – to address cohort, case-control, and conduct cross-sectional studies.

What is the Best Critical Appraisal Tool?

There is no single best critical appraisal tool for any study design, nor is there a generic one that can be expected to consistently do well when used across different study types.

Critical appraisal tools vary considerably in intent, components, and construction, and the right one for your systematic review is the one that addresses the components that you need to tackle and ensures that your research results in comprehensive, unbiased, and valid findings.

Once you identify an appropriate tool for quality assessment, it should be tested by two or more reviewers with a sample of included studies. Thereafter, reviewers should carry out quality assessment in duplicate and independently and the results summarized.

Learn More About DistillerSR