Selecting evidence of known quality to provide insights
This section is about measuring the strength of available evidence (appraisal) and what to do when the evidence is uncertain. If relevant clinical guidelines are available, you will still need to appraise quality.
Here you will learn to:
- How to appraise a research study
- The place of checklists and where to find them
- The influence of study type on appraisal
- How to practice when there is no evidence or the evidence is contested.
Getting started
-
Selecting literature for appraisal
Following your search for literature the next step is to apply clearly defined eligibility criteria to decide which articles you will keep. This helps to keep your review focused and the workload manageable.
Some of the criteria will come from your PICO (or similar) elements as this ensures article relevance to your question. For example, population characteristics might include ‘older people with dementia’. In this case, younger people with dementia might be excluded. Your criteria might also include or exclude certain types of publications, languages, recency, and contexts. Once you have selected articles that meet your criteria you are ready to appraise their quality.
-
Introduction to formal appraisal
All studies including meta-analyses and systematic reviews need to be reviewed to ensure that they are of high quality. This might begin with the study design or methodology, but also includes consistency of or confidence in findings, generalisability, and applicability of the evidence.
An introduction to Critical Appraisal
Video from The University of Queensland Library Appraisal of quantitative research helps you to determine evidence validity (closeness to the truth), impact (size of the effect), and applicability (usefulness in your clinical practice). [1]
Appraisal of qualitative research helps us assess the appropriateness of the approach taken, sampling strategy and data collection methods and whether the conclusions are justified by the results. [2]
At this step it is important that you understand different study types, the evidence hierarchy and what it means, and the different ways findings should be reported to enable appraisal (what information should be available). This will help you to select and appraise the most robust evidence. There are many online courses available about appraising literature.
Remember when appraising a study for use in your practice, the study should: [3]
- address a clearly focused question relevant to your care issue
- have used valid methods to address this question
- have clear and clinically meaningful findings
- be applicable to my patient, population, or context of care.
-
Appraisal criteria and checklists
Appraisal checklists
There are many tools or checklists available to help you assess the quality of research evidence. Using checklists helps to standardise the approach across different articles and improves objectivity when appraising research articles. They also help you to summarise your findings including relevance to your patient.
The most appropriate appraisal checklist to use will depend on the study type. For example, AGREE II is used for assessing the quality of guidelines, AMSTAR is for Systematic reviews, and AACODS is for grey literature. If there is no checklist available then reporting guidelines for that study type might be used to informally assess study quality. Since the purpose of these is to ensure sufficient information is provided to enable assessment. CareSearch Evidence Tools provides access to appraisal checklists and reporting guidelines for some of the different study types relevant in palliative care. The EQUATOR network has more.
Cochrane has developed a video series Critical Appraisal Modules 2019 on appraisal of different study types.
The use of checklists helps you to take a systematic approach to appraisal that is consistent across all of the studies you examine. However, as a general introduction to appraisal and what is important the NHMRC suggests the following examples of key quality criteria according to study type (this list is supplemented with criteria from other sources as indicated). [4]
Randomised controlled trials
- Was the study double blinded?
- Was allocation to treatment groups concealed from those responsible for recruiting the subjects?
- Were all randomised participants included in the analysis?
Cohort studies
- How were subjects selected for the ‘new intervention’?
- How were subjects selected for the comparison or control group?
- Does the study adequately control for demographic characteristics, clinical features and other potential confounding variables in the design or analysis?
- Was the measurement of outcomes unbiased (i.e. blinded to treatment group and comparable across groups)?
- Was follow-up long enough for outcomes to occur?
- Was follow-up complete and were there exclusions from the analysis?
Case-control studies
- How were cases defined and selected?
- How were controls defined and selected?
- Does the study adequately control for demographic characteristics and important potential confounders in the design or analysis?
- Was measurement of exposure to the factor of interest (e.g. the new intervention) adequate and kept blinded to case/control status?
- Were all selected subjects included in the analysis?
Systematic reviews
- Was an adequate search strategy used?
- Were the inclusion criteria appropriate and applied in an unbiased way?
- Was a quality assessment of included studies undertaken?
- Were the characteristics and results of the individual studies appropriately summarised?
- Were the methods for pooling the data appropriate?
- Were sources of heterogeneity explored?
For appraisal of guidelines consider: [5]
- Was the guideline systematically developed?
- Are the recommendations clearly linked to the evidence?
- Are all guideline developers named?
- Is a statement of goals provided?
- Is the guideline organised for ease of use?
- Are recommendations made and clearly identified?
Appraisal of qualitative studies: [6]
Use of appraisal checklists for qualitative studies is contested. However, appraisal is a necessary step of EBP and there are checklists such as CASP-qualitative Studies Checklist (271kb pdf) available. Qualitative studies might use approaches including observation, interview, focus groups and surveys. Some concepts that you might refer to when assessing qualitative studies include:
- Transferability: is sufficient study information provided for you to make connections between the study’s data and broader community settings?
- Credibility: to what extent is the research account believable and appropriate, particularly in relation to what participants say and the interpretations made by the researcher?
- Reflexivity: do the researchers examine and explain how they have influenced the research project including question development, sampling, and data collection, analysis and interpretation?
- Transparency: how well are the research sampling strategies, data collection and analysis described? Where decisions affecting the study are made are these adequately explained and justified?
-
Assessing certainty - quantitative research
Part of the role of appraisal is to help users feel more certain about the findings. Bias can introduce uncertainty into the interpretation of findings.
Bias has been defined in research as ‘any process at any stage of inference that tends to produce results or conclusions that differ systematically from the truth’. [7] In other words, any process during a research project that leads to results or conclusions that are not true or that are uncertain.
Some of the common sources of bias relevant to palliative care research are defined below, the Oxford University CEBM catalogue lists more: [8]
-
-
Assessing rigour - qualitative research
-
Assessing effect size
-
When the evidence is uncertain or absent
Last updated 16 June 2026