The JBI Approach to Evidence-Based Practice

The Joanna Briggs Institute (JBI) has as its central focus the effectiveness, appropriateness, meaningfulness and feasibility of health practices and delivery methods. Any indication that a practice is effective, appropriate, meaningful or feasible that is derived from experience, expertise, inference, deduction or the results of rigorous inquiry is regarded as a form of evidence by the Institute. The JBI regards the results of well designed research studies grounded in any methodological position as providing more credible evidence than anecdotes or personal opinion; however, when no research evidence exists, expert opinion can be seen to represent the "best available" evidence.

The JBI has supported work that has attempted to recognise the results of non-quantitative research as legitimate evidence for health care practitioners. The international interest in evidence-based practice arising largely out of the work of the Cochrane Collaboration focuses on the systematic review of evidence as a basis for the development of evidence-based guidelines and their utilisation in practice. The prevailing orthodoxy in systematically reviewing evidence elevates the results of experimental research to a position of superiority in terms of quality and applicability to practice and disregards the results of non-quantifiable research - because of its interest in subjectivity and interpretation -as legitimate evidence for practice.

Pearson¹ argues for a pluralistic approach when considering "what counts as evidence" for health care practices and Evans and Pearson² suggest that reviews that include both (or either) qualitative evidence and quantitative evidence are of importance to most practitioners. They go on to suggest,

however, that " ...optimal methods for reviewing qualitative research are still evolving" (p1). Since 2000, JBI has supported a number of programs utilising participatory processes to develop comprehensive, inclusive approaches to the review of evidence.

Clinical Effectiveness: An Evidence Base for Practice?

The evidence-based practice movement currently focuses on the effectiveness of interventions and activities and the term "systematic review" is now interpreted as a process that summarises and synthesises the result of experimental and other quantitative studies. The results of descriptive, observational and interpretive studies are therefore afforded little, if any, status in most systematic reviews. Evans and Pearson² suggest that "It can be argued that the approach of the systematic review should be used for all summaries of the research, ensuring that the care and rigour that was utilised by the primary researcher is maintained by the reviewer. However, this concept has not received wide acceptance and so the focus of most systematic reviews has remained predominantly fixed on the randomised controlled trial."

They go on to argue that the systematic review process should be expanded to include findings generated through all forms of rigorous research as well as those of randomised controlled trials and that evidence of "appropriateness" and "feasibility" are as important to health care practitioners and consumers as evidence of effectiveness. Appropriateness is defined by Popay, Rogers and Williams³ (p342) as: "the extent to which care can be said to meet the self-perceived needs of the person to whom it is being offered." Feasibility refers to the practicality and utility of an intervention or activity and to "factors that affect decision making among policy makers, clinicians, and patients (i.e., why people, both lay and professional, behave as they do when they do)"³ (p342).

JBI conceptualises evidence-based practice as clinical decision making that gives equal weight to the best available evidence; the context in which the care is delivered; client preference; and the professional judgement of health professionals/care staff⁴. The JBI approach involves the four major components of the evidence-based health care process:

1. Evidence Generation;
2. Evidence Synthesis;
3. Evidence/Knowledge Transfer; and
4. Evidence Utilisation.

Each of these components are modelled to incorporate their essential elements; and the achievement of improved global health is conceptualised as both the goal or endpoint of any or all of the model components and the raison d'être and driver of evidence-based health care (Figure 1).

Figure 1. JBI Model of Evidence-Based Health Care (From Pearson et al⁴)

Evidence-based health care is represented as a cyclical process that derives questions, concerns or interests from the identification of global health care needs and then proceeds to address these questions by generating knowledge and evidence to effectively and appropriately meet these needs in ways that are feasible and meaningful to specific populations, cultures and settings. This evidence is then appraised and synthesised and transferred to service delivery settings and health professionals who then utilise it and evaluate its impact on health outcomes, health systems and professional practice. Central to the JBI approach is a pluralistic approach to what constitutes legitimate evidence; an inclusive approach to evidence appraisal, extraction and synthesis; the effective and appropriate transfer of evidence to the point of service delivery via both print and online IT programs; and providing access to online programs to assist and motivate service providers to utilise evidence and evaluate/measure the impact of this utilisation on care delivery and on outcomes such as mortality, morbidity, health outcomes, quality of life and user satisfaction.

The JBI approach to searching for the evidence

In searching for the best available evidence, JBI policy emphasises the importance of developing a thorough search strategy for all JBI COnNECT searches and for all JBI systematic reviews of evidence.

A poorly structured search strategy can severely diminish the quality of an evidence summary or a systematic review as it may fail to identify important evidence pertinent to the topic or question. JBI therefore requires the use of a standardised search strategy for the development of evidence summaries entered into the JBI Database of Evidence Summaries. In JBI systematic reviews, the systematic review protocol must provide a detailed description of the search strategy to be used to identify all relevant literature within an agreed time frame. This should include databases and bibliographies that will be searched, and the search terms that will be used.

The JBI approach to appraising the evidence

JBI has adopted a position that requires the critical appraisal of all papers selected for inclusion in an evidence summary or a systematic review. JBI requires reviewers to use standardised critical appraisal instruments; and emphasises the need for evidence to be subjected to rigorous appraisal by two critical appraisers. The purpose of appraisal is to include only those studies or reports that are of high quality and thus to exclude those of poor quality. Given that the evidence review aims to summarise the best available evidence through, where possible, combining the results of sufficiently similar studies, it is important to note that the pooling of poor quality evidence may lead to outcomes that are less than desirable for patients. There is considerable disagreement about appraisal instruments for all evidence types; and about the appropriateness of critical appraisal in the systematic review of qualitative evidence.

Critical Appraisal Instruments

Critical appraisal is probably the most difficult component of the systematic review and a good understanding of research design is required. Many checklists have been developed to appraise quantitative reports and different checklists are currently used for the appraisal of a variety of quantitative research designs (for example, randomised controlled trials, cohort studies and descriptive studies). JBI has developed agreed appraisal instruments for:

• Economic studies;
• Experimental studies;
• Interpretive and Critical Research Studies; and
• Systematic Reviews

Click on the titles to download the JBI critical appraisal instruments

The JBI Systematic Review Process

The core of evidence-based practice is the systematic review of international literature on a particular care activity, condition, intervention or issue. The systematic review is essentially an analysis of all of the available literature (that is, evidence) and involves developing a question; establishing inclusion criteria; developing a strategy to comprehensively search for the evidence; appraising the quality of each paper; excluding papers of poor quality; extracting the findings of included papers; and synthesising the findings of included papers. JBI draws on the extensive work of the Cochrane Collaboration in its approach to systematic reviews and all JBI reviews of effectiveness are, whenever possible, conducted through a Cochrane Review Group following the Cochrane Handbook and using the Cochrane Collaboration's Review Manager software. JBI reviews of feasibility, appropriateness and meaningfulness (and, when necessary, effectiveness) are conducted using the JBI Systematic Review Handbook and the JBI SUMARI software suite. (click here - for information on JBI SUMARI) JBI systematic reviews involve the following steps:

Developing the review question

The protocol must state in detail the questions or hypotheses that will be pursued in the review. Questions should be specific regarding, for example, the consumers, setting, interventions or phenomena and outcomes to be investigated. For effectiveness questions, the PICO framework is recommended (Population, Intervention, Comparison and Outcomes); for other questions, the SPICE framework⁵ (Setting, Perspective, Intervention/phenomena of Interest, Comparison, Evaluation) is recommended.

When a review focuses on the effect of an intervention on specific outcomes or on prognosis, diagnosis or risk, precise descriptions of the interventions and outcomes are required. If the review is more concerned with the evidence on the subjective experience of a phenomenon, activity or intervention, outcomes may be less specific.

The systematic review protocol

JBI systematic reviewers must develop a protocol and submit it to JBI for peer review before commencing the review (for reviews of effects, protocols are approved by a relevant Cochrane Review Group before the review commences and JBI accepts this approval). Once a protocol is finalised, it is lodged in the JBI (or Cochrane) database of protocols so that other reviewers interested in a given topic can search these databases to avoid duplication of reviews.

The review protocol provides a predetermined plan to ensure rigour and transparency. It also allows for periodic updating of the review if necessary. Updating systematic reviews is imperative in a climate of continuous information production. New research knowledge is generated on a regular basis and therefore regular updates of systematic reviews are essential in order to ensure that health practices are based on the most current research evidence. JBI requires that this occurs on a three to five year cycle, or as deemed necessary (depending on the topic).

The JBI Protocol Template

THE JOANNA BRIGGS INSTITUTE REVIEW PROTOCOL
Title:
Background:
Objectives:
Criteria for considering studies for this review:	Types of Participants:
	Types of Intervention:
	Types of Outcome Measures:
	Types of Studies:
Search strategy for identification of studies:
Methods of the review:	Selecting Studies for Inclusion:
	Assessment of Quality:
	Methods used to collect data from included studies:
	Methods used to Synthesise Data:
	Date review to commence:
	Date review to complete:

Background literature review

The background presents a general evaluation of the literature to determine the scope and quantity of the primary research, to search for any existing reviews and to identify issues of importance. The background for a systematic review should include:

• A discussion of the review question itself and how it emerged
• An assessment of the significance of the topic to health care
• An overview of issues relevant to the review question
• An overview of previous reviews of the review topic, or of topics related to it
• A description of any controversies related to the review topic

The background should describe the issue under review including the population, interventions/phenomenom and outcomes of interest that are documented in the literature. It should provide sufficient detail to justify the conduct of the review and the choice of the various elements such as the interventions and outcomes.

Systematic Review Objectives

The protocol should state in detail the questions or hypotheses that will be pursued in the review.

Inclusion Criteria

The protocol must describe the criteria that will be used to select the literature. Specifically, in developing inclusion criteria, it is important to address:

• The types of studies to be included (for example, randomised controlled trials, pseudo-randomised controlled trials; or interpretive studies)
• The intervention, activity or phenomenon (for example, pharmaceutical and non-pharmaceutical interventions for smoking cessation)
• The outcome (for example, smoking cessation, smoking reduction)
• The specific study populations (for example, adult males aged 18 years and over who have been smoking for at least 5 years)
• Language of publication (for example, English only; or English, German, Spanish and Japanese)
• The time period (for example, study reports published or made available between 1999-2005)

Search Strategy

The protocol should provide a detailed strategy that will be used to identify all relevant literature within an agreed time frame. This should include databases and bibliographies that will be searched, and the search terms that will be used.

Assessment/Critical Appraisal Criteria

The protocol must describe how the validity of primary studies will be assessed and any exclusion criteria based on quality considerations. (In JBI reviews, the standardised JBI critical appraisal instruments appropriate to the evidence type must be used; for Cochrane reviews, reviewers select an instrument of their choice or design an appropriate instrument).

Data Extraction

The protocol must describe how data will be extracted from the primary research related to the participants, the intervention, the outcome measures and the results. Data extraction forms must be used (in JBI reviews, the standardised JBI data extraction instruments appropriate to the evidence type must be used; for Cochrane reviews, reviewers select an instrument of their choice or design an appropriate instrument).

Data Synthesis

Statistical analysis (meta-analysis) or textual analysis (meta-synthesis) may or may not be used depending on the nature and quality of studies included in the review. While it may not be possible to state exactly what analysis will be undertaken, the general approach should be included in the protocol.

The JBI Approach to the Meta-Analysis of Statistical Findings

Where possible quantitative research study results are combined in statistical meta-analysis using the Review Manager software from the Cochrane Collaboration (Review manager V4.04). All results must be double entered. Odds ratio (for categorical data) and weighted mean differences (for continuous data) and their 95% confidence intervals are calculated for analysis and heterogeneity is assessed using the standard Chi-square. Where statistical pooling is not possible the findings may be presented in narrative form.

The JBI Approach to the Meta-Syntheses of Qualitative Findings and of Opinion and Text

Where meta-synthesis is possible, qualitative research findings are pooled using the JBI Qualitative Appraisal and Review Instrument (QARI) software. This involves the aggregation or synthesis of findings to generate a set of statements that represent that aggregation, through assembling the findings rated according to their quality, and categorising these findings on the basis of similarity in meaning These categories are then subjected to a meta-synthesis in order to produce a single comprehensive set of synthesised findings that can be used as a basis for evidence-based practice. Where textual pooling is not possible the findings may be presented in narrative form.

JBI's inclusive, pluralistic approach to "what counts as evidence"

JBI takes a particular stance on regarding a wide range of information as legitimate evidence for practice; a stance that reflects the evidence interests of clinicians. Whilst the RCT is probably the 'best' approach to generating evidence of effectiveness, nurses, medical practitioners and allied health professionals are concerned with more than cause and effect questions, and this is reflected in the wide range of research approaches utilised in the health field to generate knowledge for practice. Evidence in its most generic sense has been defined as being 'the available facts, circumstances, etc. supporting or otherwise a belief, proposition, etc. or indicating whether a thing is true or valid.'⁶ The concept of evidence plays a key role in clinical decision making and health professionals draw on beliefs and understandings that are seen as rational and based on adequate evidence, even if that evidence is insufficient to ground knowledge.

The meta-analysis of the results on quantitative intervention studies is a well-established process by which data from a number of empirical studies is combined and analysed afresh to provide evidence for clinical decision making. Meta-analysis is strictly limited to quantitative data and is supported by advanced statistical techniques;⁷ furthermore, the Cochrane Collaboration has developed an approach to meta-analysis⁸. Suri⁹ asserts, however that "some form of synthesis is essential to enhance the practical value of qualitative research in policy making" (p.1) and Pearson et al⁴ argue that expert opinion can be legitimately regarded as the "best available" evidence when no evidence generated through rigorous research exists.

JBI Levels of Evidence and Grading of Recommendations

The "FAME" Scale

When clinicians and care staff make decisions about patient care, they are interested in accessing different types of evidence. JBI takes the view that the evidence interests of those who deliver care to patients can be best summarised under the four major headings of Feasibility, Appropriateness, Meaningfulness and Effectiveness – the FAME" scale of evidence.

Feasibility relates to evidence about the extent to which an activity or intervention is practical.

Appropriateness relates to evidence about the extent to which an activity or intervention is Ethical or culturally apt.

Meaningfulness relates to evidence about the personal opinions, experiences, values, thoughts, beliefs or interpretations of clients and their families or significant others.

Effectiveness relates to evidence about the effects of a specific intervention on specific outcomes.

JBI Levels of Evidence

The Joanna Briggs Institute and its collaborating centres and Evidence Translation Groups currently assign a level of evidence to all conclusions drawn in JBI Systematic Reviews. The JBI Levels of Evidence are:

Level of Evidence	Feasibility F (1-4)	Appropriateness A (1-4)	Meaningfulness M (1-4)	Effectiveness E (1-4)	Economic Evidence EE (1-4)
1.	Metasynthesis of research with unequivocal synthesised findings	Metasynthesis of research with unequivocal synthesised findings	Metasynthesis of research with unequivocal synthesised findings	Meta-analysis (with homogeneity) of experimental studies (eg RCT with concealed randomisation) OR One or more large experimental studies with narrow confidence intervals	Metasynthesis (with homogeneity) of evaluations of important alternative interventions comparing all clinically relevant outcomes against appropriate cost measurement, and including a clinically sensible sensitivity analysis
2.	Metasynthesis of research with credible synthesised findings	Metasynthesis of research with credible synthesised findings	Metasynthesis of research with credible synthesised findings	One or more smaller RCTs with wider confidence intervals OR Quasi-experimental studies (without randomisation)	Evaluations of important alternative interventions comparing all clinically relevant outcomes against appropriate cost measurement, and including a clinically sensible sensitivity analysis
3.	Metasynthesis of text/opinion with credible synthesised findings One or more single research studies of high quality	Metasynthesis of text/opinion with credible synthesised findings One or more single research studies of high quality	Metasynthesis of text/opinion with credible synthesised findings One or more single research studies of high quality	Cohort studies (with control group) Case-controlled Observational studies (without control group)	Evaluations of important alternative interventions comparing a limited number of appropriate cost measurement, without a clinically sensible sensitivity analysis
4.	Expert opinion	Expert opinion	Expert opinion	Expert opinion, or physiology bench research, or consensus	Expert opinion, or based on economic theory

 

JBI Grading of Recommendations

The Institute also develops and publishes summaries of one or more systematic reviews as "Best Practice Information Sheets for Health Professionals" (BPIS). Although these BPIS do not follow the usual format of clinical practice guidelines, their development follows the processes recommended by the Appraisal of Guidelines Research and Evaluation (AGREE) Collaboration and include recommendations for practice. Recommendations are graded on the basis of both the level of evidence that underpins them and their clinical appropriateness/relevance. JBI uses the following Grades of Recommendations:

Grade of Recommendation	Feasibility	Appropriateness	Meaningfulness	Effectiveness
A.	Strong support that merits application	Strong support that merits application	Strong support that merits application	Strong support that merits application
B.	Moderate support that warrants consideration of application	Moderate support that warrants consideration of application	Moderate support that warrants consideration of application	Moderate support that warrants consideration of application
C.	Not supported	Not supported	Not supported	Not supported

The JBI System for the Unified Management of the Assessment and Review of Information (SUMARI)

JBI SUMARI (System for the Unified Management of the Assessment and Review of Information) aims to assist health and other researchers and practitioners to conduct systematic reviews of evidence of Feasibility, Appropriateness, Meaningfulness and Effectiveness and to conduct economic evaluations of activities and interventions. The package consists of five modules:

Module 1 Comprehensive Review Management System (CReMS)

This module is designed to accommodate the planning, monitoring and management of a systematic review. It includes the protocol, reviewer information, bibliographical information and time prompts and incorporates the ability to import data from Endnote. CReMS has the capacity to generate publishing house standard systematic review reports and is suitable for researchers conducting systematic reviews that comply with the Cochrane Collaboration approach and for researchers who wish to incorporate other forms of evidence/data through the use of other SUMARI modules.

Module 2 Qualitative Assessment and Review Instrument (QARI)

This module is designed to facilitate critical appraisal, data extraction and synthesis of the findings of qualitative studies.

Module 3 Meta Analysis of Statistics Assessment and Review Instrument (MAStARI)

This module is designed to conduct the meta-analysis of the results of comparable RCTs, cohort, time series and descriptive studies using a number of statistical approaches.

Module 4 Narrative, Opinion and Text Assessment and Review Instrument (NOTARI)

This module is designed to facilitate critical appraisal, data extraction and synthesis of expert opinion texts and of reports.

Module 5 Analysis of Cost, Technology and Utilisation Assessment and Review Instrument (ACTUARI)

This module is designed to facilitate critical appraisal, data extraction and synthesis of economic data.

JBI Methods Groups

To maintain, monitor and improve the quality of reviews published in the JBI Database of Systematic Reviews and the JBI COnNECT databases; and to evaluate and improve the JBI-SUMARI software package; JBI is advised by a Scientific Editor of the JBI Systematic Review Library and by four expert methods groups.

The Scientific Editor: JBI Systematic Review Library is appointed by the Executive Director for a period of 3 years, and is eligible for reappointment. The Scientific Editor: JBI Systematic Review Library is responsible to the Executive Director for the scientific leadership of the Institute's systematic review program and should be a person of high academic standing and experienced in monitoring the quality of systematic review protocols and reports.

Principal responsibilities:

• Reporting on the Institute's performance in relation to the methodological rigour of systematic review protocols and reports to the Executive Director
• Providing advice and support to the Executive Director and JBI staff on matters relating to systematic reviews
• Promoting the Institute including, where appropriate, attending annual face-to-face Committee of Directors meetings, representing JBI at annual NICEBIRG meetings and ensuring that the Institute is represented at other appropriate meetings, committees and forums
• Such matters as the Scientific Editor agrees to accept on referral from the Executive Director

Dr Rick Wiechula from the Discipline of Nursing, School of Population Health and Clinical Practice at the University of Adelaide was appointed as Scientific Editor in April 2006.

Four focused Methods Groups made up of experts from both within and without the JBI support the Scientific Editor and the evidence review program within JBI. The four Methods Groups are:

• Qualitative Evidence Review Methods Group;
• Statistical Evidence Review Methods Group;
• Economic Evidence Review Methods Group; and
• Evidence Utilisation Methods Group.

The aims of JBI Methods groups are to:

• Provide advice to JBI on methodology and methods
• Provide advice to JBI on JBI evidence review software
• Assist the Scientific Editor to monitor the quality of systematic review protocols and reports prepared within the Collaboration.

Qualitative Evidence Review Methods Group

Terms of Reference:

The Qualitative Evidence Review Methods Group advises the JBI on the systematic review of qualitative evidence through:

• acting as expert reviewers of systematic review protocols and systematic review reports submitted to JBI;
• considering feedback from users of JBI qualitative review software and advising JBI on the improvement of software;
• endorsing JBI user guides on qualitative evidence review annually; and
• contributing to the scientific rigour of JBI reviews of qualitative evidence.

The Qualitative Evidence Review Methods Group elects its own chair who shall not be a member of the Joanna Briggs Institute or of the academic staff of the University of Adelaide. All elected or appointed members shall be elected or appointed for a period of two years and shall be eligible for re-election or re-appointment. The Qualitative Evidence Review Methods Group shall meet, via teleconference, not less than three times each year.

Composition:

• Up to ten members experienced in qualitative research and in the synthesis of qualitative evidence, appointed by the Executive Director;
• Three Collaborating Centre Representatives; and
• A Member of JBI staff.

Statistical Evidence Review Methods Group;

Terms of Reference:

The Statistical Evidence Review Methods Group advises the JBI on the systematic review of quantitative evidence through:

• acting as expert reviewers of systematic review protocols and systematic review reports submitted to JBI;
• considering feedback from users of JBI quantitative review software and advising JBI on the improvement of software;
• endorsing JBI user guides on quantitative evidence review annually; and
• contributing to the scientific rigour of JBI reviews of quantitative evidence.

The Quantitative Evidence Review Methods Group elects its own chair who shall not be a member of the Joanna Briggs Institute or of the academic staff of the University of Adelaide. All elected or appointed members shall be elected or appointed for a period of two years and shall be eligible for re-election or re-appointment. The Quantitative Evidence Review Methods Group shall meet, via teleconference, not less than three times each year.

Composition:

• Up to ten members experienced in quantitative research and in the synthesis of quantitative evidence, appointed by the Executive Director;
• Three Collaborating Centre Representatives; and
• A Member of JBI staff.

Economic Evidence Review Methods Group

Terms of Reference:

The Economic Evidence Review Methods Group advises the JBI on the systematic review of Economic evidence through:

• acting as expert reviewers of systematic review protocols and systematic review reports submitted to JBI;
• considering feedback from users of JBI Economic review software and advising JBI on the improvement of software;
• endorsing JBI user guides on Economic evidence review annually; and
• contributing to the scientific rigour of JBI reviews of Economic evidence.

The Economic Evidence Review Methods Group elects its own chair who shall not be a member of the Joanna Briggs Institute or of the academic staff of the University of Adelaide. All elected or appointed members shall be elected or appointed for a period of two years and shall be eligible for re-election or re-appointment. The Economic Evidence Review Methods Group shall meet, via teleconference, not less than three times each year.

Composition:

• Up to ten members experienced in Economic research and in the synthesis of Economic evidence, appointed by the Executive Director;

• Three Collaborating Centre Representatives; and
• A Member of JBI staff.

Evidence Utilisation Methods Group

Terms of Reference:

The Evidence Utilisation Methods Group advises the JBI on evidence utilisation through:

• considering feedback from users of JBI evidence utilisation software and advising JBI on the improvement of software;
• endorsing JBI user guides on evidence utilisation annually; and
• contributing to the scientific rigour of JBI evidence utilisation programs.

The Evidence Utilisation Methods Group elects its own chair who shall not be a member of the Joanna Briggs Institute or of the academic staff of the University of Adelaide. All elected or appointed members shall be elected or appointed for a period of two years and shall be eligible for re-election or reappointment. The Evidence Utilisation Methods Group shall meet, via teleconference, not less than three times each year.

Composition:

• Up to ten members experienced in evidence utilisation, appointed by the Executive Director;
• Three Collaborating Centre Representatives; and
• A Member of JBI staff.

The Table below demonstrates how the Scientific Editor and the Methods Groups relate to the JBI program structure.