Explanation of
Medical Literature Ratings
(Ratings “1a” through “11c” noted under summary of each study)
Ranking by Type of Evidence:
(click on links to go to explanation)
STUDIES
1. Systematic Review/Meta-Analysis
2. Controlled Trial – Randomized (RCT) or Controlled
3. Cohort Study - Prospective or Retrospective
OTHER:
6. Nationally Recognized
Treatment Guideline (from guidelines.gov)
10. Conference
Proceedings/Presentation Slides
11. Case Reports and Descriptions
Ranking by Quality within Type of Evidence:
(click on links to go to explanation)
Ranking
by Type of Evidence
1. Systematic
Review/Meta-Analysis
Systematic
Reviews: Written by reviewers who use explicit and
rigorous methods to identify, critically appraise, and synthesize relevant
studies from the published medical research.
They use the process of systematically locating, appraising and
synthesizing evidence from scientific studies in order to obtain a reliable
overview. The function of a systematic
review is: 1) to summarize the literature and 2) to provide new information
that may not be readily apparent from individual studies where the effects are
small, but become apparent in when the data from many studies are pooled
together. Example: Cochrane Database of Systematic Reviews.
Meta-analysis: A
type of systematic review that is an
overview and also uses quantitative methods to summarize the results. A quantitative method of combining the
results of independent studies (usually drawn from the published literature)
and synthesizing summaries and conclusions which may be used to evaluate
therapeutic effectiveness, plan new studies, etc., with application chiefly in
the areas of research and medicine. Any
study with the Level 1 ranking in ODG must have been accepted for publication
in a peer reviewed journal, and that journal must be one of the journals
accepted for inclusion in MEDLINE® by the National Library of
Medicine. For this Journal Selection
Criteria, see www.nlm.nih.gov/pubs/factsheets/jsel.html. Unpublished studies, or studies in magazines
that do not publish original research, would not receive this ranking.
2. Controlled Trial – Randomized (RCT) or
Controlled
These are analytical experimental studies, where variables can be better controlled on a prospective basis. In a RCT (Randomized Controlled Clinical Trial), a group of patients is randomized into an experimental group and a control group. These groups are followed up for the variables/outcomes of interest. Advantages: Unbiased distribution of confounders; Blinding more likely; Randomization facilitates statistical analysis. Disadvantages: Expensive: time and money; Volunteer selection bias; Ethically problematic at times. Any study with the Level 2 ranking in ODG must have been accepted for publication in a peer reviewed journal, and that journal must be one of the journals accepted for inclusion in MEDLINE® by the National Library of Medicine. Unpublished studies, or studies in magazines that do not publish original research, would not receive this ranking.
3. Cohort Study - Prospective or
Retrospective
Analytical observational studies involving identification of
two groups (cohorts) of patients, one which did receive the exposure of
interest, and one which did not, and following these cohorts forward for the
outcome of interest. Advantages: Ethically safe; Subjects
can be matched; Can establish timing and direction of events; Eligibility
criteria and outcome assessments can be standardized; Administratively easier
and cheaper than RCT. Disadvantages:
Controls may be difficult to identify; Exposure may be linked to a hidden
confounder; Blinding is difficult; Randomization
not present; For rare disease,
large sample sizes or long follow-up necessary. Any study with the
Level 3 ranking in ODG must have been accepted for publication in a peer
reviewed journal, and that journal must be one of the journals accepted for
inclusion in MEDLINE® by the National Library of Medicine.
Analytical observational studies
involving identifying groups of patients who have the outcome or treatment of
interest (cases) and quantifying the results.
Ideally, control patients without the same outcome are also tracked,
looking back to see if they had the exposure of interest. (The use of controls would influence the
quality rating of a Case Series.)
Generally, since the minimum ODG quality rating for studies (“c”)
requires at least 10 cases, there must be 10 or more cases for a study to be
classified as a Case Series, and otherwise the article would be classified in
ODG as Case Reports and Descriptions.
Advantages of Case Series: Quick and cheap; Only feasible method for
very rare disorders or those with long lag between exposure and outcome; Fewer subjects needed than cross-sectional
studies. Disadvantages: Reliance on
recall or records to determine exposure status; Confounders; Selection of
control groups is difficult; Potential bias: recall, selection. Any
study with the Level 4 ranking in ODG must have been accepted for publication
in a peer reviewed journal, and that journal must be one of the journals
accepted for inclusion in MEDLINE® by the National Library of
Medicine.
Descriptive (versus analytical) and
observational (versus experimental) studies, written by reviewers who describe current practice as well as relevant
studies from the published medical research, with no attempt to pool the
results analytically. Compared
to Systematic Reviews, an Unstructured Review makes little attempt to quantify
outcomes based on the body of evidence described. Any study with the Level
5 ranking in ODG must have been accepted for publication in a peer reviewed
journal, and that journal must be one of the journals accepted for inclusion in
MEDLINE® by the National Library of Medicine.
6. Nationally Recognized
Treatment Guideline (from guidelines.gov)
Accepted for inclusion in the National Guideline Clearinghouse by the
Federal Agency for Healthcare Research & Quality (AHRQ), which requires
that the guideline recommendations be based on a systematic literature search
and review of scientific studies published in peer reviewed journals, and
revised on a regular basis to maintain currency with new studies.
Treatment guidelines created for use in a specific state in the U.S.,
or for use in a province in Canada, or for use by another governmental entity,
and they have the backing of the respective jurisdictional or governmental
authority.
Other treatment
guidelines. These are typically
national treatment guidelines not accepted in the National Guideline
Clearinghouse, in many cases because the guideline publishers have chosen not
to apply for inclusion (for example, commercial guidelines such as Milliman,
McKesson, InterQual, etc.), or because they are private guidelines created for
use under the terms of a specific health insurance policy (for example, Blue
Cross, Medicare, Aetna, Cigna, United Healthcare, etc.). Since studies by healthcare insurers are generally given a rating of Level 8, they are not characterized in ODG
as among the highest quality references when there are numerous other studies
available. However, when there are
limited studies available with the high quality ratings, it may be necessary to
identify other studies that could provide guidance on a subject. In fact, many of the healthcare insurance
provider structured reviews are very high quality, they represent a thorough
analysis and quantitative weighting of all available evidence on a subject,
including unpublished studies that the insurer may have conducted, and these
healthcare insurance reviews might even rank as Level 1 if they were published in the peer-reviewed literature and
available in MEDLINE®.
Furthermore, the fact that a particular treatment is either covered or
not covered by healthcare insurance should be relevant to coverage decisions in
workers’ compensation.
Medical reference texts, which may represent standards of practice, but
which in and of themselves, are not necessarily evidence based versus consensus
based or based primarily on the personal experiences of the authors.
10. Conference
Proceedings/Presentation Slides
These are studies that have not been published in peer reviewed
journals.
11. Case Reports and Descriptions
Descriptive articles published in the peer reviewed journals covering
individual cases, and lacking any comparisons to controls. Generally, since the minimum ODG
quality rating for studies (“c”) requires at least 10 cases, there must be 10
or more cases for a study to be classified as a Case Series, and otherwise the
article would be classified in ODG as Case Reports and Descriptions. These
articles were not included in the evidence base for any treatment guidelines
except for the Council on Chiropractic Guidelines for Practice Parameters
(CCGPP) chiropractic practice guidelines.
Ranking
by Quality within Type of Evidence:
In evaluating clinical trials ODG has adopted the standards from the "Cochrane Handbook for Systematic Reviews of Interventions," as updated in September 2006. (Higgins, 2006) Specific additional criteria used by ODG include the following:
Sample size: Generally over 300, but at least 100,
depending on other factors below.
Conflict of
interest: Authors and
researchers had no financial interest in the product or service being studied.
Study design: Ideally, blinded. No identifiable bias, including recall bias, confounding factors,
selection bias, compliance bias, non-response bias, or measurement bias. If a
case series, should be a case control series.
Statistical
significance: 99% Confidence
level that the outcomes likelihood ratio will not cross 1.0 (i.e., the p value
is .01).
Sample size: From 20-50 up to 100-300, depending on other
factors below.
Conflict of
interest: Authors and
researchers had no financial interest in the product or service being studied.
Study design: No significant bias, including recall bias,
confounding factors, selection bias, compliance bias, non-response bias, or
measurement bias. If a case series,
should be a case control series.
Statistical
significance: 95% Confidence
level that the likelihood ratio will not cross 1.0 (i.e., the p value is .05).
Sample size:
Generally under 20-50, depending on other factors below, but no less
than 10.
Conflict of
interest: Authors and
researchers may have had some financial interest in the product or service
being studied, even if the sample size was large.
Study design: Some obvious bias, including recall bias,
confounding factors, selection bias, compliance bias, non-response bias, or
measurement bias.
Statistical
significance: Does not meet
the 95% Confidence level that the likelihood ratio will not cross 1.0 (i.e.,
the p value is .05).
Link
between evidence and recommendations
ODG Treatment is being updated quarterly on the Web. The Contents page indicates the last date
updated for each chapter. The hard copy
version is published once a year, but this is not recommended since it does not
link into the actual studies, and it is not as current as the Web version.
The heart of each
chapter in ODG Treatment is the "Procedure Summary", which
provides a concise synopsis of effectiveness, if any, based on existing medical
evidence, hyper-linked directly into the studies on which they are based, in
abstract form, which have been ranked, highlighted and indexed. The "Treatment Planning" section
identifies the ideal treatment plans that may be followed after illness or
injury, based on the "Procedure Summary". "Codes for Automated-Approval" maps procedure codes to
ICD-9 diagnosis codes based on the ideal treatment protocol, with a field for
“maximum occurrences”, for auto-approval of charges that meet the guideline.
For example, in the
Low Back chapter, under Fusion, it says, "Not recommended in the absence
of fracture, dislocation, or instability", so the Treatment Protocol does
not include fusion. Same for IDET, facet
injections, etc., etc. Under Epidural
injections, it says, "Recommended as an option prior to surgery when there
are radicular signs… and the number of injections should be limited to
two...", so the Treatment Protocol for "With Radiculopathy"
includes 2 ESI's, and the Codes for Auto Approval includes CPT code 62311
(Epidural steroid injection) 2 times for ICD9 722.x (Intervertebral disc
disorders).
This effort to
translate the evidence into specific auto-authorization protocols is unique,
for pre-approval of treatment plans and triage of claims management. Of course, most cases will not meet this
ideal protocol, and that is where the many other listings in the Procedure
Summary come into play.
In a recent pilot
use of these Codes for Auto Approval reduced medical costs by 64%, cut lost
days by 69%, minimized treatment delays for injured workers, and drew
considerable praise from providers.
(Ohio ODG Pilot, Comp Management, 2005)
##########################################
Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of
Interventions 4.2.5. In: The Cochrane Library, Issue 3, 2005. Chichester, UK:
John Wiley & Sons, Ltd. September 2006.
6. ASSESSMENT OF
STUDY QUALITY
6.0 Quality
assessment of studies
Quality assessment
of individual studies that are summarized in systematic reviews is necessary to
limit bias in conducting the systematic review, gain insight into potential
comparisons, and guide interpretation of findings. Factors that warrant
assessment are those related to applicability of findings, validity of
individual studies, and certain design characteristics that affect
interpretation of results. Applicability, which is also called external
validity or generalize-ability by some, is related to the definition of the key
components of well-formulated questions outlined in section 4. Specifically,
whether a review's findings are applicable to a particular population,
intervention strategy or outcome is dependent upon the studies selected for
review, and on how the people, interventions and outcomes of interest were
defined by these studies and the authors (reviewers).
6.1 Validity
In the context of a
systematic review, the validity of a study is the extent to which its design
and conduct are likely to prevent systematic errors, or bias. An important
issue that should not be confused with validity is precision. Precision is a
measure of the likelihood of chance effects leading to random errors. It is
reflected in the confidence interval around the estimate of effect from each
study and the weight given to the results of each study when an overall
estimate of effect or weighted average is derived. More precise results are
given more weight.
6.2 Sources of
bias in trials of healthcare interventions
There are four
sources of systematic bias in trials of the effects of healthcare: selection
bias, performance bias, attrition bias and detection bias.
6.3 Selection
bias
Participants and those who recruit should remain unaware of
next assignment in sequence. Empirical research has shown that lack of
allocation concealment is associated with bias. For that reason trials should
use approaches such as allocation by a central office unaware of subject
characteristics, pre-numbered or coded identical containers which are
administered serially to participants, or an on-site computer system combined
with allocations kept in an unreadable file that can be accessed only after the
characteristics of enrolled participants have been entered.
6.4 Performance
bias
This refers to systematic differences in the care provided
to the participants in the comparison groups other than the intervention under
investigation. To protect against unintended differences in care and placebo
effects, those providing and receiving care can be "blinded" so that
they did not know the group to which the recipients of care have been
allocated.
6.5 Attrition
bias
This refers to systematic differences between comparison
groups in the loss of participants from the study. The study should consider
how losses of participants (withdrawals, dropouts and protocol deviations) are
handled.
6.6 Detection
bias
This refers to systematic differences between the comparison
groups in outcome assessment.
Rating:
1a