It's time for the last domain for upgrading.

I've covered the most straightforward and most common reasons for upgrading evidence from well-done observational studies:

  1. large effect
  2. dose-response

The third and last domain is a little trickier to explain to those without methodology background. It has to do with confounding, which itself may need explaining. Confounding occurs when something is associated with both the intervention and the outcome of interest. For example, a study I published in the early 90s sought to evaluate the impact of a specific herbicide exposure on a specific eye disease. An earlier, unpublished study looked at all eye diseases in the cohort and found an association with long term exposure to the herbicide and development of cataracts. That study did not control for age. Increasing age was a confounder, since cataracts are more common as we age and long term exposure is also associated with increasing age.

Confounding is a known problem for all studies and especially for observational designs. Randomization is used in trials in an attempt to prevent confounding (which may or may not be successful, but that's another topic for another time). In observational designs, matching is used to prevent confounding. If we control for enough of the known confounders for the outcome, like age which is a common one, and sex, race, SES, or educational level if relevant, we are also hoping to match for confounders we don't know about. Confounding can also be adjusted for in the analysis through stratification or multivariate analysis.

Now that you have a bit of an understanding of confounding, here are the rules for upgrading.

Did you find an effect, even though all the known confounders should have reduced or eliminated an effect? If so, consider upgrading.

A common example is the study of an intervention where the sickest patients got the intervention and yet they improved more than the comparison group. The confounding factor of sicker patients should have reduced the impact of the intervention and yet it did not. So you would upgrade confidence in the evidence.

Of course, consider the converse, that is, did you fail to find an effect even though the known confounding factors should have increased the effect? Again you may decide to upgrade confidence in the evidence.

So there are 3 reasons to upgrade:

  1. Large effect
  2. Dose-response
  3. All known confounding should be working against the direction of the observed effect

That's it for upgrading in this very brief overview.

You know where to find more detail - GRADE Handbook

TheEvidenceDoc December 1, 2017



In addition to large effect, GRADE uses another of the causal criteria from Bradford Hill to upgrade the evidence. The second domain for upgrading recognizes that larger exposures should generally lead to increases in occurrence of the outcome. This is known as a dose-response. As the dose, or exposure, increases so do more cases of the disease or condition of interest. 

Yesterday's example of the association of smoking with development of lung cancer provides another example of this effect. Not only is the observed effect of smoking on lung cancer strong, but also includes a dose-response with increasing incidence of lung cancer as people smoke more cigarettes per day.

This provides an opportunity to explain how upgrading works. Since the effect of smoking is so strong, (remember from yesterday that current data shows a 25x increased risk of lung cancer in smokers compared to non-smokers) the evidence would be upgraded two levels from the starting of low confidence in the evidence. This increases the rating to high confidence.  When you add the impact of a dose response, you cannot raise the evidence rating above high. Still, you would note the dose response on the evidence profile. We'll take look at a couple of evidence profiles later in the series.

So there are two rather straight-forward reasons to upgrade the evidence rating for evidence derived from observational studies:

  1. Upgrade for strong effect
  2. Upgrade for dose - response

Remember the observational studies must be well designed and well conducted to be eligible for upgrading.

Tomorrow I'll finish the section on upgrading with the final domain.

As always, for more detail check out the the GRADE Handbook


TheEvidenceDoc November 29, 2017  


Congratulations. You've made it through all five domains for downgrading the evidence from RCTs using the GRADE approach. Now it's time to learn about rating up the quality.

First some details. We are still talking about rating evidence for interventions. There are some differences in rating evidence for questions about prognosis, for example, and we'll cover those later.

So you are still looking at the evidence for your PICO question for an intervention. But now you are evaluating evidence from observational study designs.

You may remember that RCTs are the preferred design for a fair test of an intervention. But for whatever reason, you have observational studies providing evidence.

Observational study designs have limitations for evaluating clinical effectiveness. In observational studies such as cohort and case-control studies, the population of interest generally self-selects the intervention, making it necessary to match or adjust for confounding factors that could impact the results. Because of this, GRADE starts the evidence rating level for observational studies at Low rather than the High rating where RCT studies begin. 

Yet it is still possible for well designed and conducted observational study designs to contribute high quality evidence. There are 3 domains for rating up the quality of evidence.

Today we'll start with the most common reason for rating up the quality of evidence. It is a large effect.

To begin, only observational studies that have no important threats to validity are eligible for upgrading. So the studies must be well-designed and well conducted and have minimized the known risks to bias for observational study designs. This means that even very good observational studies will start at Low, but they will be eligible for upgrading.

The most obvious example of observational studies that could be upgraded for for large effect would be studies of the relationship between smoking and lung cancer. Even the early studies found large effects on lung cancer incidence from smoking. Recent CDC data show that in the 1960s, smokers were 12 times more likely to get lung cancer than non-smokers and in 2010 that had increased to 25 times more likely. These aren't just large, they are huge effects. GRADE recommends upgrading evidence by one level if the exposure doubles the effect (increases by 2 times) and upgrading by 2 levels if there is an increase of 5 times. This means the studies on smoking and lung cancer would be upgraded from a rating of Low to a rating of High.

Note that GRADE issues a caution when the outcome is subjectively measured, such as assessments of pain. Remember that observational study designs are not blinded and can be impacted by patient perception or belief of benefit.

Studies that find large effects even when subject to the limitations of observational design provide greater confidence that the effects are real. Hence GRADE allows for upgrading of that evidence by one or two levels depending on the strength.

As always, for more information you can consult the GRADE Handbook

TheEvidenceDoc November 28, 2017