Good Quality Systematic Review and/or Good Quality Evidence: How to tell the difference

This is one of the most frequent questions I get in training sessions. So I'm going to use it to try out a new Q & A feature. Let me know if you like it.

Q: If I find a good quality systematic review, I’ll have good quality evidence to answer my clinical question, right?

A: Not necessarily.

Now before you say, “Spoken like a researcher (or politician or used car salesman or…) let me be clear:

A good quality systematic review is necessary but not sufficient to get to good quality evidence.

You see we’re talking about two different things.

One is the systematic review itself. The systematic review is the process used to find, gather, evaluate, and summarize all the available information on a specific topic in a transparent and reproducible way. Good ones always include a detailed description of the methods (like a specific recipe, production manual, etc.)  A good quality systematic review is the best way we currently have to assemble all the information with the least amount of error in finding and synthesizing that information.  In other Q&A topics we’ll talk about the challenges of finding all the studies that contribute to a balanced and unbiased look at all the information. We'll also talk  about the methods to evaluate and combine the information. But for now, let’s just consider that if all the necessary steps are followed and described, you are on your way to identifying a good quality systematic review. You can read the Institute of Medicine’s (IOM) or the Agency for Healthcare Research and Quality’s (AHRQ) recommended methods if you want more information now.  Let’s call the good quality systematic review a rigorous systematic review to get away from the confusing label of quality for both ideas.

So if you have a rigorous systematic review, why don’t you have good quality evidence to answer your question?  Put simply, the review is the process and the evidence is the product. Even if you follow the rigorous process, you can still end up with no evidence or low quality evidence. How you ask? What if you have a great recipe from a noted chef? You follow it exactly. But your tomatoes aren’t fresh, your olive oil is a little rancid, your garlic has dried up… Do you get the idea? The quality of the evidence is determined by the quality of the underlying studies or reports of the information you are mixing in, in addition to the way you mix it.  And sometimes, although you search many databases, conference proceedings, and even ask those familiar with the field, you still won’t find any studies. No ingredients for your pizza.

This issue is examined more carefully in the recently published JAMA Users’ Guide to the Medical Literature on How to Read a Systematic Review and Meta-analysis and Apply the Results to Patient Care. Murad and colleagues give their reason for a need to update the 1994 version as a realization of the need to clarify this exact issue. They label the first concept, what I've called the rigor of the systematic review, as credibility of the review. And they label the second issue, the quality of the evidence to be synthesized, as the confidence in estimates.  Here’s the link, though the article is not open access.

And since it’s haying season out here in the heartland, I’ve also created a “simple country evidencedoc” explanation just for fun. You’ll find it here.

So to summarize the A to the Q – a rigorous systematic review is required but can’t guarantee that you’ll find good quality evidence to answer your clinical question.

TheEvidenceDoc 2014