From Cherry Blossoms to Cherry Picking; How to spot reviews that are not evidence based

It's cherry blossom time in our nation's capital.  I visualize the puffy, pink petals and smell the sweet fragrance, before my thoughts shift to summer in Michigan tasting cherry fruits so juicy they drip down my chin. Mixed with all these pleasant memories is an image of cherry-picking, or selecting only the ripest, juiciest fruits from the trees.

What does cherry picking have to do with clinical epidemiology and evidence based medicine? The phrase is used to describe a practice of selecting, or "cherry picking" studies that support a prevailing belief or an individual's personal belief. It represents a practice that is counter to the scientific method of gathering all the evidence, which is the foundation for evidence based medicine. All studies relevant to your clinical question should be gathered, critically evaluated, and summarized, independent of their findings and conclusions.

This cherry picking of evidence is different from the selection of studies based on their relevance to your clinical question and the quality of the research.  So it's not selecting studies for a review that is the problem, it's HOW the studies are selected.

Let's say you are interested in whether or not tight control of blood glucose in very sick patients in a hospital intensive care unit leads to better survival. To answer this clinical question,  a reviewer would not select studies of blood glucose control in otherwise healthy diabetics living at home. Those studies would not help answer a question about caring for ICU patients. You are looking for a review of studies that has a good chance of answering your question. So in order to evaluate whether or not evidence based methods were followed in a review of tight control in ICU patients, you first look for clues that the review authors had a focused question that is close to one you are asking. The review question should specify the Patients or Population of interest (very sick patients in the ICU, with "sick" being defined), the Intervention (tight control of blood glucose, which should also be defined), the Comparison group (no tight control), and the Outcome (better survival, usually measured in time). These criteria are often referred to as PICO. When a question is well described, then studies identified through comprehensive search can be selected to answer that question. The review should clearly provide you with the criteria for including studies and for excluding them and these criteria should be developed before the selection process  begins.  Studies must be selected independently of the results, and solely on whether or not the studies were designed and conducted to answer these questions. This reduces error, like the error from cherry-picking studies for data supporting a conclusion the reviewer wants or expects to find, and from ignoring studies where the results differ from those expectations.

Evidence based methods should also use another selection process. This one may be harder to evaluate for the non-scientist, but again you are looking for selection that occurs regardless of the study findings, but because of the study methods. This selection is based on quality of the studies. You don't want to include data from very poor studies, because that data is likely to be compromised and misleading or just plain wrong. Reviewers should do their best to evaluate the quality of the studies and to only include studies without serious error in the design or conduct. When we say design of the study here,  we are considering much more than just the simplistic categorization of studies as experimental (such as RCTs) or observational (such as longitudinal cohort study). Whether or not an experimental design is an appropriate choice is certainly one component of quality. Additionally, the study itself, regardless of whether it's RCT or cohort, or whatever, must be designed and completed so as to minimize error in what it measures. In short, it needs to be a good RCT, or cohort, or whatever kind of study. Unfortunately, there are many examples of poor quality studies, yes, even RCTs, in the published literature, so studies must be evaluated and not accepted on faith. Methodological standards for good quality studies are available, and we'll talk about them in another post.  Evidence based reviews should include assessment for these standards of quality for each study included in the review. Then the review can reject studies if they suffer from such serious errors that the data reported in those studies is compromised.

So picking studies based on their relevance to the clinical question is good science, and picking studies based on their quality is good science, too. Picking studies because the results agree with the belief of the reviewer is bad science.

We'll talk more about other ways to tell if reviews are really evidence based, but for now, practice looking for cherry-picking. And please share questions, comments or discussion points here.

©TheEvidenceDoc April 15, 2013