The Cochrane Library published a new systematic review this week that got a lot of press. Many of the headlines about it read, “Routine physicals don’t save lives”. I was surprised by the headlines, but not for the same reason as those who believe that annual physicals are an effective way to catch disease early. Don’t get me wrong. I am not a proponent of annual visits for everyone to get batteries of non–specific lab and screening tests that are more likely to produce false positive findings than to detect real disease. No, I was surprised at the certainty in the headlines. Do we now have good evidence that a periodic visit with a physician, even for targeted screening and counseling, doesn’t provide any survival benefit for any group of people? So, of course, I had to read the full Cochrane review. I would recommend that those reporting on its findings do the same.
Let’s start by showing how to approach a systematic review (SR). Of course we want to be certain the authors followed all the rules of systematic reviews, like unbiased search and retrieval of eligible studies. But even more basic, let’s start with what the systematic review really compared. To do this, we’ll follow an approach called PICO, which looks at:
P = People in the study
I = Intervention in the study
C= Comparison or control group of the study
The SR authors started by limiting the P, the people, in eligible studies to adults and excluding any studies specific to adults over 65 years of age and those adults with known risk factors or known disease.
The authors of the SR made the decision to limit study designs to only randomized controlled trials (RCTs). I don’t necessarily agree with this decision, but that ‘s the subject for another post. For the randomized trials, they name the I, the Intervention, of interest as “general health checks”, which they define as “screening for more than one disease or risk factor and in more than one organ system, whether preformed only once or repeatedly.” This is important. Already we are not necessarily examining the annual or even periodic health visit.
The authors found 14 eligible RCTs that looked at a lot of different O or Outcomes. They picked death as the primary Outcome of interest for the SR. Nine RCTs had that data available, though not all 9 were designed to evaluate death as an endpoint.
So let’s look at what the SR authors say about the Interventions - health checkups - in each trial. Remember, an RCT should randomly assign subjects to either get the Intervention or to a Control group that does not.
Here’s the rub. The intervention in the studies was not getting a health check; it was getting an invitation to get a health check. And only 2 of the trials had any measurement of the difference between how often Intervention and Control groups actually got the health checks. That’s right, control groups could also get the health check. So how did the Intervention and Control groups differ? The best measurement of that was from a Kaiser study which reported that during the 16 years of their annual invitations for check up, the Intervention subjects got an average of 6.8 check-ups while the Control group got an average of 2.8 check ups. So it looks like the invitations increased uptake a bit, roughly 2.5 times. But it means the comparison group got health checks, too, just not as frequently. What about the other studies?
For many, the control groups were “usual care” which was unstudied and unmeasured. How often did their personal physicians recommend regular health check ups and how often did the subjects accept? Who knows?
The RCTs had a lot of other measurement challenges for the comparison. Some invited both Intervention and Control groups to get the study health check after some years had passed, and then compared mortality several years after that. Registries were used for mortality data, and registry completeness is affected by patient movement out of the registry area. The Kaiser trial acknowledges mortality measurement gaps for those leaving the state of California and estimated between 8 and 18% of their study group was lost and without mortality measurement.
Hmm. Are you feeling less confident in the RCT studies? These are not the trials you imagine when you think of RCT, random assignment of half the subjects to get an annual check and the other half does not get the check and then measure the difference in outcomes. These trials seem more like observational studies with similar problems in measurement error and bias.
The intervention was weak at best, an invitation to get one or more health checks offered in a variety of settings and administered by a variety of health professionals. Except for the Kaiser trial that offered annual checks, most of the studies offered only 2 or 3 total check ups, and in some studies one of those offers was also extended to the control group. And both Intervention and Control subjects may have had check ups outside the study, through their personal physicians. As for outcome, how completely mortality was measured is unknown or unreported for most studies. Besides the Kaiser trial, only one other had completeness of follow-up for mortality reported in the SR.
How confident are you in the evidence? Using the PICO method, we find that differences between intervention and control group in terms of getting a health check are not very distinct and are poorly measured. The outcome of interest may or may not be accurately recorded for all subjects.
The SR authors’ first sentence of their summary in the full report states, “We did not find an effect of general health checks on total or cause specific mortality.” That’s an accurate reflection of their findings from this limited evidence. Other, more sensational, statements seem to overstate the evidence they reviewed.
Is there other, better evidence of the effectiveness or ineffectiveness of periodic health exams? I honestly don’t know. We could search for other systematic reviews using the methods shared in the previous post, but that would also be the subject of another post.
Remember, not having evidence of the effectiveness of periodic health exams is not the same thing as having good evidence they are ineffective.
At least now you’ve learned how to approach a systematic
review using PICO to see if it addresses the question you are interested
in. I promise more lessons in future posts.
Krogsbøll LT, Jørgensen KJ, Grønhøj Larsen C, Gøtzsche PC. General health checks in adults for reducing morbidity and mortality from disease. Cochrane Database of Systematic Reviews 2012, Issue 10. Art. No.: CD009009.