#BadScienceOfTheDay:

Once again, a journal article claiming extremely high rates of breast cancer overdiagnosis (“up to 48%”) makes big news.

Here’s the problem: it’s a retrospective cohort study which calculated an overdiagnosis % based on a truly bizarre endpoint: the total number of “advanced” versus “non-advanced” breast cancers.

Now in any scientifically valid study, you would use a definition of “advanced” that involves lymph node and/or distant metastases – features that strongly correlate with overall survival. You might even do a secondary analysis with overall survival as an endpoint.

Nope. These clowns defined advanced as “radiographic primary tumor size >= 2cm”… then using a completely invalid endpoint they did a bunch of statistics to come up with a completely invalid conclusion. Garbage in, garbage out.

As if to illustrate the ridiculousness of their own study, the Danish researchers published several overdiagnosis estimates based on different statistical approaches. The two bottom-line numbers were 24.4% and 48.3%…

How can you have any faith in your statistics when they give you two different answers that are completely different? If I tried to sell you a car by saying that, depending on how you measure it, it either has 244 horsepower or 483 horsepower, you’d call me a fraud!

* * *

Here’s the problem with the entire concept of “overdiagnosis”:

1) Overdiganosis is a synthetic endpoint that is effectively a derivative of a derivative. The degree of statistical modeling required to estimate overdiagnosis means that any errors, biases, or design flaws in the starting data-set will be enlarged by orders of magnitude. A tiny change in the statistical assumptions leads to a 2-fold difference in your final “endpoint”.

2) Overdiagnosis is highly un-reproducible within studies, or between studies. If you look at literature reviews of “overdiagnosis in breast cancer”, the published values from top-tier journals ranges from ~5% to ~50%. That’s such a large range it is impossible to apply to real life.

3) Overdiagnosis is neither clinically apparent, nor is it provable or disprovable. No one can single out a patient and definitively prove that they were overdiagnosed, or they weren’t overdiagnosed. In my simplistic clinical mindset, that makes it more like faith healing than scientific medicine.

* * *

The conceptual basis of overdiagnosis makes some sense in some cases. It is easy to imagine a bedridden 90-year-old being diagnosed with some tiny little cancer that probably won’t kill them.

However, the statistical methods used to estimate overdiagnosis percentages are horribly unreliable, and they do a piss-poor job of helping make decisions in real life.

A physician with common sense can avoid scanning or treating the bedridden 90-year-old while still offering care to the rest of the patients, and he doesn’t need to quote an artificial “24% to 48%” number to back up his clinical judgement.

Advertisements