To Think Like a Statistician

Alexis de Tocqueville: "Without comparisons to make, the mind does not know how to proceed."

MammaPrint is a genetic testing device designed to tell women diagnosed with stage 1 breast cancer whether their cancer has a high or low risk of spreading. It was designed to replace clinical assessments that were perhaps too quick to recommend an arduous course of chemotherapy. In 2016 the MammaPrint company ran a clinical trial to see if it was safe for women at low risk according to MammaPrint, but high risk clinically, to skip chemo.

Fifteen hundred stage 1 breast cancer patients, deemed at low risk according to MammaPrint but high risk according to clinical assessment, were divided into two groups: half of them had chemotherapy while the other half followed MammaPrint's advice and went without. Here were the results five years later:

Women who had chemo: 4.1% cancer spread

Women who didn't: 5.6% cancer spread

Now we have a serious problem of comparison. Is it safe to follow MammaPrint's advice and avoid chemotherapy? The results look a little worrisome, with the avoiders having 1.5% greater spread. But of course, one of the two spread rates had to come out larger, so maybe MammaPrint had just been unlucky.

The MammaPrint experiment was a Randomized Clinical Trial (RCT), the 20th century's most important advancement in experimental methodology. Human health has had a centuries-long history of "miracle cures", wandering in the weeds and getting nowhere.

A RCT escapes the weeds by following three requirements:

The proposed new treatment is compared head-to-head against a plausible competitor.

Patients are assigned randomly to receive either the new treatment or the competitor.

The new treatment must perform significantly better than the competitor.

Just what "statistically significant" means is discussed in Chapter 4 of my book, "The Gold Standard of Comparisons", along with causality, the conclusion that the new treatment caused the patients to do better. Any drug on your medicine shelf right now has emerged victorious from at least one RCT. (MammaPrint was approved and is now in use.)

Medical hokum is always waiting at the door, making RCTs as crucial for what they prevent as what they approve. Krebiozen was introduced in the early 1950s as a cancer treatment made from the blood of Argentinean horses that had been injected with a secret bacterium. Stories of miracle cures inspired desperate patient advocacy groups that included members of Congress, who pressured the FDA to investigate. No medical benefit was discovered but that didn't quell fierce support from Krebiozen believers. Eventually it turned out to be a money-making scheme that netted millions for two brothers from Yugoslavia who were acquitted of deliberate fraud in a 1964 trial. After the verdict, a woman wearing a shirt that read "I need Krebiozen to live" kissed the foreman of the jury.