Are We All Homeopaths Now?

There are two kinds of Complementary and Alternative Medicine (CAM) - the ones that involve actually doing stuff, and the ones that don't.

Things like herbal medicine, chiropractic, and acupuncture could plausibly make someone better, as more than just a placebo, given what we know about physics and chemistry, because they involve physically acting on the body. I don't claim to know whether they do in fact work, but in theory, they could.

Other CAM techniques, however, are just magic. Homeopathy is the best example of this: it cannot work, except as a placebo, unless our understanding of nature is fundamentally wrong. The "active ingredient" in a homeopathic remedy is diluted in water to the point where not a single molecule of it remains (and then diluted more, for good measure). If some mystical "essence" or "energy" can somehow survive in water despite dilution then, logically, all water must contain the essence of pretty much everything. Homeopathy literally involves nothing beyond sugar pills and water

But there's one useful thing about homeopathy: it shines a light on the rest of modern medical science, or rather, it holds up a mirror to it. Unfortunately, the reflection is not as pretty as you'd hope.

These two graphs come from a paper by Shang et al, Are the clinical effects of homoeopathy placebo effects?, which was a major meta-analysis of 110 randomized, placebo-controlled trials (RCTs) of homeopathy. It was published in The Lancet in 2005.

Shang et al 2005 was bad news for homeopathy, because it concluded that "[the meta-analysis] is compatible with the notion that the clinical effects of homoeopathy are placebo effects." - i.e., homeopathy doesn't work. Since its publication the paper has been hotly criticized by homeopaths, and defended by skeptics, with the skeptics generally being right. But it was bad news for conventional medicine too.

These two graphs are funnel plots. Each dot represents a published RCT. Dots to the left of the vertical line are trials where the 'active treatment' did better than the placebo control; the further left, the better. The higher up the dot is, the more "precise" the results of the trial, i.e. the less variability there was in the results. They may not look like much, but they're terrifying.

The top funnel plot shows the 110 published RCTs of homeopathy for various illnesses. The bottom one shows 110 RCTs of "conventional" medical treatments that Shang et al picked out as comparisons. These trials were quasi-randomly selected from a list of over 300,000, such that each one was matched to a homeopathy trial for disease, and number of patients.

You'll notice that the two plots look rather similar - there's a lot of spread, but most of the dots are to the left of the vertical line, meaning that the treatments were better than the placebos. Quite a lot are very far to the left, meaning the treatment worked really well. Very few are on the right.

But homeopathic treatments, by definition, are placebos - they're literally sugar pills. So any trial of homeopathy should have an equal chance of finding it to be better than placebo, or worse. Placebos are placebos. It should be a coin toss, 50/50. In fact, Shang et al found only about 20 trials showing homeopathic placebos to be worse than placebo placebos, and 90 finding they were better.

How can this be? Either homeopathy works, in which case we need to rewrite physics and chemistry, or there is something very wrong with the published literature. I find it easier to believe the latter. But then how could the published literature be so wrong?

Almost certainly, a large part of the answer is publication bias, broadly speaking. If people do a trial and don't get the result they want, they generally either don't write it up for publication; or if they try to, it doesn't get published. Related to this is selective outcome bias: they pick out and write up only those results that do match what they wanted; or they pick out statistical techniques to get the result they wanted, etc.

The plot for homeopathy RCTs is what you get when people study a treatment that doesn't work, but that they believe does work, and publish their findings in a biased way. But the plot for "real" medicines looks disturbingly like that.

In other words, the whole clinical trial literature - all of those RCTs and meta-analyses, published in respectable journals, the ones we rely on to determine what treatment decisions doctors make - could be produced even if all of our treatments were no better than placebos. Like I said, terrifying.

I should stress that this doesn't mean that real medicines are no better than placebos. Shang et al's results are also what you'd see if there's no publication bias in conventional medicine, and the treatments work really well. (The evidence for positive effects in the "real medicine" trials was also somewhat stronger than in the homeopathy trials - the dots were further left - which is reassuring, but the difference was pretty small.) The problem though is that we can't tell - at least not on the basis of the clinical trial literature.

Luckily, there's an answer - mandatory registration of clinical trials. Medical journals or, ideally, governments, can require researchers to publicly announce the details of each trial, and how they plan to analyze the results, before the trial takes place, and require that the final results are made public. The USA has had such a system in place, backed by law, since 2007, and most major medical journals now demand registration.

Sadly, biases still seem to be happening in registered trials. But this doesn't mean the system doesn't work, it just means it should be more strictly enforced, and extended to other countries and, I'd argue, beyond just clinical trials. Either that, or we might as well take up homeopathy.

Shang, A., Huwiler-Müntener, K., Nartey, L., Jüni, P., Dörig, S., Sterne, J., Pewsner, D., & Egger, M. (2005). Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy The Lancet, 366 (9487), 726-732 DOI: 10.1016/S0140-6736(05)67177-2