Neuroscience Fails Stats 101?

According to a new paper, a full half of neuroscience papers that try to do a (very simple) statistical comparison are getting it wrong: Erroneous analyses of interactions in neuroscience: a problem of significance.

Here's the problem. Suppose you want to know whether a certain 'treatment' has an affect on a certain variable. The treatment could be a drug, an environmental change, a genetic variant, whatever. The target population could be animals, humans, brain cells, or anything else.

So you give the treatment to some targets and give a control treatment to others. You measure the outcome variable. You use a t-test of significance to see whether the effect is large enough that it wouldn't have happened by chance. You find that it was significant.

That's fine. Then you try a different treatment, and it doesn't cause a significant effect against the control. Does that mean the first treatment was more powerful than the second?

No. It just doesn't. The only way to find that out would be to compare the two treatments directly - and that would be very easy to do, because you have all the data to hand. If you just compare the two treatments to control you might end up with this scenario:

Both treatments are very similar but one (B) is slightly better so it's significantly different from control, while A isn't. But they're basically the same. It's probably just fluke that B did slightly better than A. If you compared A and B directly you'd find they were not significantly different.

An analogy: Passing a significance test is like winning a prize. You can only do it if you're much better than the average. But that doesn't mean you're much better than everyone who didn't win the prize, because some of them will have almost been good enough.

Usain Bolt is the fastest man in the world (when he's not false-starting himself out of races). Much faster than me. But he's not much faster than the second fastest man in the world.




Nieuwenhuis S, Forstmann BU, & Wagenmakers EJ (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nature neuroscience, 14 (9), 1105-7 PMID: 21878926

Neuroscience Fails Stats 101?

According to a new paper, a full half of neuroscience papers that try to do a (very simple) statistical comparison are getting it wrong: Erroneous analyses of interactions in neuroscience: a problem of significance.

Here's the problem. Suppose you want to know whether a certain 'treatment' has an affect on a certain variable. The treatment could be a drug, an environmental change, a genetic variant, whatever. The target population could be animals, humans, brain cells, or anything else.

So you give the treatment to some targets and give a control treatment to others. You measure the outcome variable. You use a t-test of significance to see whether the effect is large enough that it wouldn't have happened by chance. You find that it was significant.

That's fine. Then you try a different treatment, and it doesn't cause a significant effect against the control. Does that mean the first treatment was more powerful than the second?

No. It just doesn't. The only way to find that out would be to compare the two treatments directly - and that would be very easy to do, because you have all the data to hand. If you just compare the two treatments to control you might end up with this scenario:

Both treatments are very similar but one (B) is slightly better so it's significantly different from control, while A isn't. But they're basically the same. It's probably just fluke that B did slightly better than A. If you compared A and B directly you'd find they were not significantly different.

An analogy: Passing a significance test is like winning a prize. You can only do it if you're much better than the average. But that doesn't mean you're much better than everyone who didn't win the prize, because some of them will have almost been good enough.

Usain Bolt is the fastest man in the world (when he's not false-starting himself out of races). Much faster than me. But he's not much faster than the second fastest man in the world.




Nieuwenhuis S, Forstmann BU, & Wagenmakers EJ (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nature neuroscience, 14 (9), 1105-7 PMID: 21878926

Brain Connectivity, Or Head Movement?

"It's pretty painless. Basically you just need to lie there and make sure you don't move your head".


This is what I say to all the girls... who are taking part in my fMRI studies. Head movement is a big problem in fMRI. If your head moves, your brain moves and all fMRI analysis assumes that the brain is perfectly still. Although head movement correction is now a standard part of any analysis software, it's not perfect.

It may be a particular problem in functional connectivity studies, which attempt to measure the degree to which different parts of the brain are "talking" to each other, in terms of correlated neural activity over time. These are extremely popular nowadays. It's even been claimed that this data may help us understand consciousness itself (although we've heard that before).

A new paper offers some important words of caution. It shows that head motion affects estimates of functional connectivity. The more motion, the weaker the measured connectivity in long-range networks, while shorter range connections were stronger. Also, men tended to move more than women.

The effect was small - head movement can't explain more than a small fraction of the variability in connectivity.


The authors looked at 1,000 scans from healthy volunteers. They just had to lie in the scanner at rest. They looked at functional connectivity, using standard "motion correction" methods, and correlated it with head movement (which you can measure very accurately from the MRI images themselves.) Men tended to move more than women. Could this explain why women tend to have higher functional connectivity?

Disconcertingly, head movement was associated with low long range / high short range connections, which is exactly what's been proposed to happen in autism (although in fairness, not all the evidence for this comes from fMRI).

This clearly doesn't prove that the autism studies are all dodgy, but it's an issue. People with autism, and people with almost any mental or physical disorder, on average tend to move more than healthy controls.

One caveat. Could it be that brain activity causes head movement, rather than the reverse? The authors don't consider this. Head movement must come from the brain, of course. Probably from the motor cortex. The fact that motor cortex functional connectivity was positively associated with movement does suggest a possible link.

However, this paper still ought to make anyone who's using functional connectivity worry - at least a little.

Head motion is a particularly insidious confound. It is insidious because it biases between-group studies often in the direction of the hypothesized difference....even though there is considerable variation that is not due to head motion, in any given instance, a between-group difference could be entirely due to motion.

Van Dijk, K., Sabuncu, M., & Buckner, R. (2011). The Influence of Head Motion on Intrinsic Functional Connectivity MRI NeuroImage DOI: 10.1016/j.neuroimage.2011.07.044

Brain Connectivity, Or Head Movement?

"It's pretty painless. Basically you just need to lie there and make sure you don't move your head".


This is what I say to all the girls... who are taking part in my fMRI studies. Head movement is a big problem in fMRI. If your head moves, your brain moves and all fMRI analysis assumes that the brain is perfectly still. Although head movement correction is now a standard part of any analysis software, it's not perfect.

It may be a particular problem in functional connectivity studies, which attempt to measure the degree to which different parts of the brain are "talking" to each other, in terms of correlated neural activity over time. These are extremely popular nowadays. It's even been claimed that this data may help us understand consciousness itself (although we've heard that before).

A new paper offers some important words of caution. It shows that head motion affects estimates of functional connectivity. The more motion, the weaker the measured connectivity in long-range networks, while shorter range connections were stronger. Also, men tended to move more than women.

The effect was small - head movement can't explain more than a small fraction of the variability in connectivity.


The authors looked at 1,000 scans from healthy volunteers. They just had to lie in the scanner at rest. They looked at functional connectivity, using standard "motion correction" methods, and correlated it with head movement (which you can measure very accurately from the MRI images themselves.) Men tended to move more than women. Could this explain why women tend to have higher functional connectivity?

Disconcertingly, head movement was associated with low long range / high short range connections, which is exactly what's been proposed to happen in autism (although in fairness, not all the evidence for this comes from fMRI).

This clearly doesn't prove that the autism studies are all dodgy, but it's an issue. People with autism, and people with almost any mental or physical disorder, on average tend to move more than healthy controls.

One caveat. Could it be that brain activity causes head movement, rather than the reverse? The authors don't consider this. Head movement must come from the brain, of course. Probably from the motor cortex. The fact that motor cortex functional connectivity was positively associated with movement does suggest a possible link.

However, this paper still ought to make anyone who's using functional connectivity worry - at least a little.

Head motion is a particularly insidious confound. It is insidious because it biases between-group studies often in the direction of the hypothesized difference....even though there is considerable variation that is not due to head motion, in any given instance, a between-group difference could be entirely due to motion.

Van Dijk, K., Sabuncu, M., & Buckner, R. (2011). The Influence of Head Motion on Intrinsic Functional Connectivity MRI NeuroImage DOI: 10.1016/j.neuroimage.2011.07.044

Violent Brains In The Supreme Court

Back in June, the U.S. Supreme Court ruled that a Californian law banning the sale of violent videogames to children was unconstitutional because it violated the right to free speech.

However, the ruling wasn't unanimous. Justice Stephen Breyer filed a dissenting opinion. Unfortunately, it contains a whopping misuse of neuroscience. The ruling is here. Thanks to the Law & Neuroscience Blog for noticing this.

Breyer says (on page 13 of his bit)

Cutting-edge neuroscience has shown that “virtual violence in video game playing results in those neural patterns that are considered characteristic for aggressive cognition and behavior.”

He then cites this fMRI study from 2006. It's from the same group as this one I wrote about recently.

Breyer quotes this study as part of a discussion of the evidence linking violent video game use to violence. I have nothing to say about this, but I will point out than the fact that violent crime fell heavily in America after 1990, which is when the Super Nintendo and Sega Megadrive were invented.

Anyway, does this study show that playing violent games causes aggressive brain activity? Not exactly. By which I mean "no".

They scanned 13 young men playing a shooter game. The main finding was that during "violent" moments of the game, activity in the rostral ACC and the amygdala activity falls. At least this is the interpretation the authors give.

OK, but even if this neural response is "characteristic for aggressive cognition and behavior", it only lasted a few seconds. There's no evidence at all that this causes any lasting effects on brain function, or behaviour.

The real problem though is that the whole thing is based on the theory that violence is associated with reduced amygdala (and rACC) activity.

The authors cite various studies to this effect, but they don't distinguish between reduced activity as an immediate neural response to violence, as in this study, and reduced activity in people with high exposure to violent media, in response to non-violent stimuli.

This is rather like saying that because having a haircut reduces your total hair, and because bald people have no hair, haircuts cause baldness. Short-term doesn't automatically become long-term.

Besides, the whole idea that amygdala deactivation = violence is a bit weird because they used to destroy people's amydalas to reduce violent aggression in severe mental and neurological illness:

Different surgical approaches have involved various stereotactic devices and modalities for amygdaloid nucleus destruction, such as the injection of alcohol, oil, kaolin, or wax; cryoprobe lesioning; mechanical destruction; diathermy loop; and radiofrequency lesioning...

Lovely. It even worked sometimes, apparantly. Although it killed 4% of people. You can't reduce the activity of a region much more than by destroying it, yet destroying the amygdala reduced violence, or at the very least, didn't make it worse.

The truth is that aggression isn't a single thing. Everyone knows that there are two main kinds, "in cold blood" and "in the heat of the moment". Killing someone in a spontaneous bar brawl is one thing, but carefully planning to sneak up behind them and stab them is quite another.

Just based on what we know about the rare cases of amygdala-less people, I would imagine that destroying the amygdala would reduce violence "in the heat of the moment", which is motivated by anger and fear. The kind of patients who got this surgery seem to have been that kind of violent person, not the cold calculating kind.

So, even if violent video games reduced amygdala activity long term, that would probably reduce some kinds of violence.

Weber, R., Ritterfeld, U., & Mathiak, K. (2006). Does Playing Violent Video Games Induce Aggression? Empirical Evidence of a Functional Magnetic Resonance Imaging Study Media Psychology, 8 (1), 39-60 DOI: 10.1207/S1532785XMEP0801_4

Violent Brains In The Supreme Court

Back in June, the U.S. Supreme Court ruled that a Californian law banning the sale of violent videogames to children was unconstitutional because it violated the right to free speech.

However, the ruling wasn't unanimous. Justice Stephen Breyer filed a dissenting opinion. Unfortunately, it contains a whopping misuse of neuroscience. The ruling is here. Thanks to the Law & Neuroscience Blog for noticing this.

Breyer says (on page 13 of his bit)

Cutting-edge neuroscience has shown that “virtual violence in video game playing results in those neural patterns that are considered characteristic for aggressive cognition and behavior.”

He then cites this fMRI study from 2006. It's from the same group as this one I wrote about recently.

Breyer quotes this study as part of a discussion of the evidence linking violent video game use to violence. I have nothing to say about this, but I will point out than the fact that violent crime fell heavily in America after 1990, which is when the Super Nintendo and Sega Megadrive were invented.

Anyway, does this study show that playing violent games causes aggressive brain activity? Not exactly. By which I mean "no".

They scanned 13 young men playing a shooter game. The main finding was that during "violent" moments of the game, activity in the rostral ACC and the amygdala activity falls. At least this is the interpretation the authors give.

OK, but even if this neural response is "characteristic for aggressive cognition and behavior", it only lasted a few seconds. There's no evidence at all that this causes any lasting effects on brain function, or behaviour.

The real problem though is that the whole thing is based on the theory that violence is associated with reduced amygdala (and rACC) activity.

The authors cite various studies to this effect, but they don't distinguish between reduced activity as an immediate neural response to violence, as in this study, and reduced activity in people with high exposure to violent media, in response to non-violent stimuli.

This is rather like saying that because having a haircut reduces your total hair, and because bald people have no hair, haircuts cause baldness. Short-term doesn't automatically become long-term.

Besides, the whole idea that amygdala deactivation = violence is a bit weird because they used to destroy people's amydalas to reduce violent aggression in severe mental and neurological illness:

Different surgical approaches have involved various stereotactic devices and modalities for amygdaloid nucleus destruction, such as the injection of alcohol, oil, kaolin, or wax; cryoprobe lesioning; mechanical destruction; diathermy loop; and radiofrequency lesioning...

Lovely. It even worked sometimes, apparantly. Although it killed 4% of people. You can't reduce the activity of a region much more than by destroying it, yet destroying the amygdala reduced violence, or at the very least, didn't make it worse.

The truth is that aggression isn't a single thing. Everyone knows that there are two main kinds, "in cold blood" and "in the heat of the moment". Killing someone in a spontaneous bar brawl is one thing, but carefully planning to sneak up behind them and stab them is quite another.

Just based on what we know about the rare cases of amygdala-less people, I would imagine that destroying the amygdala would reduce violence "in the heat of the moment", which is motivated by anger and fear. The kind of patients who got this surgery seem to have been that kind of violent person, not the cold calculating kind.

So, even if violent video games reduced amygdala activity long term, that would probably reduce some kinds of violence.

Weber, R., Ritterfeld, U., & Mathiak, K. (2006). Does Playing Violent Video Games Induce Aggression? Empirical Evidence of a Functional Magnetic Resonance Imaging Study Media Psychology, 8 (1), 39-60 DOI: 10.1207/S1532785XMEP0801_4

The Unhelpful Brain

A reader pointed me to this study from a few months back which used fMRI to look at the effects of "Coaching With Compassion".

Unfortunately, the authors say at the outset that their paper is "Not to be quoted or reproduced without the expressed permission of one of the authors prior to publication" so I'm not going to... oh, hang on. Have I just broken the rules by quoting that? I hope not. But fair enough.

The paper describes an fMRI study of brain responses to being shown a variety of statements. The participants were students and the statements were about the university experience. They were either positive, negative, or neutral.

The authors found that the human brain responds differently to different kinds of stuff.

That's it. Well that ought to be it. The paper discusses things like Coaching With Compassion, The Ideal Self, and Intentional Change Theory, which are awesome no doubt, but they're not what this study is about.

Here's why. Before getting scanned, the students got two sessions of academic and career coaching. One session was focussed on hopes and goals for the future, dreams, and what they wanted to achieve in their studies. Yes you can! The other session, with a different coach, was all about challenges, fears, and disappointments. Maybe you can't.

The positive and the negative statements in the fMRI bit were based on these coaching interviews. The coach who did the nice bit said the nice statements (via recorded video clips) and vice versa. The positive and negative coaches were randomly assigned to each participant to avoid coach effects, and so on, which is good, the fMRI methodology was fine, and the data analysis looks good.

Who'd have thought it? Different parts of the brain were activated by positive, negative and neutral statements, and these were roughly what you'd expect from previous studies.

The reason this says nothing about coaching is that while participants got coaching beforehand, they all got the same coaching. These statements would have been positive or negative anyway - coaching or no. We don't know what, if any, effect coaching had.

Had half of them been randomized to get coached, and the other half assigned to a "placebo" coaching, say chatting about sports or the weather, then it would tell you something about coaching.

But that wouldn't mean it told you anything interesting about it, and this is the deeper problem with studies like this, of which this is only a good example.

Suppose that you found that positive, Compassionate Coaching made the brain respond more strongly to positive statements, or changed brain activity during decision-making, or whatever. That would be a result, and it might be really strong and statistically very significant, but for the life of me I can't see why you'd care, if you were interested in coaching.

Of course coaching affects the brain, and not just as a side effect: if it works, it'll work via changing the brain, in some way. But everything that changes behaviour changes the brain. That's what the brain does. How it does so is a detail of interest only to neuroscientists.

If you're a coach, or want to get coaching, or want to know whether coaching is effective, then you should look at coaching. The brain will be there, in the background, activating and deactivating happily, but it's not going to help you.

These kinds of studies happen, I think, because there's an inherent allure to seeing "the neural basis of" thoughts and feelings. It seems paradoxical and disturbing: you can't see thoughts! They're made of pixie dust and magic!

In the same way, quantum physics is universally agreed to be "weird". But it's always there, everywhere in the universe, and always has been. We're the weird ones, with our strange conviction that the most everyday thing in the world is really bizarre. God must find quantum physics incredibly boring.

Brains are not quite as commonplace as quarks, but they are at work whenever anyone, or most animals for that matter, does anything. Of course: how else would behaviour happen? We find this odd and fascinating. As a neuroscientist I'm no exception, the allure never "wears off". But that's just us.

Even people trying to be neuro-skeptical often fall into this trap. Here's Steven Rose in book review:

The weird locution – “it was not me; it was my brain that made me do it” – is increasingly used by neuroscientists who are sure that human thought and action are reducible to brain processes, and by legal defence teams pleading diminished responsibility for their clients. The trouble is that this way of speaking – and thinking, if such a term remains permissible – leaves unresolved who is the “me” that the brain drives.”

Well, human thought and action are reducible to brain processes. To deny this or (as is more common) imply that it's unhelpful, but not explain why, gets us nowhere.

The point is that all behaviour is brain activity, and that's why saying "It's brain activity" tells us nothing about any given behaviour. It’s an empty truism, like saying that a fire was started by something hot. Well, duh.

The Unhelpful Brain

A reader pointed me to this study from a few months back which used fMRI to look at the effects of "Coaching With Compassion".

Unfortunately, the authors say at the outset that their paper is "Not to be quoted or reproduced without the expressed permission of one of the authors prior to publication" so I'm not going to... oh, hang on. Have I just broken the rules by quoting that? I hope not. But fair enough.

The paper describes an fMRI study of brain responses to being shown a variety of statements. The participants were students and the statements were about the university experience. They were either positive, negative, or neutral.

The authors found that the human brain responds differently to different kinds of stuff.

That's it. Well that ought to be it. The paper discusses things like Coaching With Compassion, The Ideal Self, and Intentional Change Theory, which are awesome no doubt, but they're not what this study is about.

Here's why. Before getting scanned, the students got two sessions of academic and career coaching. One session was focussed on hopes and goals for the future, dreams, and what they wanted to achieve in their studies. Yes you can! The other session, with a different coach, was all about challenges, fears, and disappointments. Maybe you can't.

The positive and the negative statements in the fMRI bit were based on these coaching interviews. The coach who did the nice bit said the nice statements (via recorded video clips) and vice versa. The positive and negative coaches were randomly assigned to each participant to avoid coach effects, and so on, which is good, the fMRI methodology was fine, and the data analysis looks good.

Who'd have thought it? Different parts of the brain were activated by positive, negative and neutral statements, and these were roughly what you'd expect from previous studies.

The reason this says nothing about coaching is that while participants got coaching beforehand, they all got the same coaching. These statements would have been positive or negative anyway - coaching or no. We don't know what, if any, effect coaching had.

Had half of them been randomized to get coached, and the other half assigned to a "placebo" coaching, say chatting about sports or the weather, then it would tell you something about coaching.

But that wouldn't mean it told you anything interesting about it, and this is the deeper problem with studies like this, of which this is only a good example.

Suppose that you found that positive, Compassionate Coaching made the brain respond more strongly to positive statements, or changed brain activity during decision-making, or whatever. That would be a result, and it might be really strong and statistically very significant, but for the life of me I can't see why you'd care, if you were interested in coaching.

Of course coaching affects the brain, and not just as a side effect: if it works, it'll work via changing the brain, in some way. But everything that changes behaviour changes the brain. That's what the brain does. How it does so is a detail of interest only to neuroscientists.

If you're a coach, or want to get coaching, or want to know whether coaching is effective, then you should look at coaching. The brain will be there, in the background, activating and deactivating happily, but it's not going to help you.

These kinds of studies happen, I think, because there's an inherent allure to seeing "the neural basis of" thoughts and feelings. It seems paradoxical and disturbing: you can't see thoughts! They're made of pixie dust and magic!

In the same way, quantum physics is universally agreed to be "weird". But it's always there, everywhere in the universe, and always has been. We're the weird ones, with our strange conviction that the most everyday thing in the world is really bizarre. God must find quantum physics incredibly boring.

Brains are not quite as commonplace as quarks, but they are at work whenever anyone, or most animals for that matter, does anything. Of course: how else would behaviour happen? We find this odd and fascinating. As a neuroscientist I'm no exception, the allure never "wears off". But that's just us.

Even people trying to be neuro-skeptical often fall into this trap. Here's Steven Rose in book review:

The weird locution – “it was not me; it was my brain that made me do it” – is increasingly used by neuroscientists who are sure that human thought and action are reducible to brain processes, and by legal defence teams pleading diminished responsibility for their clients. The trouble is that this way of speaking – and thinking, if such a term remains permissible – leaves unresolved who is the “me” that the brain drives.”

Well, human thought and action are reducible to brain processes. To deny this or (as is more common) imply that it's unhelpful, but not explain why, gets us nowhere.

The point is that all behaviour is brain activity, and that's why saying "It's brain activity" tells us nothing about any given behaviour. It’s an empty truism, like saying that a fire was started by something hot. Well, duh.

Vaccines Cause Autism, Until You Look At The Data

According to a much-discussed new paper, vaccines may cause autism after all: A Positive Association found between Autism Prevalence and Childhood Vaccination uptake across the U.S. Population.

The author is Gayle DeLong, who "teaches international finance at Baruch College, City University of New York", according to her profile as a board member of anti-vaccine group SafeMinds. She correlated rates of coverage of the government recommended full set of vaccines in the 51 US states including Washington D.C., with registered rates of autism in those states six years later.

Uh-oh - there was a correlation between vaccination in two year kids, and the rate of autism in the state six years later, when those kids were eight. As the abstract says:

The higher the proportion of children receiving recommended vaccinations, the higher was the prevalence of AUT... The results suggest that although mercury has been removed from many vaccines, other culprits may link vaccines to autism. Further study into the relationship between vaccines and autism is warranted.

Sounds rather scary. Until you look at the data, helpfully provided in the paper. First up, here's the scatterplot of all of the vaccination rates and all of the autism-six-years-later rates:

There's more than 51 data points as you can see: there's actually 355 because each state had seven different datapoints (1995 vaccines vs 2001 autism though to 2001 vs 2007). This scatterplot shows no correlation. You can tell just from looking at it, but the correlation coefficient confirms this, as it's a tiny r 0.012 (from a possible range of 0 to 1).

To be fair, that's a very noisy measure, because each state has unique characteristics, so the effect of vaccines will be diluted. However, it's still a useful sanity check, and shows that there can't be a major effect, otherwise it would be too big to get diluted.

To get around this I next looked at the change in the rates of vaccination from one year to the next, and correlated that with the corresponding change in future rates of autism, within each state. A "change" of 1 means no change, 0.5 means it halved and 2 means it doubled, etc.

Zilch. Correlation coeffiencent r is 0.034.

Maybe the changes year-to-year were too small? So I checked the changes between the last year, and the first year.

This made the changes bigger, because more tends to change over six years than in just one. And, to be fair, this does produces a slightly stronger vaccine-autism effect... but it's still tiny. The correlation coefficient here, r, is 0.18 which means that vaccination changes accounts for 3% of the variability in autism changes (r^2 = 0.034.) The p value is 0.20, statistically insignificant.

My conclusion is that this dataset shows no evidence of any association. The author nonetheless found one. How? By doing some statistical wizardry.

The statistical model used took into consideration the unique characteristics of each state. For example, each state had a unique mixture of pollution, which may have affected the prevalence of autism, yet such an effect was not included in this study. A fixed-effects, within-group panel regression (Hall and Cummins 2005) controlled for these unique yet undefined characteristics by deriving a different starting point (intercept) for each state.

The 51 different intercepts - one for each state - reflected the base level of autism or speech disorders occurring in that state that were not explained by the other independent variables (vaccination rates, income, or ethnicity). The model then produced a single relationship between the independent variables and the prevalence of autism or speech disorders.

OK, that's all very fancy, but when the raw data shows zilch and you can only find a signal by "controlling for" stuff, alarm bells start ringing. Given sufficient statistical analysis you can make any data say anything you want.

If the author had given details of the methods, and explained why she chose to control for the variables she did, and not others, that might be different. But she didn't. Nor did she justify only looking at the effects six years later, when five or seven or ten would be just as sensible... and so on.

(Note: whenever I've said "autism", that's my shorthand for autism + SLI, which is what the paper looked at; autism alone data are not presented. Note also that by "vaccination %" I mean "% who got the full vaccine schedule"; the other kids may have got vaccines, just not all of them.)

Delong G (2011). A Positive Association found between Autism Prevalence and Childhood Vaccination uptake across the U.S. Population. Journal of toxicology and environmental health. Part A, 74 (14), 903-16 PMID: 21623535

Vaccines Cause Autism, Until You Look At The Data

According to a much-discussed new paper, vaccines may cause autism after all: A Positive Association found between Autism Prevalence and Childhood Vaccination uptake across the U.S. Population.

The author is Gayle DeLong, who "teaches international finance at Baruch College, City University of New York", according to her profile as a board member of anti-vaccine group SafeMinds. She correlated rates of coverage of the government recommended full set of vaccines in the 51 US states including Washington D.C., with registered rates of autism in those states six years later.

Uh-oh - there was a correlation between vaccination in two year kids, and the rate of autism in the state six years later, when those kids were eight. As the abstract says:

The higher the proportion of children receiving recommended vaccinations, the higher was the prevalence of AUT... The results suggest that although mercury has been removed from many vaccines, other culprits may link vaccines to autism. Further study into the relationship between vaccines and autism is warranted.

Sounds rather scary. Until you look at the data, helpfully provided in the paper. First up, here's the scatterplot of all of the vaccination rates and all of the autism-six-years-later rates:

There's more than 51 data points as you can see: there's actually 355 because each state had seven different datapoints (1995 vaccines vs 2001 autism though to 2001 vs 2007). This scatterplot shows no correlation. You can tell just from looking at it, but the correlation coefficient confirms this, as it's a tiny r 0.012 (from a possible range of 0 to 1).

To be fair, that's a very noisy measure, because each state has unique characteristics, so the effect of vaccines will be diluted. However, it's still a useful sanity check, and shows that there can't be a major effect, otherwise it would be too big to get diluted.

To get around this I next looked at the change in the rates of vaccination from one year to the next, and correlated that with the corresponding change in future rates of autism, within each state. A "change" of 1 means no change, 0.5 means it halved and 2 means it doubled, etc.

Zilch. Correlation coeffiencent r is 0.034.

Maybe the changes year-to-year were too small? So I checked the changes between the last year, and the first year.

This made the changes bigger, because more tends to change over six years than in just one. And, to be fair, this does produces a slightly stronger vaccine-autism effect... but it's still tiny. The correlation coefficient here, r, is 0.18 which means that vaccination changes accounts for 3% of the variability in autism changes (r^2 = 0.034.) The p value is 0.20, statistically insignificant.

My conclusion is that this dataset shows no evidence of any association. The author nonetheless found one. How? By doing some statistical wizardry.

The statistical model used took into consideration the unique characteristics of each state. For example, each state had a unique mixture of pollution, which may have affected the prevalence of autism, yet such an effect was not included in this study. A fixed-effects, within-group panel regression (Hall and Cummins 2005) controlled for these unique yet undefined characteristics by deriving a different starting point (intercept) for each state.

The 51 different intercepts - one for each state - reflected the base level of autism or speech disorders occurring in that state that were not explained by the other independent variables (vaccination rates, income, or ethnicity). The model then produced a single relationship between the independent variables and the prevalence of autism or speech disorders.

OK, that's all very fancy, but when the raw data shows zilch and you can only find a signal by "controlling for" stuff, alarm bells start ringing. Given sufficient statistical analysis you can make any data say anything you want.

If the author had given details of the methods, and explained why she chose to control for the variables she did, and not others, that might be different. But she didn't. Nor did she justify only looking at the effects six years later, when five or seven or ten would be just as sensible... and so on.

(Note: whenever I've said "autism", that's my shorthand for autism + SLI, which is what the paper looked at; autism alone data are not presented. Note also that by "vaccination %" I mean "% who got the full vaccine schedule"; the other kids may have got vaccines, just not all of them.)

Delong G (2011). A Positive Association found between Autism Prevalence and Childhood Vaccination uptake across the U.S. Population. Journal of toxicology and environmental health. Part A, 74 (14), 903-16 PMID: 21623535

Psychiatry and Phrenology

The notorious John P. "Most Published Research Findings Are False" Ioannidis has turned his baleful statistical gaze upon the literature on brain volume abnormalities in psychiatric disorders.


Reports of regional volume differences in the brains of people with mental illness compared to healthy people have appeared in increasing numbers in recent years. Such studies have given plenty of positive results. People with depression have smaller hippocampi. The amygdala is bigger in people with autism. And so on.

Last month, Ioannidis took a comprehensive look at this literature and he argues that it suffers from a fairly serious case of "excess significance bias" - essentially, that scientists are somehow biased towards reporting differences between patients and controls, and are not telling people about the times when there wasn't a difference. This could be because of publication bias, p-value fishing or other scientific sins.

Scientists tend to call a difference between two groups significant if it has a p value of less than 0.05. This means that if there were no real difference, just some random noise, this result would be less than 5% likely to occur.

However, there's many ways you could end up with a low (i.e. good) p value. You would get a significant result, even if the true difference was very small, if you do a big enough study. Even a small difference will be detected if you study enough people. On the other hand, when the true difference is huge, you might only need a small study to get the same p value.

A power calculation is a way of specifying how likely a given study would be to detect a difference of a given size, based on the size of the study. These are usually used ahead of time to work out how big your upcoming study needs to be, assuming you can guess roughly how big the real effect you're interested in is going to be.

Ioannidis turned this on its head and asked: assuming that the true difference in the brain volume is what the average of all the published studies says it is, how many of the published studies were big enough that they ought to have succesfully detected it?

He found 41 seperate meta-analyses for different brain regions in various disorders. These were published in 8 papers - because each paper reported on multiple regions. He only looked at meta-analyses published in the past 4 years, but these analyses will themselves have included older work. This means that this paper is a kind of meta-meta-analysis. He didn't directly consider the raw brain scans at all.

The meta-analyses found many significant volume differences - but in 29 of those 41, there was an excess of significant papers. In other words, the papers were too small to have a good chance to detect the effect that they themselves found - suggesting that something funny was going on. Although, strangely, in 10/41 there were too few, and only in 2 were there the "right" number.


For what it's worth, studies on schizophrenia and on relatives-of-people-with-schizophrenia showed the least evidence of this problem, while autism was terrible, with 4 times as many significant papers as expected by chance. I'm not sure this is worth much, though. We don't know if this tells us more about schizophrenia vs autism, or more about the researchers that study them.

Anyway, this is an important study, and the inverse power calculation approach is certainly a useful one. It's not new, but it's not used as widely as it ought to be. It does make the assumption that the meta-analyses are "right" about the effect size, and then paradoxically concludes that they are biased. However, this means that the true bias is probably even bigger than this suggests (because if the analyses as biased, the true effect size is smaller than assumed, and the studies should have been even less likely to find it.)

Unfortunately, this doesn't tell us which of the studies are wrong, so it's not directly useful for people researching mental illness. It tells us that there is something wrong with scientific publishing, however. Truth be told, I suspect that a similar picture would emerge if you did this kind of thing in many other fields of science. The only real solution, in my book, would be to require the pre-registration of scientific studies. Ioannidis actually advocates this at the end of the paper.

Ioannidis JP (2011). Excess Significance Bias in the Literature on Brain Volume Abnormalities. Archives of general psychiatry PMID: 21464342

Psychiatry and Phrenology

The notorious John P. "Most Published Research Findings Are False" Ioannidis has turned his baleful statistical gaze upon the literature on brain volume abnormalities in psychiatric disorders.


Reports of regional volume differences in the brains of people with mental illness compared to healthy people have appeared in increasing numbers in recent years. Such studies have given plenty of positive results. People with depression have smaller hippocampi. The amygdala is bigger in people with autism. And so on.

Last month, Ioannidis took a comprehensive look at this literature and he argues that it suffers from a fairly serious case of "excess significance bias" - essentially, that scientists are somehow biased towards reporting differences between patients and controls, and are not telling people about the times when there wasn't a difference. This could be because of publication bias, p-value fishing or other scientific sins.

Scientists tend to call a difference between two groups significant if it has a p value of less than 0.05. This means that if there were no real difference, just some random noise, this result would be less than 5% likely to occur.

However, there's many ways you could end up with a low (i.e. good) p value. You would get a significant result, even if the true difference was very small, if you do a big enough study. Even a small difference will be detected if you study enough people. On the other hand, when the true difference is huge, you might only need a small study to get the same p value.

A power calculation is a way of specifying how likely a given study would be to detect a difference of a given size, based on the size of the study. These are usually used ahead of time to work out how big your upcoming study needs to be, assuming you can guess roughly how big the real effect you're interested in is going to be.

Ioannidis turned this on its head and asked: assuming that the true difference in the brain volume is what the average of all the published studies says it is, how many of the published studies were big enough that they ought to have succesfully detected it?

He found 41 seperate meta-analyses for different brain regions in various disorders. These were published in 8 papers - because each paper reported on multiple regions. He only looked at meta-analyses published in the past 4 years, but these analyses will themselves have included older work. This means that this paper is a kind of meta-meta-analysis. He didn't directly consider the raw brain scans at all.

The meta-analyses found many significant volume differences - but in 29 of those 41, there was an excess of significant papers. In other words, the papers were too small to have a good chance to detect the effect that they themselves found - suggesting that something funny was going on. Although, strangely, in 10/41 there were too few, and only in 2 were there the "right" number.


For what it's worth, studies on schizophrenia and on relatives-of-people-with-schizophrenia showed the least evidence of this problem, while autism was terrible, with 4 times as many significant papers as expected by chance. I'm not sure this is worth much, though. We don't know if this tells us more about schizophrenia vs autism, or more about the researchers that study them.

Anyway, this is an important study, and the inverse power calculation approach is certainly a useful one. It's not new, but it's not used as widely as it ought to be. It does make the assumption that the meta-analyses are "right" about the effect size, and then paradoxically concludes that they are biased. However, this means that the true bias is probably even bigger than this suggests (because if the analyses as biased, the true effect size is smaller than assumed, and the studies should have been even less likely to find it.)

Unfortunately, this doesn't tell us which of the studies are wrong, so it's not directly useful for people researching mental illness. It tells us that there is something wrong with scientific publishing, however. Truth be told, I suspect that a similar picture would emerge if you did this kind of thing in many other fields of science. The only real solution, in my book, would be to require the pre-registration of scientific studies. Ioannidis actually advocates this at the end of the paper.

Ioannidis JP (2011). Excess Significance Bias in the Literature on Brain Volume Abnormalities. Archives of general psychiatry PMID: 21464342

Neuroskeptic Irreverent and Sometimes Profane, Study Finds

I was most surprised and honored to find out this morning that the Annals of Neurology has declared Neuroskeptic to be

Irreverent, sometimes profane, and can skirt the boundaries of good taste. Nonetheless, Neuroskeptic covers a rich mixture of important, engaging, or amusing topics focusing on the basic and clinical neurosciences, and does so in a data-driven, user-friendly, and comment-enabled format. Neuroskeptic is only one of a number of increasingly used web sites and blogs dedicated to promoting public education, rational discourse, and a healthy dose of skepticism around important issues in the neurosciences...

No really: Scientific literacy and the media. They also list a small number of other neuroblogs, although they leave out many outstanding ones including the blog that most inspired this one, and that everyone confuses me with, The Neurocritic.

Anyway, the editorial goes on to note that:

Last April, a series of sensationalist stories reporting the “creation of life” and a newfound capability to “play God” appeared in the national media following the demonstration that synthetic DNA could transform a mycoplasma species from one to another subtype(ref). This represented a tour de force of DNA synthesis, but probably only a modest step forward for the science of genetic engineering.

In response, President Obama directed his Presidential Commission for the Study of Bioethical Issues to prepare a comprehensive advisory report to help frame policies about synthetic biology(ref).

The Commission noted that sensationalist headlines may attract readers to scientific topics but do a terrible disservice by promoting “claims that fail to convey accurately to the public the current state of the field, the implications of research results, and the limits of scientists' present knowledge and abilities.” The Presidential Commission recommended creating a well-funded, interactive website... to monitor claims about new scientific discoveries and technologies.

Ideally, such a site would be only part of a wider effort to promote scientific literacy and critical thinking across all segments of society... In the coming years, scientific innovation is certain to play an increasingly large role in the global economy... The public discourse on these and related matters needs to be rational, evidence-based, and accurate.

Broadly speaking, this is why I write this blog, because it is indeed extremely important. Well, ok, the real reason is that it gives me an excuse to make funny pictures with MS Paint (someone accused me of using Photoshop to do those once - no, that would be too advanced). However, if a few people understand neuroscience a bit better in the process, I can live with that...

Hauser, S., & Johnston, S. (2011). Scientific literacy and the media Annals of Neurology, 69 (3) DOI: 10.1002/ana.22410

Neuroskeptic Irreverent and Sometimes Profane, Study Finds

I was most surprised and honored to find out this morning that the Annals of Neurology has declared Neuroskeptic to be

Irreverent, sometimes profane, and can skirt the boundaries of good taste. Nonetheless, Neuroskeptic covers a rich mixture of important, engaging, or amusing topics focusing on the basic and clinical neurosciences, and does so in a data-driven, user-friendly, and comment-enabled format. Neuroskeptic is only one of a number of increasingly used web sites and blogs dedicated to promoting public education, rational discourse, and a healthy dose of skepticism around important issues in the neurosciences...

No really: Scientific literacy and the media. They also list a small number of other neuroblogs, although they leave out many outstanding ones including the blog that most inspired this one, and that everyone confuses me with, The Neurocritic.

Anyway, the editorial goes on to note that:

Last April, a series of sensationalist stories reporting the “creation of life” and a newfound capability to “play God” appeared in the national media following the demonstration that synthetic DNA could transform a mycoplasma species from one to another subtype(ref). This represented a tour de force of DNA synthesis, but probably only a modest step forward for the science of genetic engineering.

In response, President Obama directed his Presidential Commission for the Study of Bioethical Issues to prepare a comprehensive advisory report to help frame policies about synthetic biology(ref).

The Commission noted that sensationalist headlines may attract readers to scientific topics but do a terrible disservice by promoting “claims that fail to convey accurately to the public the current state of the field, the implications of research results, and the limits of scientists' present knowledge and abilities.” The Presidential Commission recommended creating a well-funded, interactive website... to monitor claims about new scientific discoveries and technologies.

Ideally, such a site would be only part of a wider effort to promote scientific literacy and critical thinking across all segments of society... In the coming years, scientific innovation is certain to play an increasingly large role in the global economy... The public discourse on these and related matters needs to be rational, evidence-based, and accurate.

Broadly speaking, this is why I write this blog, because it is indeed extremely important. Well, ok, the real reason is that it gives me an excuse to make funny pictures with MS Paint (someone accused me of using Photoshop to do those once - no, that would be too advanced). However, if a few people understand neuroscience a bit better in the process, I can live with that...

Hauser, S., & Johnston, S. (2011). Scientific literacy and the media Annals of Neurology, 69 (3) DOI: 10.1002/ana.22410