Neuroscience Fails Stats 101?

According to a new paper, a full half of neuroscience papers that try to do a (very simple) statistical comparison are getting it wrong: Erroneous analyses of interactions in neuroscience: a problem of significance.

Here's the problem. Suppose you want to know whether a certain 'treatment' has an affect on a certain variable. The treatment could be a drug, an environmental change, a genetic variant, whatever. The target population could be animals, humans, brain cells, or anything else.

So you give the treatment to some targets and give a control treatment to others. You measure the outcome variable. You use a t-test of significance to see whether the effect is large enough that it wouldn't have happened by chance. You find that it was significant.

That's fine. Then you try a different treatment, and it doesn't cause a significant effect against the control. Does that mean the first treatment was more powerful than the second?

No. It just doesn't. The only way to find that out would be to compare the two treatments directly - and that would be very easy to do, because you have all the data to hand. If you just compare the two treatments to control you might end up with this scenario:

Both treatments are very similar but one (B) is slightly better so it's significantly different from control, while A isn't. But they're basically the same. It's probably just fluke that B did slightly better than A. If you compared A and B directly you'd find they were not significantly different.

An analogy: Passing a significance test is like winning a prize. You can only do it if you're much better than the average. But that doesn't mean you're much better than everyone who didn't win the prize, because some of them will have almost been good enough.

Usain Bolt is the fastest man in the world (when he's not false-starting himself out of races). Much faster than me. But he's not much faster than the second fastest man in the world.




Nieuwenhuis S, Forstmann BU, & Wagenmakers EJ (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nature neuroscience, 14 (9), 1105-7 PMID: 21878926

Neuroscience Fails Stats 101?

According to a new paper, a full half of neuroscience papers that try to do a (very simple) statistical comparison are getting it wrong: Erroneous analyses of interactions in neuroscience: a problem of significance.

Here's the problem. Suppose you want to know whether a certain 'treatment' has an affect on a certain variable. The treatment could be a drug, an environmental change, a genetic variant, whatever. The target population could be animals, humans, brain cells, or anything else.

So you give the treatment to some targets and give a control treatment to others. You measure the outcome variable. You use a t-test of significance to see whether the effect is large enough that it wouldn't have happened by chance. You find that it was significant.

That's fine. Then you try a different treatment, and it doesn't cause a significant effect against the control. Does that mean the first treatment was more powerful than the second?

No. It just doesn't. The only way to find that out would be to compare the two treatments directly - and that would be very easy to do, because you have all the data to hand. If you just compare the two treatments to control you might end up with this scenario:

Both treatments are very similar but one (B) is slightly better so it's significantly different from control, while A isn't. But they're basically the same. It's probably just fluke that B did slightly better than A. If you compared A and B directly you'd find they were not significantly different.

An analogy: Passing a significance test is like winning a prize. You can only do it if you're much better than the average. But that doesn't mean you're much better than everyone who didn't win the prize, because some of them will have almost been good enough.

Usain Bolt is the fastest man in the world (when he's not false-starting himself out of races). Much faster than me. But he's not much faster than the second fastest man in the world.




Nieuwenhuis S, Forstmann BU, & Wagenmakers EJ (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nature neuroscience, 14 (9), 1105-7 PMID: 21878926

Machine-Readable Psychiatry

The idea of trawling the internet to discover what people think about medications is a fascinating one and I've covered some attempts to do this in the past, but it's not easy. And there's something worrying about where it could lead.

A new paper aims to trawl medical records to work out how well depressed patients responded to treatment. The authors used Natural Language Processing or NLP (not that NLP) to interpret electronic medical records from over 5,000 patients treated at hospitals in New England. Each record included notes taken at multiple visits.

A crack team of "three experienced board-certified clinical psychiatrists" reviewed the notes and provided a "Gold Standard" classification as to whether patients were Depressed, Recovered or Intermediate at each visit. The problem here is that they didn't actually see the patients, they just had the notes. If the notes were bad, the result will have been bad too. Garbage In, Garbage Out. Even if you then put a big gold medal on the garbage.

They then found that an NLP algorithm was able to learn how to duplicate the expert opinion, based on the words used in the notes. Using a machine learning approach they were able to teach the computer that if the text contained the word "depressed", it was a sign that the patient was depressed while "much better" was associated with being... guess.

In fairness, it's not a bad attempt to turn text into numbers, and in future it could allow you to do interesting things such as comparing two drugs in terms of which ones make people "much better".

I'm concerned about this though. The essence of the original, narrative notes, is that they contain individual information about that patient's story. You could go through them with a computer and calculate what happens to the average patient given a certain drug. That might be useful information. But if you did that as a replacement for reading about individual patients, you'd be missing the whole point of the narrative notes.

Worse, as this kind of thing becomes feaisble it will feed back on itself and encourage clinicians to write their notes -and therefore to think, inevitably - in machine-readable terms. The authors suggest as much:

As more health care systems move to electronic medical records, there is a unique opportunity to better quantify outcomes. For example, the 16-item patient-rated QIDS-SR [questionairre] has been shown to be highly correlated with clinician-rated measures and sensitive to treatment effects.... At minimum, EMR systems that utilize templates could require clinicians to record a clinical status for example, using the 7-point Clinical Global Impression scale...

Indeed, many say that this already is happening. Now quantification is generally a good thing I think, but only so long as it's an aid to understanding, not a replacement for it.

Yet quantification often does become a replacement for understanding because there's a trap that we face when trying to deal with a complicated set of information. The temptation is to focus on the easiest bit to measure, because that's easy, and then assume that this represents the state of the whole thing. The reason that something's easy to measure is often because it doesn't capture the whole phenomena.

Perlis RH, Iosifescu DV, Castro VM, Murphy SN, Gainer VS, Minnier J, Cai T, Goryachev S, Zeng Q, Gallagher PJ, Fava M, Weilburg JB, Churchill SE, Kohane IS, & Smoller JW (2011). Using electronic medical records to enable large-scale studies in psychiatry: treatment resistant depression as a model. Psychological medicine, 1-10 PMID: 21682950

Machine-Readable Psychiatry

The idea of trawling the internet to discover what people think about medications is a fascinating one and I've covered some attempts to do this in the past, but it's not easy. And there's something worrying about where it could lead.

A new paper aims to trawl medical records to work out how well depressed patients responded to treatment. The authors used Natural Language Processing or NLP (not that NLP) to interpret electronic medical records from over 5,000 patients treated at hospitals in New England. Each record included notes taken at multiple visits.

A crack team of "three experienced board-certified clinical psychiatrists" reviewed the notes and provided a "Gold Standard" classification as to whether patients were Depressed, Recovered or Intermediate at each visit. The problem here is that they didn't actually see the patients, they just had the notes. If the notes were bad, the result will have been bad too. Garbage In, Garbage Out. Even if you then put a big gold medal on the garbage.

They then found that an NLP algorithm was able to learn how to duplicate the expert opinion, based on the words used in the notes. Using a machine learning approach they were able to teach the computer that if the text contained the word "depressed", it was a sign that the patient was depressed while "much better" was associated with being... guess.

In fairness, it's not a bad attempt to turn text into numbers, and in future it could allow you to do interesting things such as comparing two drugs in terms of which ones make people "much better".

I'm concerned about this though. The essence of the original, narrative notes, is that they contain individual information about that patient's story. You could go through them with a computer and calculate what happens to the average patient given a certain drug. That might be useful information. But if you did that as a replacement for reading about individual patients, you'd be missing the whole point of the narrative notes.

Worse, as this kind of thing becomes feaisble it will feed back on itself and encourage clinicians to write their notes -and therefore to think, inevitably - in machine-readable terms. The authors suggest as much:

As more health care systems move to electronic medical records, there is a unique opportunity to better quantify outcomes. For example, the 16-item patient-rated QIDS-SR [questionairre] has been shown to be highly correlated with clinician-rated measures and sensitive to treatment effects.... At minimum, EMR systems that utilize templates could require clinicians to record a clinical status for example, using the 7-point Clinical Global Impression scale...

Indeed, many say that this already is happening. Now quantification is generally a good thing I think, but only so long as it's an aid to understanding, not a replacement for it.

Yet quantification often does become a replacement for understanding because there's a trap that we face when trying to deal with a complicated set of information. The temptation is to focus on the easiest bit to measure, because that's easy, and then assume that this represents the state of the whole thing. The reason that something's easy to measure is often because it doesn't capture the whole phenomena.

Perlis RH, Iosifescu DV, Castro VM, Murphy SN, Gainer VS, Minnier J, Cai T, Goryachev S, Zeng Q, Gallagher PJ, Fava M, Weilburg JB, Churchill SE, Kohane IS, & Smoller JW (2011). Using electronic medical records to enable large-scale studies in psychiatry: treatment resistant depression as a model. Psychological medicine, 1-10 PMID: 21682950

Vaccines Cause Autism, Until You Look At The Data

According to a much-discussed new paper, vaccines may cause autism after all: A Positive Association found between Autism Prevalence and Childhood Vaccination uptake across the U.S. Population.

The author is Gayle DeLong, who "teaches international finance at Baruch College, City University of New York", according to her profile as a board member of anti-vaccine group SafeMinds. She correlated rates of coverage of the government recommended full set of vaccines in the 51 US states including Washington D.C., with registered rates of autism in those states six years later.

Uh-oh - there was a correlation between vaccination in two year kids, and the rate of autism in the state six years later, when those kids were eight. As the abstract says:

The higher the proportion of children receiving recommended vaccinations, the higher was the prevalence of AUT... The results suggest that although mercury has been removed from many vaccines, other culprits may link vaccines to autism. Further study into the relationship between vaccines and autism is warranted.

Sounds rather scary. Until you look at the data, helpfully provided in the paper. First up, here's the scatterplot of all of the vaccination rates and all of the autism-six-years-later rates:

There's more than 51 data points as you can see: there's actually 355 because each state had seven different datapoints (1995 vaccines vs 2001 autism though to 2001 vs 2007). This scatterplot shows no correlation. You can tell just from looking at it, but the correlation coefficient confirms this, as it's a tiny r 0.012 (from a possible range of 0 to 1).

To be fair, that's a very noisy measure, because each state has unique characteristics, so the effect of vaccines will be diluted. However, it's still a useful sanity check, and shows that there can't be a major effect, otherwise it would be too big to get diluted.

To get around this I next looked at the change in the rates of vaccination from one year to the next, and correlated that with the corresponding change in future rates of autism, within each state. A "change" of 1 means no change, 0.5 means it halved and 2 means it doubled, etc.

Zilch. Correlation coeffiencent r is 0.034.

Maybe the changes year-to-year were too small? So I checked the changes between the last year, and the first year.

This made the changes bigger, because more tends to change over six years than in just one. And, to be fair, this does produces a slightly stronger vaccine-autism effect... but it's still tiny. The correlation coefficient here, r, is 0.18 which means that vaccination changes accounts for 3% of the variability in autism changes (r^2 = 0.034.) The p value is 0.20, statistically insignificant.

My conclusion is that this dataset shows no evidence of any association. The author nonetheless found one. How? By doing some statistical wizardry.

The statistical model used took into consideration the unique characteristics of each state. For example, each state had a unique mixture of pollution, which may have affected the prevalence of autism, yet such an effect was not included in this study. A fixed-effects, within-group panel regression (Hall and Cummins 2005) controlled for these unique yet undefined characteristics by deriving a different starting point (intercept) for each state.

The 51 different intercepts - one for each state - reflected the base level of autism or speech disorders occurring in that state that were not explained by the other independent variables (vaccination rates, income, or ethnicity). The model then produced a single relationship between the independent variables and the prevalence of autism or speech disorders.

OK, that's all very fancy, but when the raw data shows zilch and you can only find a signal by "controlling for" stuff, alarm bells start ringing. Given sufficient statistical analysis you can make any data say anything you want.

If the author had given details of the methods, and explained why she chose to control for the variables she did, and not others, that might be different. But she didn't. Nor did she justify only looking at the effects six years later, when five or seven or ten would be just as sensible... and so on.

(Note: whenever I've said "autism", that's my shorthand for autism + SLI, which is what the paper looked at; autism alone data are not presented. Note also that by "vaccination %" I mean "% who got the full vaccine schedule"; the other kids may have got vaccines, just not all of them.)

Delong G (2011). A Positive Association found between Autism Prevalence and Childhood Vaccination uptake across the U.S. Population. Journal of toxicology and environmental health. Part A, 74 (14), 903-16 PMID: 21623535

Vaccines Cause Autism, Until You Look At The Data

According to a much-discussed new paper, vaccines may cause autism after all: A Positive Association found between Autism Prevalence and Childhood Vaccination uptake across the U.S. Population.

The author is Gayle DeLong, who "teaches international finance at Baruch College, City University of New York", according to her profile as a board member of anti-vaccine group SafeMinds. She correlated rates of coverage of the government recommended full set of vaccines in the 51 US states including Washington D.C., with registered rates of autism in those states six years later.

Uh-oh - there was a correlation between vaccination in two year kids, and the rate of autism in the state six years later, when those kids were eight. As the abstract says:

The higher the proportion of children receiving recommended vaccinations, the higher was the prevalence of AUT... The results suggest that although mercury has been removed from many vaccines, other culprits may link vaccines to autism. Further study into the relationship between vaccines and autism is warranted.

Sounds rather scary. Until you look at the data, helpfully provided in the paper. First up, here's the scatterplot of all of the vaccination rates and all of the autism-six-years-later rates:

There's more than 51 data points as you can see: there's actually 355 because each state had seven different datapoints (1995 vaccines vs 2001 autism though to 2001 vs 2007). This scatterplot shows no correlation. You can tell just from looking at it, but the correlation coefficient confirms this, as it's a tiny r 0.012 (from a possible range of 0 to 1).

To be fair, that's a very noisy measure, because each state has unique characteristics, so the effect of vaccines will be diluted. However, it's still a useful sanity check, and shows that there can't be a major effect, otherwise it would be too big to get diluted.

To get around this I next looked at the change in the rates of vaccination from one year to the next, and correlated that with the corresponding change in future rates of autism, within each state. A "change" of 1 means no change, 0.5 means it halved and 2 means it doubled, etc.

Zilch. Correlation coeffiencent r is 0.034.

Maybe the changes year-to-year were too small? So I checked the changes between the last year, and the first year.

This made the changes bigger, because more tends to change over six years than in just one. And, to be fair, this does produces a slightly stronger vaccine-autism effect... but it's still tiny. The correlation coefficient here, r, is 0.18 which means that vaccination changes accounts for 3% of the variability in autism changes (r^2 = 0.034.) The p value is 0.20, statistically insignificant.

My conclusion is that this dataset shows no evidence of any association. The author nonetheless found one. How? By doing some statistical wizardry.

The statistical model used took into consideration the unique characteristics of each state. For example, each state had a unique mixture of pollution, which may have affected the prevalence of autism, yet such an effect was not included in this study. A fixed-effects, within-group panel regression (Hall and Cummins 2005) controlled for these unique yet undefined characteristics by deriving a different starting point (intercept) for each state.

The 51 different intercepts - one for each state - reflected the base level of autism or speech disorders occurring in that state that were not explained by the other independent variables (vaccination rates, income, or ethnicity). The model then produced a single relationship between the independent variables and the prevalence of autism or speech disorders.

OK, that's all very fancy, but when the raw data shows zilch and you can only find a signal by "controlling for" stuff, alarm bells start ringing. Given sufficient statistical analysis you can make any data say anything you want.

If the author had given details of the methods, and explained why she chose to control for the variables she did, and not others, that might be different. But she didn't. Nor did she justify only looking at the effects six years later, when five or seven or ten would be just as sensible... and so on.

(Note: whenever I've said "autism", that's my shorthand for autism + SLI, which is what the paper looked at; autism alone data are not presented. Note also that by "vaccination %" I mean "% who got the full vaccine schedule"; the other kids may have got vaccines, just not all of them.)

Delong G (2011). A Positive Association found between Autism Prevalence and Childhood Vaccination uptake across the U.S. Population. Journal of toxicology and environmental health. Part A, 74 (14), 903-16 PMID: 21623535

How To Fix Science

Over at Bad Science, Ben Goldacre discusses a big problem with modern science - the published literature is all very well and good, but we don't know what people are finding that goes unpublished:

The scale of the academic universe is dizzying, after all. Our most recent estimate is that there are over 24,000 academic journals in existence, 1.3 million academic papers published every year, and over 50 million papers published since scholarship began.

And for every one of these 50 million papers there will be unknowable quantities of blind alleys, abandoned experiments, conference presentations, work in progress seminars, and more. Look at the vast number of undergraduate and masters dissertations that had an interesting finding, and got turned into finished academic papers: and then think about the even vaster number that don’t...

We are living in the age of information, and vast tracts of data are being generated around the world on every continent and every question. A £200 laptop will let you run endless statistical analyses. The most interesting questions aren’t around individual nuggets of data, but rather how we can corral it to create an information architecture which serves up the whole picture.

I agree with all of this. It is a problem. In fact I'd say it's the single biggest problem with science today. Scientists are required to publish ever-increasing numbers of high-impact papers, in order to get grants and promotions, with the "best" papers, usually meaning the ones with the most interesting positive results, being favored.

Findings that show that nothing especially interesting is going on here all too often get swept under the carpet or re-re-analyzed until a positive result falls out. If you do a study of a certain gene and its association to brain function, say, and find it has no association: that's bad news for you. That will make a low-impact paper, if it makes a paper at all. But maybe it has an association with brain structure? Or personality?

Anyway, that's the problem. What to do about it? Goldacre notes that in medicine, there are mechanisms in place to deal with this:

In medicine, where the stakes are tangible, systems have grown up to try and cope with this problem: trials are supposed registered before they begin, so we can notice the results that get left unpublished. But even here, the systems are imperfect; and pre-registration is very rarely done, even in medical research, for anything other than trials.

Clinical trial pre-registration is a fantastic idea. The systems are certainly imperfect, but they're getting better, and they're much better than nothing. Back in 2008 I proposed that all scientific studies, not just clinical trials, should be publically pre-registered. That way everyone could know what science was going unpublished and could tell when authors were doing analyzes they hadn't originally planned to do (which is fine, so long as you admit to it.)

I still think that would be a good idea. But how would it work in practice? Here's what I've come up with:

Scientific papers should be submitted to journals for publication before the research has started. The Introduction and the Methods section, detailing what you plan to do and why, would then get peer reviewed. The rest of the paper would obviously be a blank at this stage. Anonymous experts would have a chance to critique the methods and rationale.

If the paper's accepted, you then do the research, get the results, and write the Results and Discussion section of the paper. The journal is then required to publish the final paper, assuming that you kept to the original plan. The Introducion and primary Methods would be fixed - you can't change them once the data come in.

You can do additional stuff and run additional analyses all you like, but they'll be marked as secondary, which of course is what they are. Publication would therefore be based on the scientific merits of the experiment, the importance of the question and the quality of the methods, not the "interestingness" of the results. If you want a paper in Nature, it needs to be a great idea, not a lucky shot.

This would be a radical change from the current system. Too radical, almost certainly, to ever happen in one go. So here's another idea as to a kind of stepping-stone on the way:

Already, scientists have to spell out their original rationale and original methods before they do any work - when they apply for funding from a grant awarding body. These grant applications are often very detailed, but at the moment, they're private. And people don't always stick to them.

Why not make the full publication of the grant application a condition of being awarded the money? This would be rather like preregistration of the Introduction and Methods, although less elegant, but it would do the job. And given that most grants consist of public cash, the public really have a right to know this. These applications are usually just PDF files. It would be trivial to put them online - after redacting personal information like applicant résumés, if desired.

How To Fix Science

Over at Bad Science, Ben Goldacre discusses a big problem with modern science - the published literature is all very well and good, but we don't know what people are finding that goes unpublished:

The scale of the academic universe is dizzying, after all. Our most recent estimate is that there are over 24,000 academic journals in existence, 1.3 million academic papers published every year, and over 50 million papers published since scholarship began.

And for every one of these 50 million papers there will be unknowable quantities of blind alleys, abandoned experiments, conference presentations, work in progress seminars, and more. Look at the vast number of undergraduate and masters dissertations that had an interesting finding, and got turned into finished academic papers: and then think about the even vaster number that don’t...

We are living in the age of information, and vast tracts of data are being generated around the world on every continent and every question. A £200 laptop will let you run endless statistical analyses. The most interesting questions aren’t around individual nuggets of data, but rather how we can corral it to create an information architecture which serves up the whole picture.

I agree with all of this. It is a problem. In fact I'd say it's the single biggest problem with science today. Scientists are required to publish ever-increasing numbers of high-impact papers, in order to get grants and promotions, with the "best" papers, usually meaning the ones with the most interesting positive results, being favored.

Findings that show that nothing especially interesting is going on here all too often get swept under the carpet or re-re-analyzed until a positive result falls out. If you do a study of a certain gene and its association to brain function, say, and find it has no association: that's bad news for you. That will make a low-impact paper, if it makes a paper at all. But maybe it has an association with brain structure? Or personality?

Anyway, that's the problem. What to do about it? Goldacre notes that in medicine, there are mechanisms in place to deal with this:

In medicine, where the stakes are tangible, systems have grown up to try and cope with this problem: trials are supposed registered before they begin, so we can notice the results that get left unpublished. But even here, the systems are imperfect; and pre-registration is very rarely done, even in medical research, for anything other than trials.

Clinical trial pre-registration is a fantastic idea. The systems are certainly imperfect, but they're getting better, and they're much better than nothing. Back in 2008 I proposed that all scientific studies, not just clinical trials, should be publically pre-registered. That way everyone could know what science was going unpublished and could tell when authors were doing analyzes they hadn't originally planned to do (which is fine, so long as you admit to it.)

I still think that would be a good idea. But how would it work in practice? Here's what I've come up with:

Scientific papers should be submitted to journals for publication before the research has started. The Introduction and the Methods section, detailing what you plan to do and why, would then get peer reviewed. The rest of the paper would obviously be a blank at this stage. Anonymous experts would have a chance to critique the methods and rationale.

If the paper's accepted, you then do the research, get the results, and write the Results and Discussion section of the paper. The journal is then required to publish the final paper, assuming that you kept to the original plan. The Introducion and primary Methods would be fixed - you can't change them once the data come in.

You can do additional stuff and run additional analyses all you like, but they'll be marked as secondary, which of course is what they are. Publication would therefore be based on the scientific merits of the experiment, the importance of the question and the quality of the methods, not the "interestingness" of the results. If you want a paper in Nature, it needs to be a great idea, not a lucky shot.

This would be a radical change from the current system. Too radical, almost certainly, to ever happen in one go. So here's another idea as to a kind of stepping-stone on the way:

Already, scientists have to spell out their original rationale and original methods before they do any work - when they apply for funding from a grant awarding body. These grant applications are often very detailed, but at the moment, they're private. And people don't always stick to them.

Why not make the full publication of the grant application a condition of being awarded the money? This would be rather like preregistration of the Introduction and Methods, although less elegant, but it would do the job. And given that most grants consist of public cash, the public really have a right to know this. These applications are usually just PDF files. It would be trivial to put them online - after redacting personal information like applicant résumés, if desired.

Revenge Of The Depression Gene

Last year, the world of psychiatric genetics was rocked by the news that a highly-studied gene, believed to be associated with depression, wasn't in fact linked to depression at all.

The genetic variant was 5-HTTLPR. It's a length variant in the gene coding for the serotonin transporter protein (5HTT) which the target of antidepressants like Prozac. There are two flavors of this variant, short and long.

Many studies have shown that the short ("s") variant is associated with a high risk of getting depression in response to stress - but then last year a large meta-analysis of all the evidence concluded that there was in reality no link. Bummer.

Now another team of researchers have done a new analysis of the 5-HTTLPR & stress & depression data and they claim that there is a link after all: hooray! So who's right? I'm not sure, but the new paper raises many questions.

The new paper puts together the results of all 54 studies which have looked at this gene in the context of depression, caused by any kind of stress. The authors were intentionally liberal in their inclusion criteria: studies in any population were OK, for example they included people with Parkinson's disease or heart disease.

They say that this is the main difference between the present work and earlier meta-analyses that found no link. The famous 2010 paper, for example, only included 14 studies because they only considered certain kinds of stress.

Anyway, the short variant is associated with depression after all, across all of the studies. They extracted the p values from the results of all previous studies, and took the average of those, weighted by the sample size. They found a very significant association: P=.00002.

Here's all the results. Each square is a study, the further to the left, the more strongly they found an association. Bigger squares mean larger studies. As you can see, most studies found a link but the three largest studies - which were much larger than the others - found none. Hmm.

In terms of specific kinds of stress, they found strong evidence that "specific stressors" (like medical illness), and childhood trauma, were associated with more depression in s-allele carriers. However, in the studies on "Stressful Life Events", which is a broad category meaning pretty much anything bad that happens, the evidence was weaker. The previous meta-analyses only considered these studies.

Ultimately, I think this analysis should remind us that the issue of 5HTTLPR is still "open", but I have concerns about the dataset. The fact that larger studies seem less likely to be positive is a classic warning sign of publication bias.

The authors do consider this and say that they calculate that there would have to be over 700 unpublished, negative studies out there, in order to make the overall data negative. They also find that you could ignore the smallest 45 studies and still find a result. But still. Something doesn't feel right. Maybe I just have the wrong 5HTTLPR variant.

Karg K, Burmeister M, Shedden K, & Sen S (2011). The Serotonin Transporter Promoter Variant (5-HTTLPR), Stress, and Depression Meta-analysis Revisited: Evidence of Genetic Moderation. Archives of general psychiatry, 68 (5), 444-54 PMID: 21199959

Revenge Of The Depression Gene

Last year, the world of psychiatric genetics was rocked by the news that a highly-studied gene, believed to be associated with depression, wasn't in fact linked to depression at all.

The genetic variant was 5-HTTLPR. It's a length variant in the gene coding for the serotonin transporter protein (5HTT) which the target of antidepressants like Prozac. There are two flavors of this variant, short and long.

Many studies have shown that the short ("s") variant is associated with a high risk of getting depression in response to stress - but then last year a large meta-analysis of all the evidence concluded that there was in reality no link. Bummer.

Now another team of researchers have done a new analysis of the 5-HTTLPR & stress & depression data and they claim that there is a link after all: hooray! So who's right? I'm not sure, but the new paper raises many questions.

The new paper puts together the results of all 54 studies which have looked at this gene in the context of depression, caused by any kind of stress. The authors were intentionally liberal in their inclusion criteria: studies in any population were OK, for example they included people with Parkinson's disease or heart disease.

They say that this is the main difference between the present work and earlier meta-analyses that found no link. The famous 2010 paper, for example, only included 14 studies because they only considered certain kinds of stress.

Anyway, the short variant is associated with depression after all, across all of the studies. They extracted the p values from the results of all previous studies, and took the average of those, weighted by the sample size. They found a very significant association: P=.00002.

Here's all the results. Each square is a study, the further to the left, the more strongly they found an association. Bigger squares mean larger studies. As you can see, most studies found a link but the three largest studies - which were much larger than the others - found none. Hmm.

In terms of specific kinds of stress, they found strong evidence that "specific stressors" (like medical illness), and childhood trauma, were associated with more depression in s-allele carriers. However, in the studies on "Stressful Life Events", which is a broad category meaning pretty much anything bad that happens, the evidence was weaker. The previous meta-analyses only considered these studies.

Ultimately, I think this analysis should remind us that the issue of 5HTTLPR is still "open", but I have concerns about the dataset. The fact that larger studies seem less likely to be positive is a classic warning sign of publication bias.

The authors do consider this and say that they calculate that there would have to be over 700 unpublished, negative studies out there, in order to make the overall data negative. They also find that you could ignore the smallest 45 studies and still find a result. But still. Something doesn't feel right. Maybe I just have the wrong 5HTTLPR variant.

Karg K, Burmeister M, Shedden K, & Sen S (2011). The Serotonin Transporter Promoter Variant (5-HTTLPR), Stress, and Depression Meta-analysis Revisited: Evidence of Genetic Moderation. Archives of general psychiatry, 68 (5), 444-54 PMID: 21199959

The Media and Numbers: "It's Complicated".

According to everyone in the British media, 25% of young men are worried about the amount of porn they watch online and men watch an average of 2 hours per week.


Says who? The BBC apparently "teamed up with doctors from the Portman Clinic", a London specialist mental health hospital, to do the study. The actual survey was done online by a certain market research company, which I am not going to name, because they've already got free advertising in every newspaper.

What does this tell us about pornography? Nothing. Dr Petra Boynton explains why in a long and excellent deconstruction. In order to properly interpret these results, we'd need to know lots of details about the study design, which we weren't told. Of course this doesn't stop us from going ahead and interpreting them improperly. 25%! 2 hours. Ooh, that's a lot. Is it? This online porn, eh. Tut tut.

So, sure, 25% could be The True Proportion Of Men Who Worry About Online Porn. Or it might not be. Or the whole question might be so fraught as to be meaningless. The point is, we don't know, we cannot possibly know from the limited amount of information we were given, and we weren't meant to know, because numbers like these are essentially pornographic themselves - they're just for show.

Numbers very rarely make a good news story. When you look into it, the vast majority of them only make sense to people who know all of the background, and by definition, if you have to spend a few pages explaining the background, it's not a good news story. A good news story is one which anyone who can read can immediately understand, and get angry/scared/amused by.

Yet journalists also love numbers because everyone knows, on some level, that numbers matter. The very fact that a story has numbers in it, makes that story better. Indeed, very often, there would be no story without them. Someone doing a survey and finding some numbers can make a news story out of nothing. "Modern online pornography worries some people and is a complicated issue" isn't news; "25% of men..." is news.

So what we end up with is lots of news stories which have numbers in them, but which don't, actually, tell us anything about the world, which is what numbers are supposed to do. Numbers to most of the media are like an attractive trophy wife. They like to be seen with them in public. But deep down they're not all that attached.

The Media and Numbers: "It's Complicated".

According to everyone in the British media, 25% of young men are worried about the amount of porn they watch online and men watch an average of 2 hours per week.


Says who? The BBC apparently "teamed up with doctors from the Portman Clinic", a London specialist mental health hospital, to do the study. The actual survey was done online by a certain market research company, which I am not going to name, because they've already got free advertising in every newspaper.

What does this tell us about pornography? Nothing. Dr Petra Boynton explains why in a long and excellent deconstruction. In order to properly interpret these results, we'd need to know lots of details about the study design, which we weren't told. Of course this doesn't stop us from going ahead and interpreting them improperly. 25%! 2 hours. Ooh, that's a lot. Is it? This online porn, eh. Tut tut.

So, sure, 25% could be The True Proportion Of Men Who Worry About Online Porn. Or it might not be. Or the whole question might be so fraught as to be meaningless. The point is, we don't know, we cannot possibly know from the limited amount of information we were given, and we weren't meant to know, because numbers like these are essentially pornographic themselves - they're just for show.

Numbers very rarely make a good news story. When you look into it, the vast majority of them only make sense to people who know all of the background, and by definition, if you have to spend a few pages explaining the background, it's not a good news story. A good news story is one which anyone who can read can immediately understand, and get angry/scared/amused by.

Yet journalists also love numbers because everyone knows, on some level, that numbers matter. The very fact that a story has numbers in it, makes that story better. Indeed, very often, there would be no story without them. Someone doing a survey and finding some numbers can make a news story out of nothing. "Modern online pornography worries some people and is a complicated issue" isn't news; "25% of men..." is news.

So what we end up with is lots of news stories which have numbers in them, but which don't, actually, tell us anything about the world, which is what numbers are supposed to do. Numbers to most of the media are like an attractive trophy wife. They like to be seen with them in public. But deep down they're not all that attached.

The Tufnel Effect

In This Is Spin̈al Tap, British heavy metal god Nigel Tufnel says, in reference to one of his band's less succesful creations:

It's such a fine line between stupid and...uh, clever.

This is all too true when it comes to science. You can design a breathtakingly clever experiment, using state of the art methods to address a really interesting and important question. And then at the end you realize that you forgot to type one word when writing the 1,000 lines of software code that runs this whole thing, and as a result, the whole thing's a bust.

It happens all too often. It has happened to me, let me think, three times in my scientific career and, I know of several colleagues who had similar problems and I'm currently struggling to deal with the consequences of someone else's stupid mistake.

Here's my cautionary tale. I once ran an experiment involving giving people a drug or placebo and when I crunched the numbers I found, or thought I'd found, a really interesting effect which was consistent with a lot of previous work giving this drug to animals. How cool is that?

So I set about writing it up and told my supervisor and all my colleagues. Awesome.

About two or three months later, for some reason I decided to reopen the data file, which was in Microsoft Excel, to look something up. I happened to notice something rather odd - one of the experimental subjects, who I remembered by name, was listed with a date-of-birth which seemed wrong: they weren't nearly that old.

Slightly confused - but not worried yet - I looked at all the other names and dates of birth and, oh dear, they were all wrong. But why?

Then it dawned on me and now I was worried: the dates were all correct but they were lined up with the wrong names. In an instant I saw the horrible possibility: m ixed up names would be harmless in themselves but what if the group assignments (1 = drug, 0 = placebo) were lined up with the wrong results? That would render the whole analysis invalid... and oh dear. They were.

As the temperature of my blood plummeted I got up and lurched over to my filing cabinet where the raw data was stored on paper. It was deceptively easy to correct the mix-up and put the data back together. I re-ran the analysis.

No drug effect.

I checked it over and over. Everything was completely watertight - now. I went home. I didn't eat and I didn't sleep much. The next morning I broke the news to my supervisor. Writing that email was one of the hardest things I've ever done.

What happened? As mentioned I had been doing all the analysis in Excel. Excel is not a bad stats package and it's very easy to use but the problem is that it's too easy: it just does whatever you tell it to do, even if this is stupid.

In my data as in most people's, each row was one sample (i.e. a person) and each column was a piece of info. What happened was that I'd tried to take all the data, which was in no particular order, and reorder the rows alphabetically by subject name to make it easier to read.

How could I screw that up? Well, by trying to select "all the data" but actually only selecting a few of the columns. Then I reordered them, but not the others, so all the rows became mixed up. And the crucial column, drug=1 placebo=0, was one of the ones I reordered.

The immediate lesson I learned from this was: don't use Excel, use SPSS, which simply does not allow you to reorder only some of the data. Actually, I still use Excel for making graphs and figures but every time I use it, I think back to that terrible day.

The broader lesson though is that if you're doing something which involves 100 steps, it only takes 1 mistake to render the other 99 irrelevant. This is true in all fields but I think it's especially bad in science, because mistakes can so easily go unnoticed due to the complexity of the data, and the consequences are severe because of the long time-scale of scientific projects.


Here's what I've learned: Look at your data, every step of the way, and look at your methods, every time you use them. If you're doing a neuroimaging study, the first thing you do after you collect the brain scans is to open them up and just look at them. Do they look sensible?

Analyze your data as you go along. Every time some new results come in, put it into your data table and just look at it. Make a graph which just shows absolutely every number all on one massive, meaningless line from Age to Cigarette's Smoked Per Week to EEG Alpha Frequency At Time 58. For every subject. Get to know the data. That way if something weird happens to it, you'll know. Don't wait to the end of the study to do the analysis. And don't rely on just your own judgement - show your data to other experts.

Check and recheck your methods as you go along. If you're running, say, a psychological experiment involving showing people pictures and getting them to push buttons, put yourself in the hot seat and try it on yourself. Not just once, but over and over. Some of the most insidious problems with these kinds of studies will go unnoticed if you only look at the task once - such as the old "randomized"-stimuli-that-aren't-random issue.

Trust no-one. This sounds bad, but it's not. Don't rely on their work, in experimental design or data analysis, until you've checked it yourself. This doesn't mean you're assuming they're stupid, because everyone makes these mistakes. It just means you're assuming they're human like you.

Finally, if the worst happens and you discover a stupid mistake in your own work: admit it. It feels like the end of the world when this happens, but it's not. However, if you don't admit it, or even worse, start fiddling other results to cover it up - that's misconduct, and if you get caught doing that, it is the end of the world, or your career, at any rate.

The Tufnel Effect

In This Is Spin̈al Tap, British heavy metal god Nigel Tufnel says, in reference to one of his band's less succesful creations:

It's such a fine line between stupid and...uh, clever.

This is all too true when it comes to science. You can design a breathtakingly clever experiment, using state of the art methods to address a really interesting and important question. And then at the end you realize that you forgot to type one word when writing the 1,000 lines of software code that runs this whole thing, and as a result, the whole thing's a bust.

It happens all too often. It has happened to me, let me think, three times in my scientific career and, I know of several colleagues who had similar problems and I'm currently struggling to deal with the consequences of someone else's stupid mistake.

Here's my cautionary tale. I once ran an experiment involving giving people a drug or placebo and when I crunched the numbers I found, or thought I'd found, a really interesting effect which was consistent with a lot of previous work giving this drug to animals. How cool is that?

So I set about writing it up and told my supervisor and all my colleagues. Awesome.

About two or three months later, for some reason I decided to reopen the data file, which was in Microsoft Excel, to look something up. I happened to notice something rather odd - one of the experimental subjects, who I remembered by name, was listed with a date-of-birth which seemed wrong: they weren't nearly that old.

Slightly confused - but not worried yet - I looked at all the other names and dates of birth and, oh dear, they were all wrong. But why?

Then it dawned on me and now I was worried: the dates were all correct but they were lined up with the wrong names. In an instant I saw the horrible possibility: m ixed up names would be harmless in themselves but what if the group assignments (1 = drug, 0 = placebo) were lined up with the wrong results? That would render the whole analysis invalid... and oh dear. They were.

As the temperature of my blood plummeted I got up and lurched over to my filing cabinet where the raw data was stored on paper. It was deceptively easy to correct the mix-up and put the data back together. I re-ran the analysis.

No drug effect.

I checked it over and over. Everything was completely watertight - now. I went home. I didn't eat and I didn't sleep much. The next morning I broke the news to my supervisor. Writing that email was one of the hardest things I've ever done.

What happened? As mentioned I had been doing all the analysis in Excel. Excel is not a bad stats package and it's very easy to use but the problem is that it's too easy: it just does whatever you tell it to do, even if this is stupid.

In my data as in most people's, each row was one sample (i.e. a person) and each column was a piece of info. What happened was that I'd tried to take all the data, which was in no particular order, and reorder the rows alphabetically by subject name to make it easier to read.

How could I screw that up? Well, by trying to select "all the data" but actually only selecting a few of the columns. Then I reordered them, but not the others, so all the rows became mixed up. And the crucial column, drug=1 placebo=0, was one of the ones I reordered.

The immediate lesson I learned from this was: don't use Excel, use SPSS, which simply does not allow you to reorder only some of the data. Actually, I still use Excel for making graphs and figures but every time I use it, I think back to that terrible day.

The broader lesson though is that if you're doing something which involves 100 steps, it only takes 1 mistake to render the other 99 irrelevant. This is true in all fields but I think it's especially bad in science, because mistakes can so easily go unnoticed due to the complexity of the data, and the consequences are severe because of the long time-scale of scientific projects.


Here's what I've learned: Look at your data, every step of the way, and look at your methods, every time you use them. If you're doing a neuroimaging study, the first thing you do after you collect the brain scans is to open them up and just look at them. Do they look sensible?

Analyze your data as you go along. Every time some new results come in, put it into your data table and just look at it. Make a graph which just shows absolutely every number all on one massive, meaningless line from Age to Cigarette's Smoked Per Week to EEG Alpha Frequency At Time 58. For every subject. Get to know the data. That way if something weird happens to it, you'll know. Don't wait to the end of the study to do the analysis. And don't rely on just your own judgement - show your data to other experts.

Check and recheck your methods as you go along. If you're running, say, a psychological experiment involving showing people pictures and getting them to push buttons, put yourself in the hot seat and try it on yourself. Not just once, but over and over. Some of the most insidious problems with these kinds of studies will go unnoticed if you only look at the task once - such as the old "randomized"-stimuli-that-aren't-random issue.

Trust no-one. This sounds bad, but it's not. Don't rely on their work, in experimental design or data analysis, until you've checked it yourself. This doesn't mean you're assuming they're stupid, because everyone makes these mistakes. It just means you're assuming they're human like you.

Finally, if the worst happens and you discover a stupid mistake in your own work: admit it. It feels like the end of the world when this happens, but it's not. However, if you don't admit it, or even worse, start fiddling other results to cover it up - that's misconduct, and if you get caught doing that, it is the end of the world, or your career, at any rate.

The Decline And Fall of Effects In Science

Nature has a piece called Unpublished results hide the decline effect.
This refers to the fact that many scientific findings which seem to indicate something big is happening, end up getting smaller and smaller as more people try to replicate them until they, eventually, may vanish entirely.

The Last Psychiatrist's take is that "The Decline Effect" just represents sloppy thinking, treating different things as if they were all instances of The One True Phenomenon. Someone does a study about something and finds an effect. Then someone else comes along and does a new study, of a related but different topic, and finds a different result. Both are right: there's a difference. Only if you, sloppily, decide that both studies were measuring the same thing does the "Decline Effect" appear.

This is perfectly true and I've touched on it before, but I think it's a bit optimistic. It assumes that the first study was true. Sometimes they are. But because of the way science is published at the moment, a lot of results that get published are flukes. Some even say that the majority are.

The problem is that there are so many ways to statistically analyze any given body of data that it's easy to test and retest it until you find a "positive result" - and then publish that, without saying (or only saying in the small print) that your original tests all came out negative. Combine this with selective publication of only the best data, and other scientific sins, and you can pull positive results out the hat of mere random noise.

In the Nature article, Jonathan Schooler discusses this and suggests that an open-access repository of findings (meaning raw data rather than the end product of analyses) would be A Good Thing. I agree. However, he seems to think that if we did this, we might still observe the "Decline Effect", and would be able to find out more about it. He even seems to suggest that some kind of weird quantum effect might mean that scientists are actually changing the laws of reality by observing them

Perhaps, just as the act of observation has been suggested to affect quantum measurements, scientific observation could subtly change some scientific effects. Although the laws of reality are usually understood to be immutable, some physicists, including Paul Davies, director of the BEYOND: Center for Fundamental Concepts in Science at Arizona State University in Tempe, have observed that this should be considered an assumption, not a foregone conclusion.

Hmm. Maybe. But there is really no need to posit such magical mysteries when plain old statistical conjuring tricks seem like a perfectly good explanation. On my view a raw result repository would not explain the decline effect, but just make it disappear.

Schooler doesn't go into detail as to how this repository would be set up, but he does cite the fact that we already have a pretty good one for clinical trials of medicines conducted in the USA. Anyone running a clinical trial is required to register it in advance, saying what they're planning to do and crucially, to spell out which statistics they are going to run on the data when it arrives.

What's really silly is that most scientists already do this when applying for funding: most grant applications include detailed statistical protocols. The problem is that these are not made public so people can ignore them when it comes to publication. Back in 2008 I suggested that scientific journals should require all studies, not just clinical trials, to be publicly pre-registered if they're to be considered for publication. This would be eminently do-able if there was a will to make it happen.

Schooler, J. (2011). Unpublished results hide the decline effect Nature, 470 (7335), 437-437 DOI: 10.1038/470437a

The Decline And Fall of Effects In Science

Nature has a piece called Unpublished results hide the decline effect.
This refers to the fact that many scientific findings which seem to indicate something big is happening, end up getting smaller and smaller as more people try to replicate them until they, eventually, may vanish entirely.

The Last Psychiatrist's take is that "The Decline Effect" just represents sloppy thinking, treating different things as if they were all instances of The One True Phenomenon. Someone does a study about something and finds an effect. Then someone else comes along and does a new study, of a related but different topic, and finds a different result. Both are right: there's a difference. Only if you, sloppily, decide that both studies were measuring the same thing does the "Decline Effect" appear.

This is perfectly true and I've touched on it before, but I think it's a bit optimistic. It assumes that the first study was true. Sometimes they are. But because of the way science is published at the moment, a lot of results that get published are flukes. Some even say that the majority are.

The problem is that there are so many ways to statistically analyze any given body of data that it's easy to test and retest it until you find a "positive result" - and then publish that, without saying (or only saying in the small print) that your original tests all came out negative. Combine this with selective publication of only the best data, and other scientific sins, and you can pull positive results out the hat of mere random noise.

In the Nature article, Jonathan Schooler discusses this and suggests that an open-access repository of findings (meaning raw data rather than the end product of analyses) would be A Good Thing. I agree. However, he seems to think that if we did this, we might still observe the "Decline Effect", and would be able to find out more about it. He even seems to suggest that some kind of weird quantum effect might mean that scientists are actually changing the laws of reality by observing them

Perhaps, just as the act of observation has been suggested to affect quantum measurements, scientific observation could subtly change some scientific effects. Although the laws of reality are usually understood to be immutable, some physicists, including Paul Davies, director of the BEYOND: Center for Fundamental Concepts in Science at Arizona State University in Tempe, have observed that this should be considered an assumption, not a foregone conclusion.

Hmm. Maybe. But there is really no need to posit such magical mysteries when plain old statistical conjuring tricks seem like a perfectly good explanation. On my view a raw result repository would not explain the decline effect, but just make it disappear.

Schooler doesn't go into detail as to how this repository would be set up, but he does cite the fact that we already have a pretty good one for clinical trials of medicines conducted in the USA. Anyone running a clinical trial is required to register it in advance, saying what they're planning to do and crucially, to spell out which statistics they are going to run on the data when it arrives.

What's really silly is that most scientists already do this when applying for funding: most grant applications include detailed statistical protocols. The problem is that these are not made public so people can ignore them when it comes to publication. Back in 2008 I suggested that scientific journals should require all studies, not just clinical trials, to be publicly pre-registered if they're to be considered for publication. This would be eminently do-able if there was a will to make it happen.

Schooler, J. (2011). Unpublished results hide the decline effect Nature, 470 (7335), 437-437 DOI: 10.1038/470437a