How To Fix Science

Over at Bad Science, Ben Goldacre discusses a big problem with modern science - the published literature is all very well and good, but we don't know what people are finding that goes unpublished:

The scale of the academic universe is dizzying, after all. Our most recent estimate is that there are over 24,000 academic journals in existence, 1.3 million academic papers published every year, and over 50 million papers published since scholarship began.

And for every one of these 50 million papers there will be unknowable quantities of blind alleys, abandoned experiments, conference presentations, work in progress seminars, and more. Look at the vast number of undergraduate and masters dissertations that had an interesting finding, and got turned into finished academic papers: and then think about the even vaster number that don’t...

We are living in the age of information, and vast tracts of data are being generated around the world on every continent and every question. A £200 laptop will let you run endless statistical analyses. The most interesting questions aren’t around individual nuggets of data, but rather how we can corral it to create an information architecture which serves up the whole picture.

I agree with all of this. It is a problem. In fact I'd say it's the single biggest problem with science today. Scientists are required to publish ever-increasing numbers of high-impact papers, in order to get grants and promotions, with the "best" papers, usually meaning the ones with the most interesting positive results, being favored.

Findings that show that nothing especially interesting is going on here all too often get swept under the carpet or re-re-analyzed until a positive result falls out. If you do a study of a certain gene and its association to brain function, say, and find it has no association: that's bad news for you. That will make a low-impact paper, if it makes a paper at all. But maybe it has an association with brain structure? Or personality?

Anyway, that's the problem. What to do about it? Goldacre notes that in medicine, there are mechanisms in place to deal with this:

In medicine, where the stakes are tangible, systems have grown up to try and cope with this problem: trials are supposed registered before they begin, so we can notice the results that get left unpublished. But even here, the systems are imperfect; and pre-registration is very rarely done, even in medical research, for anything other than trials.

Clinical trial pre-registration is a fantastic idea. The systems are certainly imperfect, but they're getting better, and they're much better than nothing. Back in 2008 I proposed that all scientific studies, not just clinical trials, should be publically pre-registered. That way everyone could know what science was going unpublished and could tell when authors were doing analyzes they hadn't originally planned to do (which is fine, so long as you admit to it.)

I still think that would be a good idea. But how would it work in practice? Here's what I've come up with:

Scientific papers should be submitted to journals for publication before the research has started. The Introduction and the Methods section, detailing what you plan to do and why, would then get peer reviewed. The rest of the paper would obviously be a blank at this stage. Anonymous experts would have a chance to critique the methods and rationale.

If the paper's accepted, you then do the research, get the results, and write the Results and Discussion section of the paper. The journal is then required to publish the final paper, assuming that you kept to the original plan. The Introducion and primary Methods would be fixed - you can't change them once the data come in.

You can do additional stuff and run additional analyses all you like, but they'll be marked as secondary, which of course is what they are. Publication would therefore be based on the scientific merits of the experiment, the importance of the question and the quality of the methods, not the "interestingness" of the results. If you want a paper in Nature, it needs to be a great idea, not a lucky shot.

This would be a radical change from the current system. Too radical, almost certainly, to ever happen in one go. So here's another idea as to a kind of stepping-stone on the way:

Already, scientists have to spell out their original rationale and original methods before they do any work - when they apply for funding from a grant awarding body. These grant applications are often very detailed, but at the moment, they're private. And people don't always stick to them.

Why not make the full publication of the grant application a condition of being awarded the money? This would be rather like preregistration of the Introduction and Methods, although less elegant, but it would do the job. And given that most grants consist of public cash, the public really have a right to know this. These applications are usually just PDF files. It would be trivial to put them online - after redacting personal information like applicant résumés, if desired.