The statistical point that neuroscience papers get wrong half the time

Ben Goldacre explains.

The explanation does take a few dense paragraphs; Goldacre admits his own writing is going to cause some "pain" for readers, and I had to read it twice before I got it. But it's worth it for anyone interested in brain studies.

UPDATE: I posted the article to Metafilter, where a commenter named "valkyryn" takes another shot at explaining it:

Say we've got two samples, A and B. For our sample size and subject matter, if we expose it to chemical X, any detected results need to be above 20 Units (U) to be statistically significant.

We expose A to X, and we get a result of 30U. That's statistically significant.

We expose B to X, and we get a result of 15U. That isn't statistically significant.

But note that the result for A is less than 20U from the result for B. This means that while we did have a statistically significant finding for A, the difference between A and B is not statistically significant.

This significantly diminishes the value of our findings, as the math only supports a relatively unambitious claim. We want to say that A and B are different, but the math won't let us, as the difference in reaction between A and B is too small. We're left with a relatively uninteresting finding, namely that something seems to happen with A and X, but we aren't sure how much that matters.

Careers tend not to be made out of this kind of finding, and the author is implying that because scientists have an interest in careers, they're making more ambitious claims than their studies actually permit.

IN THE COMMENTS: LemmusLemmus illustrates the problem with this graph:

LemmusLemmus explains:

The result for condition A is significantly different from zero (i.e., it "is significant"), while the result for condition B is not (the confidence interval for A does not include zero, but the confidence interval for B does). However, the two confidence intervals overlap, which means that the results for conditions A and B are not significantly different from each other.