Of Cabbages and Kings and Confirmation Bias

source

"It is a good morning exercise for a research scientist to discard a pet hypothesis every day before breakfast."

- Konrad Lorenz, On Aggression

Lately I've been thinking more about skepticism in light of some reporting on scientific findings and the scientific method. The idea that we should not be overly invested in our thoughts, and that we should subject them to the utmost scrutiny, is a fundamentally important one, I think.

Good science has to meet a few criteria, namely that the results of any particular study should be reproducible, or that another scientist replicating the conditions of the first study will obtain the same result. The replication (or reproducibility) crisis in psychology underscores the importance of questioning what we think we know, whether it's power-posing or gender-priming. Our investment in our own ideas can quickly lead to confirmation bias, whereby we ignore evidence that contradicts what we already think is true.

The problem with scientific reproducibility is related to the gaming of statistical power (or p-values). As Paul Smaldino recently pointed out in The Guardian, it doesn't take "bad" scientists to produce bad science, just those who are under enough pressure to produce something, anything, with sufficient significance to be published. When sample sizes are small, there are random effects that can produce something that looks significant but that may evaporate upon further investigation. Scientists can choose to focus on such anomalies without subjecting their findings to additional tests before publishing them. It's difficult to guard against such bad scientific practices in the context of ever-increasing pressure on academics to produce articles and grant proposals.

The best thing that we can do to fight against degrading scientific standards is to question ourselves and each other, and to be open to such criticism. As scientists we might sometimes hold on too tightly to our pet ideas because of our egos, or because we are literally invested in an idea through grant funding and recognition of our work. But the best science is that which has stood the test of scrutiny, and we should be glad to discard bad ideas, especially our own.