Andrew McG responds to my Random Observations . He was heading not entirely down my path, but he makes a good point that’s worth repeating. Andrew references the Monty Hall problem .

Your paper will include a calculation of significance. This is essentially an estimate of the probability that a correlation as strong as the one you found would exist purely as a result of randomness in the data, even if your theory is false.

This calculation assumes the "proper" sequence of events. You have a theory, and you test the data for a correlation. Since you in fact poked around for correlations, then came up with a theory, the significance calculation is not valid. The true significance depends on the probability that, having found a randomly-caused correlation somewhere, you can then invent a theory to explain it. That probability is very difficult to estimate, but is probably much greater - meaning that the significance of the correlation is much smaller.

As you say, theories aren't formed in a vacuum, and so there is not such a clear division between the "right" way of doing it and the "wrong" way of doing it. Nobody is completely ignorant of the data when they start to theorize. That is a real problem with nearly all statistics-based results that are published today. They are all presented with significance calculations based on the assumption that the forming of the theory was independent of the data - an assumption that is very unlikely to be completely true. Therefore nearly every significance estimate published is an overestimate.

I believe the issue Andrew mentions to be very nearly universal. In order to publish, I must read and reference the related literature. That action, alone, nearly guarantees that I am NOT ignorant of the data.

In light of this fact, I am occasionally dismayed when reviewers demand a particular type of “purity” in the scientific process, which, I suppose, was the point of my original post.

I also find it blackly amusing when we bring such arguments to bear against research that contradicts our preferences, but tend not to use such arguments against research the conforms to out preferences. Yes, I’m guilty, too.