Wednesday, August 24, 2011

When is a conclusion good enough?

I've been doing a lot of reading recently on modern probability theory, Bayesian analysis, and information theory. One of the central tenets of these theories is that any proposition has associated with it a degree of plausibility, i.e. a probability of being true, that reflects the amount of information available. The proposition, "the sky is blue" is extremely plausible since it is based off of daily self-observations and confirmed by others. As another example, I judge the proposition, "thirty million Americans have blue eyes" as plausible based on the knowledge of the current population of the United States and my own observations on the frequency of encountering blue-eyed people. However, this is not as likely to be true as the first statement.

A conclusion in a scientific paper is nothing more than a proposition and thus possesses its own degree of plausibility. The information available to the reader for determining the plausibility of the conclusion is the data presented in the paper and all previously published work on the same topic. Of course, other factors weigh in, such as prejudices for particular theories and affinities or dislikes for the authors on the paper. Temporarily placing these other factors aside, I wonder, "when does a conclusion possess a large enough degree of plausibility to be considered true?"

Of course, a definite answer doesn't exist. No scientific proposition can be true with 100% certainty and it is silly to think that we can even assign a threshold probability for evaluating a scientific paper's correctness. Just imagine a paper successfully passing through the peer-review process so long as it is evaluated to be 78.63% or more true by its reviewers. But the question remains relevant. Science is a culture and every culture has criteria by which it evaluates claims.

In a closely-related post I wrote about the fallacies that workers commit while evaluating other work. But I find it much more difficult to identify the criteria for establishing the truthfulness and quality* of research. Unfortunately, I don't think I'll be able to fulfill my full potential as a scientist until I am capable of doing so.

Note: The problem of defining quality has been approached at great length by American author Robert Pirsig in his popular novel "Zen and the Art of Motorcycle Maintenance." One of his primary arguments is that Quality is actually an undefined construct present at the seminal moment when an observation is made and processed by the brain. People may know if something possesses Quality, but it is inherently undefinable.