How not to argue in science

I'd like to expand a little on yesterday's post. I'm beginning to better understand what constitutes proper debate of a scientific work. Whether the following logic is actually practiced by most researchers is questionable, but this is nevertheless an interesting and important point.

Data from a study provides information that does one of three fairly obvious things to a conclusion: it increases, decreases, or leaves unaffected the likelihood that the conclusion is correct. I place emphasis on the word likelihood because any given conclusion can not be demonstrated as being correct with 100% certainty, and I highly doubt that conclusions can be proven false with certainty.

I think that this—the likelihood that a conclusion is correct given all information—as well as the competency with which an experiment was performed are the two objects open to debate within science. The debate becomes unscientific when researchers and journal reviewers perform the following errors:

1. Assigning too much weight to prior information, thus making the likelihood that another work's results are correct less likely then it perhaps should be.
2. As a corollary to the first point, workers would be in error if they didn't properly balance the weighting of all prior information. For example, the media, in their coverage of climate change, has been chastised by some for giving equal attention to climate change skeptics as they do to proponents. This is because the proportion of scientists against climate change is significantly fewer than the proportion who see it as a true occurrence.
3. Assuming that a finding is false given prior information or prejudices. If one accepts that a finding can not be false but rather highly unlikely, then arguing to reject a journal article because it contradicts previous findings is itself fallacious. The wider scientific community should (with its more balanced opinions) be a better interpreter of the likelihood that the claims are real.

Of course, if these errors were corrected, they could very well lead to many more published works, which would in turn dilute the field. As a result, grants may be harder to obtain (since they are in part based on published works) and the dissemination of knowledge would become greatly impaired; there would simply be too much information to analyze.

Uniqueness of probability allows for assertion

But until it had been demonstrated that [the probabilities] are uniquely determined by the data of a problem, we had no grounds for supposing that [the probabilities] were possessed of any precise meaning.--E. T. Jaynes and G. Larry Bretthorst, from "Probability Theory: The Logic of Science (emphasis mine)

I take this to mean that any experiment whose results are debated is under question because either 1) the logic of the critics is flawed or 2) there is not enough information in the data to reach the argued for conclusion to a large degree of plausibility. Uniqueness of the probabilities assures this. Furthermore, there can be no argument for the truth or falsehood of the claims.

More notes from Jaynes

The introduction to Probability Theory: The Logic of Science has been useful for explaining what various statistical procedures are used for when making an inference about data.

Maximum entropy is a technique used to establish probabilities for outcomes from data given no prior information or assumptions. It is essentially an algorithm that comes to a conclusion without any bias from the experimenter. Bayesian techniques, on the other hand, require some prior information, and this will affect the conclusion.

Typically, when performing acts of inference, one begins with maximum entropy if very little is known except what's given in the data. Once more is known, one may turn to Bayesian analysis.

Bayesian analysis requires five things: a model, sample space, hypothesis space, prior probabilities, and sampling distributions.

There is much work to be done in developing techniques when even little is known about the raw data; this could lead to steps that can assist in studies where even maximum entropy may fail to give adequate results.

Coming to conclusions

In the introduction of E. T. Jaynes's Probability Theory: The Logic of Science, Jaynes states

...the emphasis was therefore on the quantitative formulation of Polya’s viewpoint, so it could be used for general problems of scientific inference, almost all of which arise out of incomplete information rather than ‘randomness’.

As I learn more about the field of sensing, I find that this is the mentality, whether acknowledged by a practitioner or not, that is adopted when coming to a conclusion about the interpretation of data. The uncertainty involved in coming to a conclusion is not because the measurement process is inherently random but rather that one has not collected enough data to say whether this conclusion is true or false.

And in the light of Bayesian analysis, one will never be able to claim with 100% certainty that the conclusion is true (or false, for that matter).

Fighting through boredom

Graduate school can often be frustrating, disruptive, and downright stressful. I probably did not need to state this since all graduate students are aware of this fact. I have been especially bored with my graduate studies lately and have been struggling to identify both the cause and a solution. Through talking with friends and lots of time thinking (especially during my recent vacation to the UK), I've slowly been able to reason out the cause.

Quite simply, I've forgotten my large-scale fascination with science. As an undergrad, I would marvel at every piece of popular physics literature I read, from discoveries at particle accelerators to the development of new nanotechnologies. In graduate school, I've become so mired in one specific area of research that I forgot that very cool things are happening all over the scientific world, such as the "bump" in the data seen at Fermilab.

Though not the only reason for my recent lull, it is a major one. And it points to a solution: take time out of my day to peruse the myriad of popular scientific articles and rekindle my interests. Though I may not be working on these famous projects, I find that I am much happier in the lab after having contemplated these things. They place my work within a greater context, and though I tend to be an individualist, I think that I at least need to do this to find satisfaction with my own work.