Why I like Dr. Ben Goldacre

I've been trying to keep my posts to once a week on Wednesdays, but sometimes I simply just want to share something interesting. In a recent Bad Science post, Dr. Ben Goldacre discussed sampling error in relation to unemployment figures in the UK. The article itself is interesting, but I found the description of systematic sampling error especially amusing:

Firstly, you’ll be familiar with the idea that a sample can be systematically unrepresentative: if you want to know about the health of the population as a whole, but you survey people in a GP waiting room, then you’re an idiot.

The great physicist Subrahmanyan Chandrasekhar gave a famous lecture at the International Symposium in Honor of Robert R. Wilson in April, 1979 entitled "Beauty and the Quest for Beauty in Science." In the second paragraph of the lecture, he quotes Poincaré:

The Scientist does not study nature because it is useful to do so. He studies it because he takes pleasure in it; and he takes pleasure in it because it is beautiful. If nature were not beautiful, it would not be worth knowing and life would not be worth living.... I mean the intimate beauty which comes from the harmonious order of its parts and which a pure intelligence can grasp....

It is because simplicity and vastness are both beautiful that we seek by preference simple facts and vast facts; that we take delight, now in following the giant courses of the stars, now in scrutinizing with a microscope that prodigious smallness which is also a vastness, and, now in seeking in geological ages the traces of the past that attracts us because of its remoteness.

Good science is not forced and does not evolve from long hours in the lab and a work-centric lifestyle alone, though I admit that these are necessary to maintain a healthy scientific career. Rather, science is intimately related to one of the most basic of human traits: the impassioned drive to create order out of chaos.

And this is why, despite my affection for Poincaré, I disagree with part of the above quote. Beauty does not exist within Nature because its parts are inherently balanced and "harmonious." Beauty exists as a result of the mind imparting its own structure to random titillations of the senses. Men and women do not render Nature meaningful through physical exertion alone. The reduction of Nature to coherent understanding is the realm of Science.

What a timely survey

Nature Jobs posted an article last week that summarizes the results of a survey conducted on graduate students in the sciences about their satisfaction with graduate school. Here are some of the more interesting points:

1. 78.8% of first year PhD students responded that they were "very" or "quite" likely to continue on to a university research position after graduate school.  In comparison 62% of fifth years answered the same.
2. Competition is the biggest factor in steering students away from academic careers.
3. 44.6% of students thought about post-school career options before entering graduate school.
4. 71.6% of European-based students reported that they were somewhat or very satisfied with their overall graduate school experience, compared to just 57.1% in the US and 62.3% in Japan.

This last point is interesting because I recently had a discussion about it with my advisor (who is Romanian). He claims that the reason for the significant difference between US and European students concerning the level of satisfaction with  their graduate studies is the perception of education in the two cultures. Education is held in high esteem in many European countries and professors carry a highly-valued social status.

In contrast, scientific jobs often carry a certain stigma with them in the US. Amusingly, I had a travel buddy--an engineer for an aerospace company--once explain to me what lengths he went to to hide the fact that he studied engineering when hitting on girls in college.

I'm not certain that being an intellectual is a turn-off, however. Consider the tech industry (Google, etc.) and the fairly well-received social status that its employees enjoy despite their aptitude for computers and technology. The real substance here is that they also enjoy good incomes and a stable job. So, the point is that intelligence is not socially undesirable; rather, it's that a good career and income is more highly valued.

The survey's findings seem to corroborate my recent conclusions about life as a graduate student. I must admit that I'm a little jealous of my engineer friends who went to work immediately after school. They're making much more money, have more free time, and are generally moving forward with their lives at a comfortable pace. And since we came from the same educational background, it's sometimes demoralizing to consider where I'm at with my life and find that I'm lagging them in these respects due to my role as a graduate student.

But I'll be damned if a job in industry can elicit the same oh-so-sweet feeling when an experiment works, the results fit neatly within the model, and one little mystery of nature finds itself tamed by my own cognitive exertions that is rewarded by a career in science.

What I wish I had known about academia (before I entered graduate school)

I'm entering my fifth year of graduate school this upcoming semester and, accordingly, have been increasingly thinking about my life afterward. Moments of reflection and talks with other students have revealed that there are a good number of things that I was unaware of concerning a career in academia when I began my graduate studies. Most of these things have taken me a long time to learn because the points were subtle or I was too naive to honestly assess the matter. Though these thoughts may not be true by the actual numbers (e. g. I haven't looked at the availability of teaching positions or the average income of post-docs), they certainly have found other voices, such as a few of the authors in this April Nature issue. And since the thought of a large number of people holds a good deal of influence regardless of its validity, I will take these thoughts to be true and offer them as advice to those who are considering a career in academia.

Career Point Number One: A career in academia—specifically in science—requires more hard work and dedication than most careers. This is due to a number of reasons, including a saturation of workers in the field, competition over resources, and a career trajectory that is difficult to advance through. Too many people within academia could be considered the cause of the competition over resources, but it's significant in its own right since it dilutes the quality of work being done. And as for a difficult career trajectory: a colleague told me the scariest thing one could do of all career moves is enter upon an assistant professorship with a family and mortgage with no guarantee of tenure.


CPN Two: Teaching positions are sparse (this was a surprise to me!). Many tenured professors or industry veterans enjoy retiring into academic teaching positions. There will not be much leverage for new graduates in obtaining a desired teaching position against those more experienced in the field.

CPN Three: No matter how amiable your advisor, it is not in their personal interest to graduate students. Losing experienced and knowledgeable students hurts their ability to publish, obtain funding, and generally proceed through their own academic career paths.

CPN Four: A post-doc may not be the best option for advancing a scientific/academic career. Many advisors interpret the post-doc role as one similar to a graduate student's but unencumbered by educational burdens such as attending class. A move to industry following graduation, however, may provide better networking opportunities and more chances to evolve one's professional skill set. One can always return to academia.

CP Five: One need not work in academia to retain one's interest in science. This was in no way obvious to me from the start. After having identified my interest in physics, I proceeded through my education with the idea that physics would be my career. But herein lies the most important distinction of all: science is not a career.

Science is the pursuit of truth and the delight in discovering new things through experimentation. It need not be grand in scale or require a large amount of resources. It is an unfortunate development that a career in academia and science have assumed the same position in most people's minds.

I admit that this assessment may appear a bit bleak at first, but for me it is rather liberating. I'm OK finding a career that is not centered squarely within academia because the requirements of such a job are too demanding given my other interests. This in no way means that my life or work can not contribute significantly to science. But to conclude that academia alone is the only way to significantly impact science is to commit a fallacy that could launch one onto a difficult and unsatisfying career path.

How not to argue in science

I'd like to expand a little on yesterday's post. I'm beginning to better understand what constitutes proper debate of a scientific work. Whether the following logic is actually practiced by most researchers is questionable, but this is nevertheless an interesting and important point.

Data from a study provides information that does one of three fairly obvious things to a conclusion: it increases, decreases, or leaves unaffected the likelihood that the conclusion is correct. I place emphasis on the word likelihood because any given conclusion can not be demonstrated as being correct with 100% certainty, and I highly doubt that conclusions can be proven false with certainty.

I think that this—the likelihood that a conclusion is correct given all information—as well as the competency with which an experiment was performed are the two objects open to debate within science. The debate becomes unscientific when researchers and journal reviewers perform the following errors:

1. Assigning too much weight to prior information, thus making the likelihood that another work's results are correct less likely then it perhaps should be.
2. As a corollary to the first point, workers would be in error if they didn't properly balance the weighting of all prior information. For example, the media, in their coverage of climate change, has been chastised by some for giving equal attention to climate change skeptics as they do to proponents. This is because the proportion of scientists against climate change is significantly fewer than the proportion who see it as a true occurrence.
3. Assuming that a finding is false given prior information or prejudices. If one accepts that a finding can not be false but rather highly unlikely, then arguing to reject a journal article because it contradicts previous findings is itself fallacious. The wider scientific community should (with its more balanced opinions) be a better interpreter of the likelihood that the claims are real.

Of course, if these errors were corrected, they could very well lead to many more published works, which would in turn dilute the field. As a result, grants may be harder to obtain (since they are in part based on published works) and the dissemination of knowledge would become greatly impaired; there would simply be too much information to analyze.

Uniqueness of probability allows for assertion

But until it had been demonstrated that [the probabilities] are uniquely determined by the data of a problem, we had no grounds for supposing that [the probabilities] were possessed of any precise meaning.--E. T. Jaynes and G. Larry Bretthorst, from "Probability Theory: The Logic of Science (emphasis mine)

I take this to mean that any experiment whose results are debated is under question because either 1) the logic of the critics is flawed or 2) there is not enough information in the data to reach the argued for conclusion to a large degree of plausibility. Uniqueness of the probabilities assures this. Furthermore, there can be no argument for the truth or falsehood of the claims.

More notes from Jaynes

The introduction to Probability Theory: The Logic of Science has been useful for explaining what various statistical procedures are used for when making an inference about data.

Maximum entropy is a technique used to establish probabilities for outcomes from data given no prior information or assumptions. It is essentially an algorithm that comes to a conclusion without any bias from the experimenter. Bayesian techniques, on the other hand, require some prior information, and this will affect the conclusion.

Typically, when performing acts of inference, one begins with maximum entropy if very little is known except what's given in the data. Once more is known, one may turn to Bayesian analysis.

Bayesian analysis requires five things: a model, sample space, hypothesis space, prior probabilities, and sampling distributions.

There is much work to be done in developing techniques when even little is known about the raw data; this could lead to steps that can assist in studies where even maximum entropy may fail to give adequate results.