Two types of breakthroughs

An editorial in this month's Nature Photonics entitled "Transcending limitations" asserts that there are two types of breakthroughs: technological and conceptual. Technological breakthroughs occur when some experiment manages to measure something better or more accurately than in previous works. Conceptual breakthroughs often lead to greater scientific understanding because they force us to look at some phenomenon in a new way.

Often, conceptual breakthroughs require strong patience and steady work to explain previously unexplainable results in an experiment.

I would guess that funding agencies and governments prefer technological breakthroughs because of their immediate economic payoff, whereas academic institutions prefer conceptual breakthroughs.

The debate between data-centric science and hypothesis testing in soil microbiology

I've recently explored the topic of data-centric science, i.e. the art of answering scientific questions using data mining instead of generating hypotheses and testing them. I was therefore very interested by an article in this week's issue of Nature entitled "Microbiology: The life beneath our feet." The article was written by two scientists who study the relation between the microbial content of soil and its encompassing environment. One of them, Janet K. Jansson, promotes the use of data-mining from "omic" studies while the other, James I. Prosser, argues more in favor of hypothesis-driven experiments. Omic studies is jargon for the practice of identifying microbial species through detection of DNA (genomics), RNA (transciptomics), proteins (proteomics), and metaboloites (metabolomics).

Dr. Jansson argues that data-mining reveals new species of microbes and provides a sufficient base from which to carry out further experiments and tests of hypotheses. She claims that the primary critique of omic studies (which is that it provides only descriptive data) is weak since we simply don't know enough about the different species of microbes to begin with. Data mining from omics fills those gaps in our knowledge. Finally, she provides several examples where omic data mining has led to new discoveries and understanding around the world.

Dr. Prosser, on the other hand, claims that better value is obtained from hypothesis-driven research since it provides new concepts and logical frameworks for understanding the microbe-environment relationship. One quote from him that I particularly liked was the following:

Hypotheses lack value, however, if they are based solely on observations, or if they are relevant only to the data used to construct them. They are worthwhile if they incorporate novel ideas and flashes of inspiration; they can propose (ideally universal) explanations and mechanisms; and they generate predictions that can be tested by experimentation. It is this process, and not the initial observations, that truly increases understanding. Hypothesis-driven research can thus provide counter-observational, non-intuitive predictions and conceptual frameworks, and can indicate which techniques are, and are not, needed to test them.

He furthermore argues that the information obtained from data mining can generate new hypotheses but cannot be used to test these hypotheses, an argument that I have never before considered but believe to be true.

One final sentiment worth noting is delivered by his statement "In practice, purely descriptive studies of microbial communities are rare." I believe that he is arguing that, while one would argue there is value in using both approaches, the payoff from hypothesis testing is much greater and should therefore receive more resources.

I am quite pleased to see someone arguing against the use of data mining for no other reason than to balance out the arguments, but since I believe that scientific man power is growing faster than the number of hypotheses that can be generated, I see no reason to abandon data-mining as tool in this regard.

Data-centric science - My initial thoughts

"The scientific method is built around testable hypotheses. These models, for the most part, are systems visualized in the minds of scientists. The models are then tested, and experiments confirm or falsify theoretical models of how the world works. This is the way science has worked for hundreds of years... But faced with massive data, this approach to science — hypothesize, model, test — is becoming obsolete."

This quote is from a 2008 article in Wired Magazine called "The End of Theory: The Data Deluge Makes the Scientific Method Obsolete" by Chris Anderson. In this article, Anderson addresses our ability to solve scientific problems by looking for correlations in data without the need to form models. This ability has been enabled by the huge amount of searchable data that the internet has generated over the past two decades, which has led us into the so-called Petabyte Age.

This approach, sometimes referred to as analytics, has been successfully employed to translate between written languages, sequence genomes, match advertising outlets to customers, and provide better healthcare to people. Now, Anderson argues, it may be applied to problems across the full range of sciences. This is a welcome evolution, partly because many fields now possess too many theories and lack the experiments to validate or deny their predictions. Take particle physics or molecular biology, for example. There are arguably more theories and models now about these systems  than ever before, and many of them can not be verified. A data-centric approach could solve this problem.

This is a very interesting idea and I've been thinking about it for a few weeks now. I think that, to make any sense of it, I need to address several issues and assumptions. Questions to consider include:

• What is a model? When is it useful and when is it not?
• Are only certain fields of science able to benefit from a data-centric approach?
• What is the human component to research? How would it change if this approach was implemented?
• What has already been done to solve scientific problems with data-driven solutions?
• What are the philosophical implications to changing our idea of science? The scientific method has existed in some form or another for almost 2000 years (I'm referring all the way back to Aristotle, even if his ideas contained flaws). A significant change to the scientific method, especially given its importance to modern society, could have major sociological consequences.

I'll consider these questions in future posts.

Google Scholar Citations is up

This morning I received my notice that Google Scholar Citations, Google's new online tool for tracking your own publications, citations, etc., was up and running.

I haven't played around with it much, but I'm impressed that it automatically found everything I've produced that's on the web with only one mistake—it mistakenly concluded that I had authored a religious text.

I'm not quite so certain that it will be useful. After all, the number of citations that my papers generate don't improve the quality of my work. However, it's still fun to easily track these statistics, even if it only satisfies my own curiosity.

How not to argue in science

I'd like to expand a little on yesterday's post. I'm beginning to better understand what constitutes proper debate of a scientific work. Whether the following logic is actually practiced by most researchers is questionable, but this is nevertheless an interesting and important point.

Data from a study provides information that does one of three fairly obvious things to a conclusion: it increases, decreases, or leaves unaffected the likelihood that the conclusion is correct. I place emphasis on the word likelihood because any given conclusion can not be demonstrated as being correct with 100% certainty, and I highly doubt that conclusions can be proven false with certainty.

I think that this—the likelihood that a conclusion is correct given all information—as well as the competency with which an experiment was performed are the two objects open to debate within science. The debate becomes unscientific when researchers and journal reviewers perform the following errors:

1. Assigning too much weight to prior information, thus making the likelihood that another work's results are correct less likely then it perhaps should be.
2. As a corollary to the first point, workers would be in error if they didn't properly balance the weighting of all prior information. For example, the media, in their coverage of climate change, has been chastised by some for giving equal attention to climate change skeptics as they do to proponents. This is because the proportion of scientists against climate change is significantly fewer than the proportion who see it as a true occurrence.
3. Assuming that a finding is false given prior information or prejudices. If one accepts that a finding can not be false but rather highly unlikely, then arguing to reject a journal article because it contradicts previous findings is itself fallacious. The wider scientific community should (with its more balanced opinions) be a better interpreter of the likelihood that the claims are real.

Of course, if these errors were corrected, they could very well lead to many more published works, which would in turn dilute the field. As a result, grants may be harder to obtain (since they are in part based on published works) and the dissemination of knowledge would become greatly impaired; there would simply be too much information to analyze.

Fighting through boredom

Graduate school can often be frustrating, disruptive, and downright stressful. I probably did not need to state this since all graduate students are aware of this fact. I have been especially bored with my graduate studies lately and have been struggling to identify both the cause and a solution. Through talking with friends and lots of time thinking (especially during my recent vacation to the UK), I've slowly been able to reason out the cause.

Quite simply, I've forgotten my large-scale fascination with science. As an undergrad, I would marvel at every piece of popular physics literature I read, from discoveries at particle accelerators to the development of new nanotechnologies. In graduate school, I've become so mired in one specific area of research that I forgot that very cool things are happening all over the scientific world, such as the "bump" in the data seen at Fermilab.

Though not the only reason for my recent lull, it is a major one. And it points to a solution: take time out of my day to peruse the myriad of popular scientific articles and rekindle my interests. Though I may not be working on these famous projects, I find that I am much happier in the lab after having contemplated these things. They place my work within a greater context, and though I tend to be an individualist, I think that I at least need to do this to find satisfaction with my own work.

When is too much data necessary?

Ben Goldacre wrote an interesting article last week about research findings that never receive the attention or scrutiny of the academic arena or the public. His point was a bit blurred by the article's strange introduction, but I believe that he was trying to say that our prejudices selectively filter results that we find interesting. This in turn leads to "walled gardens" of knowledge that can negatively impact not just ourselves but society.

However, without any type of filtering I suspect that the amount of information that needs to be processed and weighed is simply too much to handle. Goldacre alludes to this in the concluding paragraph.

The most interesting questions aren’t around individual nuggets of data, but rather how we can corral it to create an information architecture which serves up the whole picture.

What is a good information architecture for society to adopt? Clearly a single individual can not process everything, so the task must be performed by a complex body of professionals and academics.

Creativity in academia

I recently read this very interesting article that is a followup to the author's original book "Hackers," a look into the subculture of the computer geeks who laid the foundation for today's computer-based society. Two of the common qualities of these influential tech giants is their obsessive drive for quality and their playful creativity. Indeed, many modern companies, such as Google, go to great lengths to foster creativity in their employees by giving them freedom and resources to work on side projects and time to think about new products. The idea, I think, is to keep employees' minds fresh and slightly unfocused so that inspiration strikes more often to the company's benefit.

A similar and equally interesting article came out recently on Talking Philosophy's blog in which the author, Benjamin S. Nelson, discusses the creative process itself in relation to a man, John Kanzius, who invented a radio frequency generator to both attack cancer cells and split water molecules (awesome!). Philosophers, starting with Poincare, have broken the creative process into four successive steps: preparation, incubation, illumination, and verification. I will take these steps to be self-evident in their meaning, but I only wish to note that I believe that creative environments strive to improve the preparation and incubation steps so that illumination happens more often and with better results.

This being said, I wonder now why such environments are not fostered in academia. Graduate students are frequently overburdened with many menial tasks such as grading papers and acting as teaching assistants, attending class, writing portions of grant reports, attending frequent group meetings, and staying up-to-date on the relevant literature. Add to this exercise, chores, and hope for a meaningful social life and one can quickly see that this lifestyle does not support creative solutions to research problems. In no way are these other tasks without benefits, but if the resources of the mind are constantly employed for a menagerie of many simple duties, then what room is there to allow ideas to incubate in their minds?


I think academia could really benefit from adopting some of the creative strategies that many companies now use to better the quality of their products. What do you think?

Note: In college there was a video that was often shown in our engineering business classes from some evening tabloid (Dateline or something similar) which followed a company's process for developing a new and improved shopping cart. I can't remember the name of the show or the company, but it is highly relevant here. Does anyone know what I'm talking about?

 Update: Found part of the video: http://www.youtube.com/watch?v=M66ZU2PCIcM. The company's name is IDEO and focus on innovative designs. Their take on the creative process is very characteristic of the stance that some new companies are taking.

You spin me right round

Ben Goldacre always has interesting things to say about the science behind health care and the pharmaceutical industry. In one of his recent posts he writes about a research project that examined 72 trials with negative results, i.e. an investigated drug or treatment did not cause a desired effect. Out of all of these trials, he quotes that only 9 gave any figures in the trials' abstracts and that 28 gave no numerical results at all.

What was in the reports was "spin," or the authors' attempts to project the results in a positive light. In order to prevent this, he says, trials are supposed to be registered before they are performed so that their intended purpose can not be changed. Additionally, there are guidelines that dictate what must be included in a report. These rules, however, are more akin to suggestions since there is no enforcement of them.

Can such a system be implemented in the physical sciences? I don't think so. Often, we're actually learning about the topic as we proceed through the research. No amount of preparation can allow us to establish a hypothesis sufficient for inclusion in a detailed report before we undertake the experiment. Hypotheses, I feel, are best constructed concurrent with an experiment. And as for report guidelines? Well, anyone who has had to deal with reviewers' critiques of their papers will tell you that there is rarely any consensus about what makes a good report.

I suppose that one could argue that a grant proposal tries to satisfy this purpose, but I can't say that I'm experienced enough to comment one way or another on it.