Interpreting Science data- Don't be mislead.

Hello,

There is always a lot of things happening in Medical science.  I often spend long time (Whenever i can of course) reading research articles. Many a time i pick up the article from pages like science daily or from podcasts or sometimes even by chance. Though this blog is truly dedicated to write about Microbiology only, i felt this is worth making an exception.

Let me try telling you a story (A small part of it is fabricated to give you the right glimpse) to get you my point.

A study was initiated by a group of Industrial researchers on the use of a toothpaste containing an additive that is supposed to reduce bleeding of gums. As study population, certain poor people were selected. (Ah you know, they happily agree for a free toothpaste). So a 100 people were randomized and assessed for dental hygiene and asked to brush with test paste. The 2nd group of people was given a placebo.

The investigators came back after an year and found there is no significant difference in terms of oral hygiene in 2 populations. Years of research, millions of money is a loss for the industry. The industry which made it decided to look deeper into the matter. Least to say, their preliminary test results indicated things other way round.

So a second team of experts went around collecting data from the same population and came up with a paradoxical result- "Inconclusive evidence". Boy, same study population, same toothpaste and 2 ambiguous result? Before reading further i encourage you to come up with your own possibilities on what might be the reason.

Ok. So what happened precisely is, the people who were given the toothpaste were selling them in the loose to other people. So the study population didn't really use the material at al. When the second group came across this fact, they declared that the findings were totally inconclusive.

Another quick scenario. A scientist decided to look into a problem of particular disease in a village. He found that every person who drank milk was infected with this strange disease and concluded that the milk had something to do (Statistical co relation is a player) with this disease. However, much latter they found that its not the case. The fact was, the milkman was a carrier of that disease, though milk itself had nothing to do with it.

In both case there is an obvious poor data collection. But you wouldn't know it until a deeper study was done. If a proper study was done in the first place, it would have saved additional work up. So what am i trying to tell you? Simple, a lot of data will be generated when we do studies, but the meaning of the data may be nothing but bogus. We should be careful when we analyze a simple observational study and relying on the published material.

 

This cartoon is taken from "Scientific Misconduct Blog" (Click here for Source)

            Coming to my second argument. Suppose a study for variable Y event is  Z and you gather data on expression of Z in response to Y and say "yes, Z is expressed in response to Y" or "no". The first thing you need to understand is that the answer in science is often dependent on the question you ask (I was listening to a podcast in "Brain science" an stuck with this idea! Sorry but i don't remember the episode number). Its turns out that if you were to ask some other question, you would be possibly arrive at some other solution for the same problem or question.

           Last part of my argument. A lot of studies is done at a molecular level looking for variables in an event. Let me borrow an idea from Matt Ridley (From his famous book "Agile genes"). If there are N number of variables effecting a phenomenon, then the effect of individual variable is usually low from a single stand point. But when we suppress all other variables, (in terms of looking only at one, or finding controls that negates other variables effect statistically) the variable in question looks significantly important. Many a times this is what happens in molecular studies. We many a times over estimate the function of a variable. This leads to problems when we take the entire scenario into consideration.

         I would like to make a strong note here. In no way, am advocating that research is misleading. Instead am advocating the way we handle data in science. Many a times the interpretation of the data is a subjective matter. Only when the whole study makes a prediction of events, which is true on testing and multiple such tests prove the same, do we accept a study and its theory. And very honestly i must say, because of publishing pressure, i see many papers that just throws the data without making meaningful predictions and proving it with a set of tests.

       This blog is my response to many questions, by many of my students on how to interpret the data in a Journal. All i have done is put the known facts am aware of in a single basket and try to tell you, why we come up with subjective opinions at times when we read an article. Sometimes, we are also mislead because we haven't looked enough, or data is not presented in the right way.

I surely want to conclude with a lay statement that i have often heard "Believe science. Not scientists"

Further reading

1.  Validomics: How do we ensure biologically relevant data? (Link)

2.  Ethics of qualitative research: are there special issues for health services research? (Link)