Blogger's Desk- What’s wrong with our science

Greetings

      Lately, a few days ago I happened to stumble on an important point regarding “publication of research work”. Several blogs, podcasts, high level meetings, editorials etc.. are now pouring in opinions on how science is not working, the way it is supposed to. I have written on topics of very broad interest in this blog. Things that I have expressed below, is my personal opinion based on what I have found from polling (Link), talking in person to people who do science world over. I don’t wish to place offence on any Institution / Body. Moreover, the number of poll entries has been low, so my opinion may represent a bias of low count. As of what I understand, there are 3 key problems- funding, science, and publication.

    The first major issue is funding. Let us admit it. Science is an expensive process, and returns are not expected always. Conclusively, it is not expected of Industrial R&D giants to do all the research. That is why proper funding is an important point. As said in a podcast by Vincent in his now famous TWiV program (I don’t remember the exact episode), “To calculate the outcome of research is not possible always and everything cannot be translational. Sometimes we simply have to pursue the basic research”. I don’t see how I can disagree. I put a questionnaire for people doing science. Though very little people have actually polled their opinions, it is clear to me that a good number of researchers feel that they could have done better if there had been more funds. In context of Indian scenario, it is clear as to funding system needs drastic improvement. The funds are less adequate and reported time for the funds to be available to researcher is almost more than 6 months. Anecdotally, PhD and Post Doc candidates are underpaid world over. And there is too much competition for that too.

Fig 1: Results from Poll.

     “Don’t complain, if you have no idea how to possibly solve it”. Globally there is a crisis in funding. Almost every country has cut down on the amount of spending on science. As I said, Translational science is not always possible. However, without the basic science discoveries, improvements in research techniques, there would be nothing to translate. I had seen multiple blogposts, by many regular bloggers, which reads “Spending on science research is less than 10% of what is spent on warfare”. I don’t know of how true it is, and citations would be better.

      In some scenarios, funding maybe available, perhaps not utilized in the right direction. What I intend to say is, there are standard funding agencies such as the CSIR, NIH, DBT etc. But there are also certain research foundations such as the Bill gates foundation which focus on a specific area of interest. I have often talked to people who grunt when money is not available from the government budget. They have never even tried to put a proposal to more private research foundations. And now there is an emerging concept of public funding (Link). Public willing to donate, no matter how small the sum is, collected and used for a specific research.

Quoting from a Nature India article published a few months ago (Link), says it all

"A limiting step in the government funding agencies is the lengthy, not so transparent process, lots of paperwork and too much time taken between submission of a research proposal to approval of grant. There's also a lot of peeve with the processes involved. For instance, in scientifically advanced countries, if a researcher orders a chemical or antibody, he gets it at his doorstep within 24-48 hours. The process could take months in most Indian labs."

Fig 2: Schematic representation of knowledge.

     The second question is doing the science itself. People into research, very much polled that they do science to earn the degree/ promotion. The requirements of university guidelines, to have a research in paper (irrespective of what it means), has given way to low quality science. It is been said that from a million papers produced in life science each year, at least a significant proportion is junk. Then a question is to be asked. How does research qualify as a criterion for degree or promotion? This question is exempted from the cases of advanced research degrees (PhD, Post doc, International Fellowships etc.). But what about in cases of say M. Sc degree. The answer is, this. The world literature on life science doubles almost every 4-5 years. If you have not kept yourself updated by research, in a span of 10 years you had be teaching or practicing the science which no longer is pragmatic. With low quality science this goal is not achieved.

     Another point of note, “People believe too much in published material than their own experience”. People who wish to research further on published material, don’t verify the primary paper. Let us consider for argument sake, a hypothetical publication in journal with the impact factor of more than 30. X group discovers that Y compound can inhibit an important bacterial pathway. If I need to use that chemical as a possible antibiotic, I must verify the primary observations, especially if it is a very recent paper. Many people, often blanket the research as valid if it has a high impact factor and not so much if it is on a lower impact factor journal. And a person will try the same experiment till he gets the same result, but wouldn’t argue with his PI that possibly paper has flaws, especially if it has been published in a “big buck” journal. One important way of looking, is to communicate the corresponding author and learn of any fine adjustments that is not in paper. It helps a lot, than simply replicating without results. And there is always a chance that the paper is not right.

     A point that is worth noting here is a recent nature comment on p-value. The notion that almost everything depends on p-value is drilled so much into the student, is wrong. Today, researchers manipulate data, by changing values in a data set so that somehow it fits the golden p value (<0.5). P-value is just a mathematical way of looking and saying, does this data need a second look.

     The last but important point is publication. I just happened to have read an article in mBio, on how science and scientists have been pulled into publishing in “High Tier” journals, and how it is effecting the quality of science that is being done today. Though this issue has been raised by multiple people over the past couple of years, including me, the gravity towards having a high impact journal publish has not changed. Several proposals have been made in the article, on how this could be handled.

     This leads me to a question. What is the concept of a journal? Journal is a magazine that details the latest research findings of a particular subject. And why do they publish? It is a business. That is exactly why different strategies of publishing (Open access or pay wall access), are called as “business models”. Of course there are claims of nonprofit based journals. A journal is however, expected to have a scientific standard in contrast to other routinely available magazines. So how do you measure the scientific standard of a journal? There is a common accepted idea that when the science is important, it gets cited. So Impact factor (which is a calculation based on citations), represents quality. Right?

     NO. Impact factor is a measure of journals average. In other words, when I say, Journal X has impact factor of 2.7, it means on an average an individual paper is cited 2.7 times. “On an average” is a tricky business. Consider this. If a journal publishes say 3 papers which is cited 30 times each and you have another 30 papers never cited, you still have the journal impact factor of 2.72. With no citation for 30 papers!!! A more apt way of looking is how many times the paper itself has been cited than the journal itself. And remember there is no negative marking for retraction. The calculation shown is not the actual way of doing it, since previously published papers will add to the calculation. But you get the point. For the accurate methodology of calculating Impact factor, refer here.

       So if a journal will chew out only those papers that will potentially make the headlines, you will still have a high impact factor, even if a good number of papers never performs. And there is an additional underline. A paper which makes news will generally be discussed more in literature, thereby harvesting more impact.

      IF you had not known, the original DNA structure paper by Watson and crick, was never peer reviewed. Luckily, it was flawless (Listen to nature pastcasts series; Link). However, i should also make a note that at that time, Nature didn’t have a system of formal peer review in the 1950s, and papers were reviewed only when it seemed necessary. So how did it potentially end up there? Simple. Chances of making a news and being cited. In fact that paper was one of the hottest cited papers (More than 700 citations if I remember it correct). I say this because, I have often seen reluctance in high tier journals to publish stuff which is not well reviewed, but are news making stuff. Offhand I recall, two papers- Paper on stimulus-triggered acquisition of pluripotency (Link), arsenic DNA (Link). What happened to the highest quality peer review?

     Now you can see the loop. An idea that high impact factor journals is always right, tends to be cited easily. This leads researcher to believe that publishing in high impact journal is recognition of their work. If the article is retracted, the journal is not effected for its impact. When this became the case, everyone’s work was valued on the basis of journal and not their work itself. Let me quote from mBio article by Arturo Casadevall and Ferric C. Fang (Open access).

“In this context, the impact factor of the journal where work is published is often used a surrogate marker of excellence, despite the fact that citation frequency for the journal does not predict the citation frequency for individual papers”.

Fig 3: Poll results on choice of publication

       The above shown figure is the response to a question, on where would you want to publish your work. Most of them that have taken the poll said, they would publish in a lower impact factor, but specialty open access journal. Publishing in a an open access journal allows more readers for your work. This further helps in recognition of work. Moreover, specialty journal is read by people working in a similar group, which allows the chance for wide acceptance compared to a journal that covers a very broad area.

   Randy Schekman, a nobel laureate, in his famous boycott of top journals (Link) makes a point that probably there is a distortion of science. I don't know to what extent it is true, cause I have never published with top journals yet. But what amuses me his boycott doesn't make any difference for him. He has reaped the rewards, and so it is not going to effect his career. But for people at stake, they still want the recognition, it effects them directly.
   

     Probably everyone remembers the science sting operation. The study showed the concept of “predatory publishing” Link. Magazines claiming to be journals. I say so, as I have defined “Journal is a scientific standard magazine”. Remove the scientific standard and it becomes a magazine. The central problem is there is no good peer review. But that leads me to another question. How standard is a peer review process?

    This is a very slippery question. Peer review is not a perfect way of doing science, but is the best we have. Peer review is a procedure where peers with similar scientific background will check for validity of the work in question. This identifies issue in plagiarism, inconsistencies and inferences drawn from the data. However, there are no measures to assess the peer review itself (unless someone does the sting operation). I always find that when a paper is retracted, say because there is inconsistency in data the authors are convicted. My contention is that the reviewers should also equally hold the responsibility. I always had a thought in my mind that by publishing the reviewer’s response, along with paper, reviewers will be more accountable. There will be a definite quality if the names of reviewers are also published (with their comments), but then the question of anonymity of reviewer and its advantages will come into question. There is a website for learning what papers have been thrashed. Link

     By no means, I say that science has gone bad. But I do think that we need improvements. Science has suddenly become a business in itself leading to some dirty competition, for funding and publishing. This is a self-fulfilling loop. So I have suggested improve funding (everyone’s suggestion), don’t rely on journals impact factor, journals be subject to quality control by revealing the reviewers comments, and most important discuss science freely. Science was ans is, a self correcting process.

     I must owe thanks to all the people who gave their opinions freely, that allowed me to have better clarity of opinion. Maybe I will have more to say, next time. And sorry for the length of this post. But I needed to put it all.

Bohannon J (2013). Who's afraid of peer review? Science (New York, N.Y.), 342 (6154), 60-5 PMID: 24092725

Nuzzo, Regina. (2014-02-12) Scientific method: Statistical errors. , 506(7487), 150-152. DOI: 10.1038/506150a

Casadevall A, & Fang FC (2014). Causes for the persistence of impact factor mania. mBio, 5 (2) PMID: 24643863

Siegel, V. (2008). The promise of peer review Disease Models and Mechanisms, 1 (2-3), 73-77. PMCID: PMC2562197