Questionable research practices, peer review and an open access future?

Blimey—it’s been five weeks since my last post and I’m now a five weeks into my postgraduate studies. It’s gone quickly and I’ve been very busy.

As part of the doctoral training centre’s new/modern/[positive adjective] approach to a PhD we get (well, have) to take courses that ’round us out’ as modern researchers. A few weeks ago, we had a course on research ethics taught by Marianne Talbot. I did Philosophy A-level and especially enjoyed moral philosophy, so I was looking forward to it.

The course was attended not just by PE DTC students but also the CQD and TMS DTCs. Rather unsurprisingly (but disappointingly) there was a bit of a unfriendly vibe between the different DTCs. “We get MacBook Pros!” said one, “we don’t have to do experiments!” said another, to which we all replied “we get ¬£18,000 to spend and we like lab experiments!” The conversation never progressed any further…

Overall the course was excellent and very enjoyable. I loved how Marianne dealt efficiently and firmly with the few people who wanted to deny the existence of everything! One of the afternoon sessions was on open access publishing, a topic I already had an interest in. I’ve read about it before but have never been entirely convinced (I’m not sure why). Marianne gave a strong case for open access is good. She referenced this website as a good overview. If you don’t know what open access is, then it’s worth a quick read. There was unanimous support of the open access concept.

Marianne then introduced a distinction I had never heard of before: green and gold open access methods. In the green method, papers are deposited in a public online repository. Papers are not peer reviewed prior to being published and anyone can upload an article. The most famous example of this is probably arXiv. In the gold method, you submit a paper to journal, it’s peer reviewed, and if accepted it’s published in a journal that is either entirely open access or permits some open access articles. An example of the former type is PLoS ONE.

The question Marianne asked us to discuss was “Do you think it is acceptable for scientist to self-archive pre-prints in repositories with peer-review?” The answers from students were quite vague. But generally it seemed that peer review was held in extremely high, almost reverent, regard.

I found this odd considering we had just been discussing questionable research practices. One example of a questionable research practice that stuck out to me was:

leaving important information out of methodology section of a manuscript or refusing to give peers reasonable access to unique research materials or data that support published papers.

One would expect that if peer review functioned as well as my fellow students said then readers would rarely come across this practice in the literature. Yet in my field of research, I encounter it all the time! Authors brag that they’ve found the way to make the biggest, smallest, longest or generally ‘best’ nanoparticle but then fail to tell you crucial information such the number of moles of reagents, reaction times and temperatures that allow you to repeat the work. I spent an unbelievable amount of time last year trying to figure out the required conditions to synthesise heterostructured quantum dots. If peer review did it’s job, then things like this wouldn’t get through.

Other students were arguing that because anyone can publish a paper in a green OA repository that there is no quality control. I disagree. I think a lot of students are assuming that readers are idiots and need peer review. If you uncritically read a paper or think that because it’s in a journal it must be true then you’re at best naive or at worst incompetent. Decent researchers will spot questionable claims and results.

Is peer review even really that good a quality control method? Typically you only have two reviewers. Can you be sure they read the paper instead of give it to a PhD/postdoc?

Imagine that rather than submitting papers to traditional peer reviewed journals researchers published their work in open access green repositories. No real scientist is going to post rubbish because their reputation is on the line. Rather than having only two reviewers as with traditional journals, you could have tens or even hundreds of reviewers. They could post their comments—the peer review—publicly on the repository article page (I’m thinking more along the lines of threaded discussions rather than linear blog-style comments).

I think it would be awesome. The authors could respond to readers’ questions, for example, asking for clarification of an experimental technique or reagent used, or post new versions of the article correcting mistakes or providing further information.

At present, reviewers’ comments are made privately and anonymously. These comments would be useful to the scientific community. There’s no reason why it should stay private. Science is all about debate, questioning and (a moderate dose of) scepticism. At conferences and in department presentations, researchers handle criticism and questions. There’s no reason why journal articles should be any different.

I do wonder whether I’m being overly optimistic or if I’ve missed out something crucial. What do you think? I’d like to know…

4 thoughts on “Questionable research practices, peer review and an open access future?”

  1. Lot of interesting points raised here, ripe for discussion, but i’ll give you a couple of my thoughts on some things you mentioned… It’s terrible really, but I think the reason why people leave stuff out of their papers (as you said, often in the ‘experimental’ sections) is because they don’t want other people to ‘catch up’ with their research. It’s a shame, because it shows to me that the current way with science research is often that of rivalry rather than shared information for overall progression in the field. Although saying that, if I spent several months perfecting a reaction, I can’t deny that I would feel a strong urge not to put it all out to the world – there’s always someone out there better, and if they see your published research and take it further faster than you can that could be your PhD idea ruined. This is a pessimistic view, although perhaps one fostered by the current reseach culture. (Not to say that I am blameless for my potentially selfish attitude!)

    1. Apologies for the incredibly late reply!

      I think you’re right that it is definitely to do with staying ahead of competition. That leads on to the question of why scientists are so desperate to stay ahead. Probably many different factors, like publication metrics, lack of funding‚Ķ And ultimately the scientist’s career is at stake. Competition is good, but as I heard someone say at a debate last week, it’s reached counter-productive levels.

  2. part 2…

    saw this in the news today: http://www.bbc.co.uk/news/science-environment-15889634

    if you go on the professor’s website, you can see some papers he’s published. I had a look at a paper from 2007, which is from around the time of the claim that his papers weren’t peer reviewed. I’m no physicist, but even from a casual glance I know that I would be sceptical of the paper. It’s short on references, and half of the references are his own work. This may be an extreme example but perhaps it gives a glimpse into scientifc papers published without peer-review. Also just to say I am sort of playing devil’s advocate here – I generally agree with your post.

    1. I think papers like that will be really easy to spot though. If the repositories had thread comment/discussion pages for each paper, readers could flag stuff like this up and readers less knowledgable about the area would be able to see that others reckon it’s dodgy.

Comments are closed.