Questionable research practices, peer review and an open access future?

Blimey—it’s been five weeks since my last post and I’m now a five weeks into my postgraduate studies. It’s gone quickly and I’ve been very busy.

As part of the doctoral training centre’s new/modern/[positive adjective] approach to a PhD we get (well, have) to take courses that ’round us out’ as modern researchers. A few weeks ago, we had a course on research ethics taught by Marianne Talbot. I did Philosophy A-level and especially enjoyed moral philosophy, so I was looking forward to it.

The course was attended not just by PE DTC students but also the CQD and TMS DTCs. Rather unsurprisingly (but disappointingly) there was a bit of a unfriendly vibe between the different DTCs. “We get MacBook Pros!” said one, “we don’t have to do experiments!” said another, to which we all replied “we get £18,000 to spend and we like lab experiments!” The conversation never progressed any further…

Overall the course was excellent and very enjoyable. I loved how Marianne dealt efficiently and firmly with the few people who wanted to deny the existence of everything! One of the afternoon sessions was on open access publishing, a topic I already had an interest in. I’ve read about it before but have never been entirely convinced (I’m not sure why). Marianne gave a strong case for open access is good. She referenced this website as a good overview. If you don’t know what open access is, then it’s worth a quick read. There was unanimous support of the open access concept.

Marianne then introduced a distinction I had never heard of before: green and gold open access methods. In the green method, papers are deposited in a public online repository. Papers are not peer reviewed prior to being published and anyone can upload an article. The most famous example of this is probably arXiv. In the gold method, you submit a paper to journal, it’s peer reviewed, and if accepted it’s published in a journal that is either entirely open access or permits some open access articles. An example of the former type is PLoS ONE.

The question Marianne asked us to discuss was “Do you think it is acceptable for scientist to self-archive pre-prints in repositories with peer-review?” The answers from students were quite vague. But generally it seemed that peer review was held in extremely high, almost reverent, regard.

I found this odd considering we had just been discussing questionable research practices. One example of a questionable research practice that stuck out to me was:

leaving important information out of methodology section of a manuscript or refusing to give peers reasonable access to unique research materials or data that support published papers.

One would expect that if peer review functioned as well as my fellow students said then readers would rarely come across this practice in the literature. Yet in my field of research, I encounter it all the time! Authors brag that they’ve found the way to make the biggest, smallest, longest or generally ‘best’ nanoparticle but then fail to tell you crucial information such the number of moles of reagents, reaction times and temperatures that allow you to repeat the work. I spent an unbelievable amount of time last year trying to figure out the required conditions to synthesise heterostructured quantum dots. If peer review did it’s job, then things like this wouldn’t get through.

Other students were arguing that because anyone can publish a paper in a green OA repository that there is no quality control. I disagree. I think a lot of students are assuming that readers are idiots and need peer review. If you uncritically read a paper or think that because it’s in a journal it must be true then you’re at best naive or at worst incompetent. Decent researchers will spot questionable claims and results.

Is peer review even really that good a quality control method? Typically you only have two reviewers. Can you be sure they read the paper instead of give it to a PhD/postdoc?

Imagine that rather than submitting papers to traditional peer reviewed journals researchers published their work in open access green repositories. No real scientist is going to post rubbish because their reputation is on the line. Rather than having only two reviewers as with traditional journals, you could have tens or even hundreds of reviewers. They could post their comments—the peer review—publicly on the repository article page (I’m thinking more along the lines of threaded discussions rather than linear blog-style comments).

I think it would be awesome. The authors could respond to readers’ questions, for example, asking for clarification of an experimental technique or reagent used, or post new versions of the article correcting mistakes or providing further information.

At present, reviewers’ comments are made privately and anonymously. These comments would be useful to the scientific community. There’s no reason why it should stay private. Science is all about debate, questioning and (a moderate dose of) scepticism. At conferences and in department presentations, researchers handle criticism and questions. There’s no reason why journal articles should be any different.

I do wonder whether I’m being overly optimistic or if I’ve missed out something crucial. What do you think? I’d like to know…

[^mywork]: I hope to blog about my work in less vague terms at some point but I’ll probably have to wait a while for various reasons.

*[DTC]: doctoral training centre

*[PE]: plastic electronics