Details matter

Blog Syn is a new chemistry blog where chemists post their attempts to reproduce reactions from the literature. Each post starts with the following disclaimer:

The following experiments do not constitute rigorous peer review, but rather illustrate typical yields obtained and observations gleaned by trained synthetic chemists attempting to reproduce literature procedures…

I disagree completely. What could be more rigorous than actually trying a reaction?

So far there are three posts. The first gave a lower yield than reported. The second was “moderately reproducible”. The paper omitted details essential to the reaction’s success. The third was “difficult to reproduce” and is well worth reading—there’s a great response from one of the authors, Prof. Phil Baran.

It’s unacceptable for anyone to publish a paper without all the information necessary to replicate the results. It wastes researchers’ time and money. I’ve written before about my difficulties trying to replicate results. It’s infuriating. How do papers like this slip through peer review?

I suspect some authors don’t really know why a reaction gives a particular product, especially in nanoparticle synthesis. They manage to pull something off a few times and publish their findings, but (unknowingly) neglect parameters crucial for other researchers to be able to reproduce it. It could be something seemingly trivial, like the method used to wash the glassware. The next researcher does it differently because it’s not mentioned in the paper and gets a different result.

The only way to deal with this is for reviewers to demand thorough experimental sections. (But to do so they must have a good understanding of typical experimental procedures. This is a problem if your reviewer hasn’t been in the lab for years.)

An alternative scenario could be that the researchers, in the early stages of the work, find that doing X doesn’t work. Later they find doing Y does work. Y gets published. X stays in the laboratory notebook.

X is a negative result. On it’s own, it’s not very useful. Loads of attempted reactions don’t work. But in the context of the positive result (i.e. the paper) the negative result is actually very valuable to anyone who wants to repeat the paper. Serious consideration should be given to including them in the supplementary information.

Experimental methods are grossly oversimplified. We like things to be elegant and simple, but chemistry is complicated. There’s no excuse not to include more information because everything is published online and space constraints aren’t a problem.

Blog Syn shows that subtleties in chemistry are important. We should all acknowledge that in our own papers and demand that others do the same.

3 thoughts on “Details matter”

  1. I agree with you overall and you make some excellent points. (Naturally, all of my opinions coincide with excellent points…) That said, I want to quibble with your original quibble: this is not “rigorous peer review”.

    Perhaps it’s more rigorous than actual peer review: they actually test the reactions, rather than taking the claim at face value, as reviewers invariably do. I feel that to earn the word “rigorous”, the reproduction would have to be more thorough and systematic than is feasible for BlogSyn to attempt. Checking multiple substrates, repeating multiple times to check typical yields, etc., and presumably carried out with someone with detailed knowledge of the kind of chemistry under investigation rather than the average synthetic chemist. For example, I’d be a terrible ‘peer reviewer’ for strictly anhydrous, anoxic chemistry – most of my work is aqueous.

    None of the above is a criticism of BlogSyn; what they’re doing is perhaps more useful to your average chemist. I just feel that their disclaimer is justified.

    Good post though, I’ll be subscribing.

  2. I think this is largely a matter of semantics. What is “rigorous” peer review? There isn’t a standard definition anywhere, so your definition will vary form mine. So it goes. Just that fact that when a editor asks for three referee reports and gets three different one says that we can’t agree even on that aspect alone, let alone anything more “rigorous”.

    We all agree that BlogSyn is an important effort, but I’m not sure about the whole idea of not contacting the authors at some point. I’d trying running the synthesis a few times as they did, and then contact the authors for further input as needed. This should all be part of the published report – it will still be clear that the authors did/didn’t provide enough information in the publication, but that the reaction can still be made to work.

  3. You both make a good point about what “rigorous” reviewing actually is. I’ve been thinking about this over the weekend and I can’t really think of anything more to say… I guess most reviewers simply have to assume that the reaction presented does actually work. Checking the reaction does is an improvement on the reviewing process in my mind.

    John – I agree with you about contacting authors. Having said that, posting it publicly does “incentivise” them to respond properly. I’ve only emailed an author once and the response was very disappointing, to the point where it’s only made me even more suspicious that something isn’t quite right. Obviously posting it online borders on blackmail, which isn’t exactly nice. Post-publication review is something scientists need to work on. In my opinion it doesn’t really happen.

    Andrew – Thanks, hope you enjoy reading.

Comments are closed.