Negative results and dodgy papers: keep quiet or publish?

Negative results are very rarely published in the literature. After all, the literature is bursting with new positive results and we don’t have enough time to read all of these, let alone papers describing what doesn’t work. Negative results are dull—who would want to read anything in the Journal of Negative Results?

Up until recently I haven’t had a problem with the status quo. I’m afraid the following discussion is a bit vague because I’m (still) not sure about how much detail I can go into my work, but please bear with me.

I came across a paper published this year which describes the effect of doing something quite specific in a synthesis on nanoparticle shape. Do the thing, get a particular nanoparticle shape (usually quite challenging to obtain); stop doing the thing, you get another shape (easy to obtain). I was quite excited because if it worked it would get around a major barrier to my desired nanoparticles.

I repeated the reaction exactly as the paper described, but it didn’t work.

I repeated the reaction in a flow reactor as it would make it easy to intensify the “thing”. According to the paper, this should definitely give the desired nanoparticles because the morphology selectivity/yield is directly proportional to the intensity of the “thing”. But it still didn’t work.

I’ve now given up on the reaction and moved on to something else. But that my results will not be published means that someone else could also waste a lot of time and money—on equipment, reagents, electron microscopy—repeating the experiment.

What can I do? I think I have three options:

Option 1: Do nothing.

I’ve already made it clear that I don’t like this option. I’m fairly sure the paper is wrong. It bugs me that it exists without some kind of mark against it.

Option 2: Email the authors.

I’m not too keen on this either. I suspect that my email would be ignored. Plus, I would rather any discussion happened in the open, which brings me on to…

Option 3: Blog about it (and possibly email the authors telling them that I blogged about it).

I feel uneasy about this. Could it be perceived as confrontational? Would I get a reputation as a troublemaker? I feel like it is the proper, scientific and open thing to do, but in reality it is absolutely not the done thing. I suspect most researchers would go for option one and do nothing. I could be right and the paper is wrong, but I’d be very happy to be proven wrong and get the reaction working.

What you think? Keep quiet, email or blog? Any other suggestions are welcome.

Noise annoys (or why you should keep it down in the office)

Back in January the New York Times published a piece by Susan Cain titled “The Rise of the New Groupthink” in anticipation of her new book Quiet: the power of introverts in a world that can’t stop talking. I remember reading it at the time and thinking “that’s me! I’m not alone!”. I haven’t read her book yet, but, from the reviews I’ve read and her NYT piece, I believe the gist of her argument is that extroverts have taken over the world, much to the detriment of introverts. In addition, extroversion and introversion have lost their meaning as descriptions of personality and become laden with the value judgements “good” and “bad” respectively.

“The new groupthink” is my worst nightmare. For as long as I can remember, from primary school to university, I have despised being needlessly coerced into group work when I would have much preferred to work alone. Research has shown that you are more creative with privacy and in solitude. Simply, I’d rather work autonomously, alone. I work in a group now—two post-docs, another PhD student and a couple of exchange students—but we work on independently on our own closely-linked projects, whilst sharing ideas, helping and asking each other for advice. When I need to make a major decision, or am unsure which option to take, I speak to my supervisor. I’m really happy with this arrangement; I value my freedom and independence.

Whilst I have a lot academic freedom, I feel trapped in my office. Another story published this weekend, also in the NYT, covered the growing disquiet and discontent with the ubiquitous open plan office.

Aside from saving money on rent, the rationale behind the open plan office was to increase communication, collaboration and innovation. In a university, especially one like mine in the middle of a major city, space is at premium so everyone gets crammed in an open office.

However research mentioned in the article shows that workers in open plan offices suffer because they have little privacy and lots of noise. Speech noise is especially bad because it is directly understood by the brain and disrupts one’s focus from the task at hand. Introverts are affected more than extroverts. Extroverts—the noisemakers—don’t care. They simply can’t comprehend introversion.

Before I started my PhD I was excited about getting my own desk. There is nowhere quiet to work on campus. Wherever you go, there will be someone making a racket, even in “silent areas”. The library is packed and full of people talking and the computer rooms in my department are often louder than the cafe next door. So I thought my desk would be a chance to get some quiet.

Unfortunately my desk is no better. It’s right by the office kitchen area, where, for at least three hours a day, people from the surrounding offices drop by for breaks and lunch and make a lot of noise in the process. Furthermore, I’m near the entrance walkway, so everyone can see what I’m doing and often can’t help but comment on it.

One person in the NYT story said that “headphones are the new wall”. I wish! I can only work to music for so many hours a day before my brain turns to mush (I can program but generally not read to music) and my in-ear passive headphones don’t drown out background noise without being uncomfortably loud. I’ve even tried listening to white, pink and brown noise, but it doesn’t work.

Up until recently, I was writing a review and so spent most of my day at my desk. I found it rather stressful and impossible to concentrate. Fortunately, I’m back in the lab a lot (drone of the fume hoods or 6music on the radio—equally good), but it’s still a pain trying to work at my desk.

I’ve no idea what to do about the situation. Asking people to be quiet will give me a reputation for being grumpy. Moving to another office would be rubbish, as I’d be away from my group and labs. Ultimately, I think I’m going to have to put up with it. Reading the aforementioned articles has made me realise I’m not alone in hating my open plan office and that introversion is perfectly normal. But it has also reinforced my view that I should be able to work in peace and others should be a bit more considerate.

Friday Night Experiments

Tonight I watched a BBC documentary about Nobel Laureate Andre Geim. Each episode of Beautiful Minds (am I alone in thinking the title is a little bit naff?) covers the story behind the success of a particular scientist. Geim is a really interesting character and I recommend watching it.

One part I found particularly inspiring. He attributes a lot of his success to “Friday Night Experiments”, during which he does some quick experiments to try out new, more adventurous ideas. It was during one of these that he discovered that you can use scotch tape to mechanically exfoliate graphene from graphite.

Obviously there were loads of unsuccessful Friday nights before the discovery of graphene. He went on to say that the most important thing to remember is to know when to cut your losses and try something else. By trying out lots of new ideas every week, seeing what doesn’t work and what is promising, he has made some great breakthroughs in a wide range of fields.

I can see how for a postgraduate it is easy to become obsessed with getting a particular experiment to work or become completely blinkered on a particular sub-sub-sub-area of a discipline. You are meant to work really hard on a particular area in a PhD after all. But rather than working solely on one approach to my research, Geim has inspired me to get in the lab and try out some of my slightly more adventurous ideas every now and then. Most probably won’t work, but one might.

On the value of journal editors (and why green open access won’t work)

Previously I argued that traditional journals should be abandoned and green open access repositories like arXiv are the way forward. More recently I praised the “DIY” open access journal The Journal of Machine Learning Research run by researchers, writing that chemists should do something similar.

But now I think I’m wrong because I’ve underestimated the value of journal editors.

On Stephen Curry’s blog a commenter said:

“The current system of peer-reviewed journals is altogether very flawed. … [A]t the end of the day, the journals make millions just formatting, laying them out and sending a few emails. This just cannot be right.”

6 months ago I would have probably agreed. Anonymous Publishing Employee replied (it’s worth reading in full) saying that they are wrong because they underestimate the work a journal really does. Editors have to decide whether a paper fits in with their journal and is worth sending for review, obviously requiring technical knowledge. If it is worth sending they have to decide who to send it to, requiring personal knowledge of the scientists. A lot of administrative time is spent chasing up reviewers, but once the reviews are in the editor has to make a decision or repeat the review process again. If accepted, subsequent copy editing and layout takes time (money) and there are other indirect costs too, e.g. IT and rent. The main expense, they believe, is salaries (not that surprising).

Before I’ve said that peer review would work in green OA repos, but now I think I was wrong. Editors have a lot of specialist knowledge that ensures the right people review papers. It’s also required to finally decide whether to accept or reject a paper. I now doubt that a comparable level of peer review would happen in a repository. There’s no incentive for scientists to review post-publication. With a journal, there’s a certain amount of flattery involved when a scientist is asked to review by an editor. In effect, the editors drive the peer review process forwards, whereas it might never get started in a repo.

Furthermore, if we only had green OA repositories there would be another loss that I’ve never considered before: the commentaries, reviews, editorials and research highlights that complement the original research articles.

Screenshot from the current issue web page for the latest issue of Nature Chemistry

These are written or commissioned by editors. Recently I’ve really enjoyed the extra content in Nature Chemistry. An interview with Chief Editor Stuart Cantrill goes into more depth about the work behind the scenes. Lab on a Chip is another journal that I like to keep track of—obviously much more specialised than Nature Chemistry—and it has similar articles.

A complete shift to green OA would result in the loss of this valuable content. Websites or blogs might spring up to take it’s place, but I doubt it would be of the same calibre. It would be a real shame to lose it because it’s a great way to broaden one’s knowledge and stumble across interesting work.

Overall I think I was wrong about green OA repositories. Journal editors (rather than the “journal” in itself) are a valuable asset to the peer review process and scientific endeavour as a whole. Still, more could be done to enhance the transparency of the peer review, but I think that open access publication simply won’t succeed post-publication peer review in green repositories.

First conference: any advice?

On Monday I’m going to my first conference! It’s titled Continuous Flow Technology in Industry (more detailed information here if you’re interested). I’m not presenting anything, just attending.

I came across it on a Royal Society of Chemistry mailing list and thought it’d be good as it’s quite closely related to my group’s work[^groupwork] (four of us are going). It’s relatively small in size (no parallel sessions) so I thought it’d be a reasonable choice of first conference. I hope to pick up some ideas that will help solve a few particular problems in my own work.

I’ve been wondering whether there’s anything I should do in preparation. I’ve consulted my trusty guide The Unwritten Rules of PhD Research but, aside from networking (which I’m fairly happy about), it doesn’t have that much to say on the topic unless you’re presenting.

I’ve looked up all the speakers to have a quick look at what they do, but the majority of them are from industry so they don’t have a web page summarising their work like most academics. I’ve got myself some Imperial-branded business cards.

I’m unsure whether to take my laptop. I’m leaning towards no, as I think it’s a bit rude to sit typing away (and it won’t do my hands any good either). I could always read a couple of papers if a particular speaker is really that bad.

Does anyone have any advice? Please let me know in the comments.

[^groupwork]: See this paper for my group’s most recent publication on our work on flow chemistry.

DIY open access for chemists

Never before have I had the urge to start writing a blog post on the tube on a Monday morning, but after reading a post about the do-it-yourself open access Journal of Machine Learning Research, I have now.

To summarise the post by Stuart Shieber (although I encourage you to read it for yourself): the Journal of Machine Learning Research formed after the entire editorial board of Machine Learning quit. Since October 2010, JMLR has peer-reviewed and published over 1000 open access articles, at no cost to authors. It’s successful and well-respected—it has the highest impact factor of all journals in its category on Web of Science.

How do they do it? Volunteers. The same volunteers that peer reviewed for Kluwer’s (now Springer’s) Machine Learning and helped the publisher to make huge profits.

There are some costs, of course. The domain name—about £10 a year for .org. Hosting is provided free by MIT, but you can get fairly decent hosting for about £10 per month. Their biggest cost was a tax accountant! So far it has cost them about $10 per article—a far cry from the thousands of dollars most publishers want to publish OA.

This makes me think what on earth are publishers doing (aside from profiteering) charging at least $1500 per OA article? The JMLR demonstrates that the whole argument of scholarly publishing necessarily being an expensive process is patently false. Publishers rely on academics for their entire publishing process. The Internet—and backing of a university like MIT (which would cost them far less than a typical journal subscription)—provides academics all they need to take control of their field.

Computer scientists have the advantage over chemists of being highly proficient computer users and hence find it a lot easier to sort out typesetting with LaTeX and the installation of one of many available open source journal publishing systems. But with a bit of assistance, most chemists could submit articles in LaTeX, which is really quite straightforward. Furthermore, you only need a couple of people to get the web site going and maintain it.

The whole DIY ethos of JMLR is brilliant. Academics put so much work into their research, give it all up to publishers and then pay to read their community’s work. It’s great to see scientists publish their work themselves.

I would love chemists to start a DIY OA journal, though I think chemists, out of physicists who have arXiv and biologists and medics who have PLoS, are a conservative bunch. I’m not sure why. Shieber wrote that computer scientists are used to volunteering; I don’t think chemists are. Nonetheless, I think it could happen, especially with the backing of a department or university (I think libraries are in a good position to help here). I think this might be the way scholarly publishing moves in the future. I’d be more than happy to help get a DIY OA chemistry journal going.

*[OA]: open access *[JMLR]: Journal of Machine Learning Research *[DIY]: do-it-yourself

The death of my paper lab book?

Nature recently had a feature on the “paperless” lab which mostly focused on electronic laboratory notebooks (ELNs). As a computer nerd, I’ve been thinking about using one for a while.

ELNs have lots of advantages over paper notebooks. They’re searchable, easily backed up and can automatically incorporate data from instruments—no more cutting and pasting. Businesses like them as it’s easier to find out what an ex-employee did in an ELN than in loads of paper notebooks.

I’ve always used the my department’s standard synthetic chemistry lab book which has a risk assessment and reaction scheme on every left page and lines on every right. It works quite well. I number every reaction TWP001, TWP002 etc and samples are labelled TWP001-A, TWP001-B, etc. Spectra follow a similar convention, e.g. TWP001-A_em_spec.txt or TWP001-A_abs_spec.txt, and all data and code used for data analysis is kept in a folder called TWP001_brief_description.[^git]

But there are a few things that I really hate about paper lab books. Going back through my notes when writing up work is a real chore, especially with seemingly never ending notes along the lines of “same as TWP050 except…”. Reaction TWP050 says: “same as TWP049 except…”. With an ELN you can just copy and paste.

The inherent linearity of a paper lab book is a pain. Entries are in chronological order and reactions are performed sequentially, one at a time, but I usually work on two or three reactions at a time. Leaving blank pages looks sloppy, but cramming notes into small gaps is messy.

The biggest problem is that paper notebooks have become incomplete records of research in the modern laboratory. A lab book should be a complete record of your thoughts, observations, measurements and results. However with modern lab instrumentation it’s impractical or impossible to include all the data by printing, cutting and sticking it in. For example, a search on my computer (not a look in my lab book) reveals 510 UV-vis absorption, fluorescence and excitation spectra recorded since August 2010. There’s no way I could print that out (and even if I did, the data is useless in that format). Furthermore, a paper lab book can’t capture any of the data analysis on the computer. My MATLAB (and now Python) code is riddled with comments. With paper lab books, this information is highly fragmented.

Considering these problems I’ve been looking at electronic alternatives for some time, but what I’ve disliked about them boils down to two things: inflexibility and how they handle data. They seem to try to fit everything into a particular template or form. With a paper lab book, I can write and draw whatever I want, which is important to me as I’m not a “normal” synthetic chemist—I with flow reactors and I’m more interested in my residence time than yield.

I want to be able to access my plain text data files as plain text files and not have them converted into horrible proprietary binary formats subject to the whims of the ELN vendor. Think of the hassle caused when Microsoft switched from .doc to .docx—I don’t want this happening with my data. Plain text files from 30 years ago can still be read today and will be readable for longer than I’ll be alive. It also worries me that a web based ELN could disappear and leave me with a load of horribly formatted files to wade through.

Researching online I found advocates of open notebook science—the (left field) practice of making your entire lab book and data available online as it is recorded—using blogs and wikis as ELNs. Cameron Neylon’s blog-like open lab book used the University of Southampton’s free LabTrove software. Lab book entries are like blog posts, with attachements for data, and you can organise posts using tags, e.g. “NMR” or categories, perhaps to organise posts related to a single reaction. Jean-Claude Bradley’s group notebook, called the UsefulChem Project used a wiki. I really like Bradley’s wiki and there are lots of nice examples if you click about on the list of reactions. His group upload and link to spectra and photographs—a complete research record.

I did a bit more research into using a wiki for an ELN and they seem to be the perfect match. They’re flexible in terms of organising data however I want and pages are versioned so you can see what was written when. There are loads of different wiki applications available, so I narrowed the possibilities down with the following criteria:

  • active development
  • proven large scale deployment for stability and reliability
  • open source and free
  • page access control
  • supports attachments
  • self-hosted because I don’t trust anyone
  • written in a nice programming language
  • stores data nicely, i.e. not binary formats

This boiled down to MediaWiki (runs Wikipedia), FosWiki (used for loads of corporate intranets) and MoinMoin (large scale deployments are the Apache Software Foundation, Python and Ubuntu wikis).

MediaWiki doesn’t handle attachments very well for ELNs since attachments are available globally, i.e. across the whole wiki at the top level rather being linked to individual pages. The latter makes more sense to me as spectra or photos (the attachment) are related to the experiment (the page) rather than the whole notebook (the wiki). MediaWiki is designed for open content, so it doesn’t do access control without dodgy extensions. It’s also written in PHP, which I have no intension of learning. So that’s MediaWiki struck off.

FosWiki is aimed at corporations, which I think you can tell from it’s look and feature list. It’s also written in Perl, which I really don’t want to learn. So that’s FosWiki gone.

Last is Moinmoin. Unlike MediaWiki, attachments are linked to pages. MoinMoin is written in Python, a really nice language I’ve started to use instead of MATLAB, so there’s the possibility of writing my own extensions. It’s currently at version 1.9.4, so it should be very stable, and version 2.0 is under active development. It’s very clean and tidy.

I spoke to my supervisor about an ELN and he was extremely keen so I’ve decided to give MoinMoin a go. I’ve installed it on a Linode virtual server running Ubuntu linux.[^VPS] It took a about 6 hours to install the whole server from scratch—not bad having never administered a server before! Initially I was a little worried about security, with data being on a internet server, but I’ve locked down the server pretty tight and am going to make off site backups to my office machine. If anyone is interested, I’ll write up how to set it up.

It would be cool to make MoinMoin chemically-savvy—perhaps by pulling in data from ChemSpider or Wolfram Alpha, or COSHH info from Sigma-Alrich? I think this could be done with a little Python scripting. I’ll open source anything good for others to use. I’m also planning on setting up an old scanner in the lab to upload paper drawings.

This could all prove to be an embarrassing experiment or even a complete nightmare and ending with me dusting off my most recent lab book and finding a pen. On the other hand, it could be great. We’ll have to wait and see!

[^git]: I use Git to keep track of all changes to the files in each experiment folder. I don’t know anyone in my department who has heard of it, which is a shame as it’s a great tool. I’ll blog about it sometime.

[^VPS]: I could have installed it on a dedicated machine in the office, but we’re a bit short on machines and didn’t want to have to deal with hardware.

*[COSHH]: control of substances hazardous to health *[ELN]: electronic lab notebook

Burnout

Long time, no blog. The run up to Christmas was rather hectic because of my research proposal and literature review. This in itself wasn’t a problem—I enjoy reading and writing—but it was that my RSI stopped me from using my computer.

Throughout most of my undergraduate degree I remember having mild discomfort in my arms from typing, mostly because I had a job on Saturdays which meant I had spend the whole of Sunday writing lab reports. It was fairly bad while I was writing up my undergraduate project and then over the summer with temporary work that involved a lot of typing. Unfortunately my literature review caused it to evolve into a chronic and excruciatingly painful condition.

Thankfully I’m now on the mend. I spoke to Occupational Health and I’ve now got a new chair that works properly. Through a GP I’ve got physiotherapy every week, which seems crazy as you normally associate it with sports injuries or something you have after a major operation—not typing! I’ve also got a Kinesis Advantage keyboard. Ugly and expensive, but so comfortable to type on.

Reading that I was at risk of permanent disability made me realise I need to start taking care of myself.[^RSIbook] In the past, I worked and worked and worked. Lab reports had to be finished, lectures had to be revised and money had to be earned. I ignored my body to the point where I ended up in A&E in 2010 with suspected appendicitis (turned out it was probably a stomach ulcer).

With hindsight it was unsustainable. I’m under much less pressure now, perhaps because I’m driven by a desire to work on something I really enjoy rather than a pressure to manage a huge workload set by the department. But I can see how it could happen again.

To prevent it, I’ve issued myself with a rather plain prescription of exercise, less booze and a healthier diet. It seems obvious, but it’s easy to forget that work isn’t the be all end all and neglect your well-being. As my supervisor pointed out, you’re no good to anyone if you’re so knackered you can’t work.

*[RSI]: repetitive strain injury

[^RSIbook]: From Repetitive Strain Injury: A Computer User’s Guide by Emil Pascarelli and Deborah Quilter. I highly recommend it. Also check out Matt Might’s article on handling repetitive strain injury.

The start of a real education? Differences between undergraduate and postgraduate study

A friend who is considering a PhD at Imperial recently asked me what I thought were the key differences between being an undergraduate and a PhD student. We had a good chat over a cup of tea about it and I thought I’d share my thoughts here as I’m sure other people considering a PhD are wondering the same thing. I’m only seven weeks in but this is what I reckon so far.

You no longer learn for the sake of doing well in exams

I feel that I’m now under much less pressure to assimilate information. As an undergraduate, my sole aim was to cram everything I needed to know into my head for the exam. Post-exam, I forgot nearly everything.[^1]

I feel that I rarely ever appreciated the subtleties of a reaction mechanism or generally knew why something was the case (unless I was especially enthusiastic about the topic). Since it probably wouldn’t get me any extra marks, there wasn’t any point spending the time learning details.

Furthermore, as an undergraduate I never had the opportunity to explore anything in depth because the sheer volume of information I needed to know was already overwhelming. Now I can read the literature and listen to lectures or presentations without thinking “what do I need to know for the exam?”. It’s quite liberating. Reading literature is a now a pleasure.

Work is now endless

As an undergraduate everything you need to do is a discrete, self-contained project with a specific start and end. You get given most of the information you need and usually you can find answers in the literature. Once you’ve done everything you need to do, you forget about it and move on to the next lab report or problem sheet…

As a postgraduate this isn’t the case. You’re meant to be finding out new things that no one else knows or has done before. There isn’t a definitive start and end; you don’t know what you’ll find or how long it’ll take. ‘Answers’ don’t exist yet.

I love that I’m doing things that no one else has tried before, but it’s also slightly scary and a little overwhelming. You have to decide what is the best problem to tackle. I don’t really know how long things should take, so sometimes I worry I’m not making progress fast enough, but I think I’ll get used to this eventually.

No one will tell you how to do your PhD

My supervisor gave me a couple of papers and basically said “make these nanoparticles”. If you don’t know what to do, you have to sort it out as no one else knows. If someone else does know, then your work might not be that original.

I think this is the biggest change from being an undergraduate and I know a couple of people who seem to struggle with this. They aren’t used to being so independent and I think they really want their supervisors to give them specific instructions rather than guidance and a general prod in a particular direction.

Miscellaneous perks

You get a desk so you no longer have to work in undergraduate work areas and access to a kettle, microwave and fridge means you can avoid overpriced, depressing campus food. You get access to the senior common room (insanely cheap cooked breakfast) and PG bar where you can drink too much and embarrass yourself in front of your supervisor and other staff. Not that I’m speaking from experience.

Managing your research budget means you can buy things like a new Mac Mini and a enormous display! I’ve also been helping in a undergraduate quantum mechanics workshop and will hopefully start demonstrating in physical chemistry labs soon—extra money and valuable teaching experience. Soon I’ll get to go abroad to conferences and I’ve looked after a visitor from my previous supervisor’s group at ETH Zürich already.

Do I recommend it?

Definitely. I’m under no illusion that my PhD is going to be easy. I’ve been working hard the last few weeks and I think it’ll get tougher over the next few years. Undoubtedly I could earn more in a ‘proper’ job, but I really enjoy it so it’s all worthwhile

If anyone has any questions, feel free to ask in the comments.

Update (27 November 2011): Inspired by David Smith’s tweet I changed the title from “Undergraduate/postgraduate differences” to “The start of a real education? Differences between undergraduate and postgraduate study.”

[^1]: I don’t recommend this to current undergraduates: it will come back to haunt you in final year vivas.