Not everything that counts can be counted

My first post for physicsfocus described a number of frustrating deficiencies in the peer review system, focusing in particular on how we can ensure, via post-publication peer review, that science does not lose its ability to self-correct. I continue to rant about discuss and dissect the issue of post-publication peer review in an article in this week’s Times Higher Education, “Spuriouser and Spuriouser”.

Here, however, I want to address some of the comments left under that first physicsfocus post by a Senior Editor at Nature Materials, Pep Pamies (Curious Scientist in the comments thread). I was really pleased that a journal editor contributed to the debate but, as you might be less than surprised to hear, I disagree fundamentally with Pep’s argument that impact factors are a useful metric. As I see it, they’re not even a necessary evil.

I’m certainly not alone in thinking this. In an eloquent cri de coeur posted at his blog, Reciprocal Space, last summer, Stephen Curry bluntly stated, “I am sick of impact factors. And so is science”. I won’t rehearse Stephen’s arguments – I strongly recommend that you visit his blog and read the post for yourself, along with the close to two-hundred comments that it attracted – but it’s clear from the Twitter and blog storm his post generated that he had tapped into a deep well of frustration among academics. (Peter Coles’ related post, The Impact X-Factor, is also very well worth a read.)

I agree with Stephen on almost everything in his post. I think that many scientists will chuckle knowingly at the description of the application of impact factors as “statistically illiterate” and I particularly liked the idea of starting a ‘smear campaign’ to discredit the entire concept. But he argues that the way forward is:

“…to find ways to attach to each piece of work the value that the scientific community places on it though use and citation. The rate of accrual of citations remains rather sluggish, even in today’s wired world, so attempts are being made to capture the internet buzz that greets each new publication; there are interesting innovations in this regard from the likes of PLOSMendeley and altmetrics.org.”

As is clear from the THE article, embedding Web 2.0/Web 3.0/Web n.0 feedback and debate in the peer review process is something I fully endorse and, indeed, I think that we should grasp the nettle and attempt to formalise the links between online commentary and the primary scientific literature as soon as possible. But are citations – be they through the primary literature or via an internet ‘buzz’ – really a proxy for scientific quality and the overall value of the work?

I think that we do science a great disservice if we argue that the value of a paper depends only on how often other scientists refer to it, or cite it in their work. Let me offer an example from my own field of research, condensed matter physics – aka nanoscience when I’m applying for funding – to highlight the problem.

BANGING A QUANTUM DRUM

Perhaps my favourite paper of the last decade or so is “Quantum Phase Extraction in Isospectral Electronic Nanostructures” by Hari Manoharan and his co-workers at Stanford. The less than punchy title doesn’t quite capture the elegance, beauty, and sheer brilliance of the work. Manoharan’s group exploited the answer to a question posed by the mathematician Mark Kac close to fifty years ago: Can one hear the shape of a drum? Or, if we ask the question in rather more concrete mathematical physics terms, “Does the spectrum of eigenfrequencies of a resonator uniquely determine its geometry?”

For a one dimensional system the equivalent question is not too difficult and can readily be answered by guitarists and A-level physics students: yes, one can ‘hear’ the shape, i.e. the length, of a vibrating string. But for a two dimensional system like a drum head, the answer is far from obvious. It took until 1992 before Kac’s question was finally answered by Carolyn Gordon, David Webb, and Scott Wolpert. They discovered that it was possible to have 2D isospectral domains, i.e. 2D shapes (or “drum heads”) with the same “sound”. So, no, it’s not possible to hear the shape of a drum.

What’s this got to do with nanoscience? Well, the first elegant aspect of the paper by the Stanford group is that they constructed two-dimensional isospectral domains out of carbon monoxide molecules on a copper surface (using the tip of a scanning tunnelling microscope). In other words, they built differently shaped nanoscopic ‘drum heads’, one molecule at a time. They then “listened” to the eigenspectra of these quantum drums  by measuring the resonances of the electrons confined within the molecular drum head and transposing the spectrum to audible frequencies.

SO FAR, SO IMPRESSIVE

But it gets better. A lot better.

The Stanford team then went on to exploit the isospectral characteristics of the differently shaped quantum drum heads to extract the quantum mechanical phase of the electronic wavefunction confined within. I could wax lyrical about this particular aspect of the work for quite some time – remember that the phase of a wavefunction is not an observable in quantum mechanics! – but I encourage you to read the paper itself. (It’s available via this link, but you, or your institution will need a subscription to Science.)

I’ll say it again – this is elegant, beautiful, and brilliant work. For me, at least, it has a visceral quality, just like a piece of great music, literature, or art; it’s inspiring and affecting.

…and it’s picked up a grand total of 29 citations since its publication in 2008.

In the same year, and along with colleagues in Nottingham and Loughborough, I co-authored a paper published in Physical Review Letters on pattern formation in nanoparticle assemblies. To date, that paper has accrued 47 citations. While I am very proud of the work, I am confident that my co-authors would agree with me when I say that it doesn’t begin to compare to the quality of the quantum drum research. Our paper lacks the elegance and scientific “wow” factor of the Stanford team’s publication; it lacks the intellectual excitement of coupling a fundamental problem (and solution) in pure mathematics with state-of-the-art nanoscience; and it lacks the sophistication of the combined experimental and theoretical methodology.

But yet our paper has accrued more citations.

You might argue that I have cherry-picked a particular example to make my case. I really wish that were so but I can point to many, many other exciting scientific papers in a variety of journals which have attracted a relative dearth of citations.

Einstein is credited, probably apocryphally, with the statement “Not everything that counts can be counted, and not everything that can be counted counts”. Just as multi-platinum album sales and Number 1 hits are not a reliable indicator of artistic value (note that One Direction has apparently now out sold The Beatles), citations and associated bibliometrics are not a robust measure of scientific quality.

Philip Moriarty

Philip Moriarty

Philip Moriarty is a Professor of Physics at the University of Nottingham. His research interests lie in what has occasionally been described as “extreme nanotech” in that he works alongside a talented bunch of nanoscientists to prod, poke, push, pick, and pull individual atoms and molecules in order to explore forces and interactions down to the single chemical bond limit. Moriarty also has a keen and long-standing interest in science communication and public engagement. He is a member of the Sixty Symbols team that was awarded the Institute of Physics Kelvin prize in 2016 for “innovative and effective promotion of the public understanding of physics”. While he doesn't share his infamous namesake's fascination with the binomial theorem, Moriarty enjoys exploring the maths-music-physics interface including, in particular, the deep and fundamental links that exist between quantum mechanics and heavy metal music (a theme discussed at length in his book, “When the Uncertainty Principle Goes to 11”). He blogs at https://muircheartblog.wordpress.com.
Philip Moriarty
FacebooktwitterFacebooktwitter

Comment via Facebook

Comment via Disqus

iop-blog

Comment via Google+