Metricide

Recently I have been giving a lot of thought to bibliometrics and the way they are being used. I'll come clean first, as an information professional I am unduly fascinated by the data on they way information is created, shared, used, and transformed. To me, bibliometrics are fascinating. However, there are some issues around the way bibliometrics are used and abused that I want to untangle.

So, what exactly is bibliometrics? Bibliometrics, a subset of informetrics, studies the quantitative data associated with the written word, and in particular, academic literature. Though it bears some relation to the algorythms used by search engines to assign page ranks to results. It is predicated largely on citation analysis and has produced measures such as;

Journal-Level Metrics

Thompson Reuters Journal Citation Reports is the famous example of this. JCR takes the number of times a journal title has been cited and divide it by the articles published by that same journal. For example, the journal Annual Review of Nutrition published 39 articles during 2012 and 2013. These same articles were cited 326 times during 2014. Hence 326/39 equals a 2014 impact factor of 8.35. All very simple really. The higher the number, the more the journal is being discussed.

Eigenfactor scores are another example of this. Jevin West created the Eigenfactor score (an 'amalgamation of eigenvector centrality and impact factor') in partnership with his PhD supervisor Carl Bergstrom in 2005. You can read the methods for calculating Eigenfactor scores for yourself here. Because they are difficult for the layperson to interpret, West and Bergstrom have also devised the Normalised Eigenfactor - where it is assumed the average score for a journal title is equal to one. As with JCR impact factors, the higher the number the greater the impact of the journal.

Author-Level Metrics

Also in 2005, Jorge Hirsch proposed a way of calculating "the cumulative impact and relevance of an individual’s scientific research output" which he named the h-index. Like journal impact factors, the h-index balances the number of research outputs against the number of times they have been each been cited. The h-index has become such an established measurement of academic impact, that it is now used by Google Scholar.

Google also started displaying the i10-index in 2011, which is the number of research outputs by an author that have ten or more citations. West and Berstrom later redeveloped the Eigenfactor for the calculation of author-level metrics in 2013.

Article Level Metrics

Another Thompson Reuters product, Essential Science Indicators uses citation thresholds to calculate the average number of citations a journal article receives. Because citation rates between subjects can differ, ESI produces annual citation thresholds for subjects roughly grouped into 22 research fields. For example, articles from the field of Space Science published in 2010 have received an average of 19 citations each - while articles in Social Sciences for the same year have received an average of just under seven citations each. So if you published an article on Space Science in 2010 that received ten citations, you would know your research isn't doing too well. Alternatively, if your 2010 Social Science article received ten citations, you'd be feeling pretty happy about it.

So what?

These might sound like safe, established, scientific measures of value, but they are easily manipulated. Sometimes unwittingly as in the case of Acta Crystallographica – Section A and sometimes wilfully though self-citation, and citation-stacking. For example in 2013 academics from Brazil were caught citing each others' papers hundreds of times in an effort boost the impact factor of those journals. They did so because fledgling journals typically receive a very low impact factor for their first few years. Why would they want to do so? Because the postgraduate programmes they were teaching are evaluated on criteria that includes the impact factor of the journals they publish in. Article and author level metrics are increasingly becoming the unofficial standard with which academics and the institutions they work for are measured. In some cases, the official standard. More alarming for information professionals such as myself, the higher the impact factor, the higher the subscription costs tend to be.

In 2014 Donald Gillies covered the apparent systematic bias in terms of research grants, but you could easily say the same thing about the metrics used to evaluate academic publishing. It's a grey area isn't it? Which gets to the heart my misgivings about the way bibliometrics are used: that is the application of a quantitative method to reach a qualitative conclusion.

I'm not the only person who feels this way. In 2013 at the Annual Meeting of The American Society for Cell Biology the San Francisco Declaration on Research Assessment was drafted, advocating for the elimination of journal based metrics. Researchers from America and the Netherlands collaborated to produce the Leiden Manifesto in early 2015. Recently in Australia there are plans to remove the criterion of publications for the allocation of government block funding. Jeffrey Beall even went so far as to call both the Eigenfactor and the h-index "dead metrics," with "little or no practical value to researchers or librarians." Who knows, perhaps he's right?