In an INET funded paper just published in PlosOne, Baccini, De Nicolao and Petrovich show the bibliometric equivalent of Goodhart’s famous law for banking regulation: When something becomes a target, it ceases to be a good measure.
Their striking results, based on the natural experiment offered by the Italian reform of 2010, confirm that scientific performance evaluation based on bibliometry corrupts researchers’ behavior immediately after implemented, even when multiple indicators are used (as recommended by the influential Leiden manifesto) and when the system is mixed (when it includes also a peer-review stage). The paper calls for a radical rethinking of the way research is evaluated and researchers are hired.
The authors developed a new inwardness indicator to estimate country-level citation clubs and self-citations which shows that Italian researchers reacted to the new bibliometric evaluation system by artificially boosting the number of times they cited themselves and each other.
Ironically, a system touted as a means to fight cronyism in academia ends up pushing that very same behavior. The Institute for New Economic Thinking has for many years supported the view that the quality of research work eludes simple summary in a handful of numbers. Already in 2006, Luigi Pasinetti warned that promoting evaluation by “point systems” would lead to faux accounting and gaming the system Now we know it’s true.
But what Baccini et al. show is only one measurable part of the problem, which hints at a host of other consequences. Those have to do, for instance, with research works that never see the light of day, because they do not pay off in terms of citations, or publication ranking.
The effects involve also the political environment, as we discussed at the 2018 G20 Global Solutions Summit in Berlin: Metrics often offer a blanket for politics to exclude uncomfortable scientific results and policy suggestions. Economics is particularly subject to that interference. There is a fundamental contradiction in how expertise is perceived and used socially. On the one hand, academics are seen as those to keep under control and accountable; on the other, carefully selected experts mysteriously become indisputable when it comes to showing scientific support for some policy without offering room for counter-arguments. On the contrary, INET believes that the autonomy, freedom and the plurality of views in the academia should be respected and the public opinion allowed to see the full range of views on policy issues.
Press Highlights
CORRIERE: I professori si citano da soliCosi si gonfia la ricerca
NATURE: Italy’s rise in research impact pinned on ‘citation doping’
SCIENCE: Clubby and ‘disturbing’ citation behavior by researchers in Italy has surged
NATURE: Elsevier investigates hundreds of peer reviewers for manipulating citations
PHYSICS TODAY: Researchers in Italy are increasingly citing researchers in Italy
