The journal impact factor. Does it tell us anything about the scientific quality of an individual publication? Is scientific quality the same as scientific impact? What alternatives do we have to assess someone’s research?

Another metric that I find useful is the Altmetric score. It focuses on trying to measure the impact of a paper outside of scientific publications. This includes for example the attention that the paper creates in social media networks, but also in news outlets, blogs, Wikipedia etc. The Altmetric score reflects both, the attention that it creates within the scientific field (e.g. through the number of Tweets in the scientific Twitter world), but also the attention outside, in the real world. Consequently, a paper that covers a topic that is of interest to a very broad audience has the potential of creating a very high Altmetric score. This is something that needs to be taken into account when comparing Altmetric scores. For example, the Altmetrics scores that I get for my papers basically reflect the attention within the world of microscopy people. Papers that my wife publishes on topics such as how your lifestyle affects the chances of developing Dementia when you get older achieve Altmetric scores that are orders of magnitudes higher than mine. This also shows that the Altmetric score allows to give an indication of how much impact a paper has for the society. As with any metric, one also has to be careful to use the numbers not blindly. Many basic research is very important as a basis for subsequent more applied research, but may sound quite boring to the wider public in the first instance.

The alternative question was: Should we not get away from numbers that we calculate to compare each other? Here, the ‘San Francisco Declaration on Research Assessment’ (DORA) is an important step in the right direction. The aim of DORA is to establish more suitable criteria to assess scientific quality and research output. One of its central points is to avoid ill proxies such as the JIF and focus more on the content and the “real” impact of individual papers. Impact is difficult to measure and not the same as attention. How much impact someone’s research creates requires looking at the long term effects: How is it picked up by other researchers in the field? How does it transpire to other fields? What are the benefits to society? Coming back to the comparison above of a microscopy paper and a lifestyle and Dementia paper: Dependent on in which field you are working and how close this field is to direct applications and consequences in society, an assessment of the “real” impact can in one case maybe be made based on one publication or, in another case, might take many years. Going away from numbers that we can calculate quick and easily also means more work for someone in the position of trying to evaluate the work of another researcher.

At the time of writing, 21,810 individuals and organizations world-wide have signed DORA. I haven’t done a proper assessment, but it feels like most universities in the UK and a lot in the US have signed the agreement. When I checked how it looks for Germany, I counted 3(!) universities. At least the DFG (German Research Council) and the Volkswagen-Foundation have singed. My hope for the scientific community in Germany is that this encourages more universities to also sign the agreement and implement changes in how they assess researchers. Changes will only happen if the institutions will start to actively contribute to the aim of a fairer research environment. In an article about improving the research culture of an institution (‘Setting the right tone’), Tanita Casci and Elizabeth Adams from the University of Glasgow make this point quite clear: “Success will not come from issuing policies […]. Even if university policies are read, they will be forgotten unless the principles are embedded in standard practice. And if we are not serious about our practices, then we are not credible about our intentions”.