With the deadline looming for submissions to the 2014 Research Excellence Framework (REF), many in the research community are thinking about impact. Though assurances have been made that no sub-panel will make use of journal impact factors, rankings, lists or the perceived standing of publishers in assessing the quality of research outputs, many are sceptical.
And they’re frustrated too; particularly over the scientific community’s reliance on journal-based metrics such as the journal impact factor. Originally created as a tool for librarians when deciding which journals to subscribe to, it has now become a surrogate measure for the quality of research outputs. However, this measure can be misleading as the impact factor refers to the average number of citations per article in a journal – often these citations originate from a very small percentage of articles. The impact factor can also be readily manipulated by the type of articles present in a journal; inclusion of reviews and other highly-cited outputs can lead to a boost in impact factor.
Basing research evaluation on a metric which can be so readily skewed is clearly not best practice. Hence, the San Francisco Declaration on Research Assessment (DORA) seeks to improve the ways in which the outputs of scientific research are evaluated. The movement was initiated by the American Society for Cell Biology (ASCB) together with a group of editors and publishers of scholarly journals who met in December 2012 during the ASCB Annual Meeting in San Francisco. DORA is now a worldwide initiative covering all scholarly disciplines.
The Declaration sets out a series of recommendations for improving the way in which the quality of research outputs are evaluated. Rather than setting out a new and alternative strategy, the declaration instead seeks to formalise the growing vexation surrounding journal impact factors.
The recommendations are aimed at specific stakeholders including funding agencies, academic institutions, journals, organisations that supply metrics and individual researchers, and focus primarily on practices relating to research articles published in peer-reviewed journals.
A number of themes run through these recommendations:
- the need to eliminate the use of journal-based metrics, such as journal impact factors, in funding, appointment, and promotion considerations;
- the need to assess research on its own merits rather than on the basis of the journal in which the research is published
- the need to capitalise on the opportunities provided by online publication (such as relaxing unnecessary limits on the number of words, figures, and references in articles, and exploring new indicators of significance and impact).
Overall, the Declaration calls on all stakeholders to refrain from using journal-based metrics – such as journal impact factors – as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions. In place of this, a limited number of strategies are suggested to publishers, such as presenting the metric in the context of a variety of journal-based metrics (e.g., 5-year impact factor, EigenFactor, SCImago, h-index, editorial and publication times, etc.) that provide a richer view of journal performance.
Full details of the 18 recommendations outlined in the Declaration, including those aimed at researchers, can be found on the DORA website.
Time will tell whether impact factors are indeed taken into account in the 2014 REF. However, with the aid of movements such as DORA, perhaps future impact assessment will be very different.