Faculty of Physics, Hanoi University of Science

English (United Kingdom)

Advancing Evaluation: Moving Forward with DORA

E-mail Print PDF
User Rating: / 0
PoorBest 
There are no translations available.

 

 

Advancing Evaluation: Moving Forward with DORA

“The declaration itself remains unchanged, but our aim is to spread the word much more effectively—about DORA and, especially, about the good practices it has already helped to establish in many institutions.” – Stephen Curry, Imperial College, London

Five years ago at the American Society of Cell Biology Annual Meeting in San Francisco, leading cell biologists, editors and publishers dissatisfied with the near exclusive reliance on journal impact factor as the primary means of measuring success in academia began creation of what would several months later become known as the San Francisco Declaration on Research Assessment, or DORA. The declaration calls attention to the inappropriate and flawed use of journal impact factors and the community need for assessment tools to measure research outcomes other than peer-reviewed publications.

The Current State

DORA states there is a “pressing need to improve the ways in which the output of scientific research is evaluated by funding agencies, academic institutions, and other parties.” The erroneous and inflated value of publishing in high-impact journals is seriously affecting the way that scientists judge each other, as well as adversely impacting the reproducibility of research. To make more fair and broaden the way scientists are evaluated, within DORA specific recommendations for publishers, funders, institutions, metrics organizations and, perhaps most importantly, researchers themselves, were built around the following tenets:

  • eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations
  • assess research on its own merits rather than on the basis of the journal in which the research is published
  • capitalize on the opportunities provided by online publication (relaxing page, figure and reference limits, and exploring new indicators of significance and impact)

Anniversaries are often a time for retrospection, and as this year’s ASCB/EMBO meeting marks the fifth anniversary of the ASCB conference where DORA was born, those involved are taking the opportunity not only to look back at what has been achieved, but also to look forward at what more can be done. “DORA has been very useful in stimulating discussion and action on what truly robust processes of research and researcher evaluation should look like. It is focused on addressing the deleterious effects that the journal impact factor have had, particularly on research careers, but also on the pace and integrity of the scientific record,” says Stephen Curry of Imperial College London, one of the original signatories. Now, he says, is time for the initiative to “gain new ground, not just in Europe and North America, but all around the world.”

Grassroots Transitions

Those working now on revitalizing DORA see this upcoming anniversary year as an opportune window for an energetic transition from consensus-building to action. Says Bernd Pulverer, also an original signatory, “We’re seeing three stages, if you will, of DORA. The original declaration of the critical need to move away from journal impact factors was followed by a phase of community-building and signature gathering through the website. The stage has been set; all stakeholders, from scientists and policymakers to funders and publishers, need us to take action.”

Curry hopes that those stakeholders (especially researchers, funders, universities) who have been thinking about how to improve their research evaluation processes will be motivated to actually implement alternative or additional evaluation tools once they hear what is already taking place, often under the radar. “At my own institution, Imperial College, which is now a signatory, DORA was a valuable element in helping us to think through how to improve our hiring and promotion procedures.” The problem however, is that a critical mass of grassroots initiative and effort are needed to help propel the scientific community forward in this area. “To be sure,” says Pulverer, “change is not trivial to implement at either individual or institutional level, and one important function of the revitalized DORA project is to point to concrete examples of positive change and best practice.”

The barriers to shifting conversation to action are real: some countries offer direct financial incentives to authors for publishing in certain impact factor journals, in other places tenure and funding are often linked to those same publications. Early career researchers often feel compelled to restrict themselves to those journals (or have no input as to where their work is submitted), delaying publication. Sharing stories of change, both small and large, will help remove institutional and individual bias, integral considerations for DORA to be successful moving forward.

The Future State

To enable the revitalization, coming in 2018 are a new DORA website, extended outreach, and real-world examples of practices at institutions already thinking about and implementing innovative assessment mechanisms. There is no one size fits all solution to the research assessment quagmire, but those actively engaged with DORA believe that the scientific community is empowered to change the system in a grassroots manner. “Every one of us can act to change the system for the better—even without formal policy changes,” says Pulverer. “Research assessment invariably involves the research community, either directly as referees, as hiring principal investigators or in an institutional leadership function.”

The fact that misuse of journal impact factors transcends geography and subject area is illustrated by the wide variety of disciplines and countries represented both in the list of institutional and individual signatories of DORA. Plans for 2018 are shaping up. An influx of new funding has facilitated the hiring of a community manager to help promote DORA online and at conferences and meetings. Says Curry, “There are lots of exciting plans for 2018!” With discussions of appropriate recognition and credit for openly sharing data, datasets, microscopy images and analytical tools, the next five years hold promise for bringing the rhetorical concepts of DORA into practical implementation for the benefit of science and scientists of all career stages and in all geographies.

 

*********************

For this post PLOS interviewed Stephen Curry, Assistant Provost (Equality, Diversity & Inclusion) and Professor of Structural Biology, Department of Life Sciences at Imperial College in London and Bernd Pulverer, Chief Editor, The EMBO Journal, and Head of Scientific Publications at EMBO in Heidelberg.