The University of Calgary signed DORA in January 2021, joining thousands of institutions and individuals globally. Major Canadian funders such as NSERC, CIHR, SSHRC, Canadian Foundation for Innovation and Genome Canada have also signed DORA.
General Recommendations of DORA
- Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.
Recommendations for Institutions
- Be explicit about the criteria used to reach hiring, tenure, and promotion decisions, clearly highlighting, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.
- For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.
Recommendations for Researchers
- When involved in committees making decisions about funding, hiring, tenure, or promotion, make assessments based on scientific content rather than publication metrics.
- Wherever appropriate, cite primary literature in which observations are first reported rather than reviews in order to give credit where credit is due.
- Use a range of article metrics and indicators on personal/supporting statements, as evidence of the impact of individual published articles and other research outputs.
- Challenge research assessment practices that rely inappropriately on Journal Impact Factors and promote and teach best practice that focuses on the value and influence of specific research outputs.
DORA also provides recommendations for funding agencies, publishers and organizations that supply metrics. Read the full Declaration on Research Assessment here.
DORA Implementation at University of Calgary
The University of Calgary is committed to implementing the recommendations of DORA in a way that reflects the diversity and priorities of our research community. The Vice-President (Research) has created a DORA/Research Impact Assessment Working Group to investigate how DORA concepts and practices may be integrated into our research practices and culture.
Dr. William Ghali, Vice-President (Research)
Dr. Penny Pexman, Associate Vice-President (Research)
Dr. Marcello Tonelli, Associate Vice-President (Research)
Dr. Marcello Tonelli, Associate Vice-President (Research) (Co-Chair)
Dr. Penny Pexman, Associate Vice-President (Research) (Co-Chair)
Adnan Ahmed, Director, Office of Institutional Analysis
Dr. Tara Beattie, Associate Dean (Graduate Sciences Education), Cumming School of Medicine
Dr. Lyndsay Campbell, Associate Dean, Faculty of Law (Environmental and Social Governance)
Chelsea Crowshoe, Manager, Indigenous Research Support Team
Dr. Kirsten Exall, Director, Institutional Programs Division, Research Services
Dr. Kathryn Graham, Executive Director, Impact Action Lab, Alberta Innovates
Christie Hurrell, Associate Librarian, Libraries and Cultural Resources
Dr. Guillaume Lhermie, Associate Professor, Faculty of Veterinary Medicine and Director, Simpson Centre for Agricultural and Food Innovation and Public Education
Dr. Stephen MacGregor, Assistant Professor, Werklund School of Education
Jenny McLean, Manager, Strategic Communications and Marketing, Office of the Vice-President (Research)
Dr. David Nicholas, Associate Dean (Research), Faculty of Social Work
Dr. Adesola Olaleye, Research Impact Assessment Specialist, Knowledge Engagement, Research Services
Dr. Kinga Olszewska, Senior Research Advisor, Faculty of Arts
Karen Pankraz, Executive Recording Secretary
Dr. Sylvie Roy, Associate Dean (Research), Werklund School of Education
Dr. Fei Shu, Consultant, Research Assessment, Knowledge Engagement, Research Services
Dr. Francine Smith, Professor, Cumming School of Medicine and representative from the Academic Staff Criteria and Processes (ASCP) Working Group
Dr. Aruna Srivastava, Associate Professor, Faculty of Arts, and representative from the Office of Equity, Diversity and Inclusion
Marcelo Suarez, Research Data Analyst, Office of the Vice-President (Research)
Dr. Stephanie Warner, Manager, Knowledge Engagement, Research Services
Lorna Very, Executive Director, Office of the Vice-President (Research)
Dr. Svetlana Yanushkevich, Associate Dean (Research - Faculty Research & Scholarship) and Associate Dean (Equity, Diversity & Inclusion), Schulich School of Engineering
Joining DORA is another step toward more meaningfully evaluating research output and impact. Our scholars are increasingly engaged in new forms of knowledge creation, and as a university we have emphasized the need to better demonstrate the impact of our research. Including an assessment of impact along with more traditional metrics will help us to fully recognize the value of their work.
Dr. Marcello Tonelli
Associate Vice-President (Research)
The process of implementing DORA at the University of Calgary is iterative and consultative as shown below:
- Review existing literature
Consider the UCalgary context
Listen to stakeholder needs/ perspectives
Select implementation framework
- Co-develop a DORA best-practices “rubric”
Collect, consider and evaluate UCalgary RIA processes
Identify areas of strength/ development
- Develop resources and frameworks for campus
Gather feedback from internal and external stakeholders, and iterate
Build capacity/socialize DORA/RIA
- Support integration of DORA best practices into hiring, tenure & promotion, and funding policies
Share resources with campus and external stakeholders
- Assess effectiveness, uptake and perceptions of DORA best practices and tools
Assess impact at an institutional level
What is the purpose/objective of DORA?
DORA aims to improve the ways in which the outputs of scholarly research are evaluated. The objectives of DORA are to 1) call attention to new tools and processes in research assessment and the responsible use of metrics that align with core academic values and promote consistency and transparency in decision-making; 2) aid development of new policies and practices for hiring, promotion, and funding decisions; 3) spread research assessment reform broadly by working across scholarly disciplines and globally; and 4) call for broader representation of researchers in the design of research assessment practices that directly address the structural inequalities in academia.
Why did UCalgary sign DORA?
UCalgary signed DORA to change how the university assesses research. We believe that an enhanced focus on research content and impact, rather than on bibliometrics alone, is expected to benefit all types of research, from curiosity-driven inquiry to the most applied forms of investigation. This will also help to align policies and practices with those of leading funders, many of whom have also signed DORA.
What is the timeline of DORA implementation at UCalgary?
In 2021, the Vice-President (Research) established a Research Impact Assessment Working Group to review existing research assessment practices and implement the recommendations of DORA at UCalgary. A 5-stage action plan was proposed to begin the DORA implementation across campus in 2021-2023. Meaningful change takes time, communication and collaboration, and we expect full DORA implementation to be iterative and ongoing in the coming years.
How do individual UCalgary researchers get involved in DORA implementation?
As a researcher, you are encouraged to follow the DORA recommendations for researchers, and consider how they apply in your discipline and individual context. You are also invited to actively engage in the DORA implementation process in all phases - within your department, faculty or more broadly. You can share your questions, feedback, or recommendations with the team, or request more information at DORA@ucalgary.ca. Please check back regularly for updates, events and new opportunities.
What are the expectations on UCalgary as an organization now that we have signed DORA? Is DORA a mandate for future research assessment?
DORA is not a mandate or accreditation, but a commitment on behalf of signing individuals and institutions. In November 2022, DORA published a policy on engagement and outreach for organizational signatories. In this statement, DORA asks organizations that sign the declaration to share a public statement detailing their commitment to DORA and responsible research assessment to their communities. It also aims to ensure that an organization's implementation of the DORA principles is informed by ongoing dialogue with staff and students who are involved in research and research-enabling activities.
Frequently Asked Questions | Influence
What changes will DORA bring to the practice of research assessment (hiring, tenure and promotion, awards, grants application)?
As stated in the recommendations for institutions and individual researchers, DORA emphasizes transparent and responsible practices across all aspects of research assessment. This includes a shift away from journal-based metrics such as the Journal Impact Factor in assessing individual research contributions or researchers, and an emphasis on considering a broader range of research outputs and impacts. This will be interpreted in ways appropriate for each Faculty and Department, with guidance from relevant governing bodies for institutional processes (for example, academic tenure, promotion and merit). Importantly, researchers will still be expected to contribute high-quality original research outputs that peer scholars within that field recognize.
How might UCalgary researchers benefit from DORA-aligned research assessment?
DORA stands to benefit all researchers. The principle that researchers should be recognized for the full array of research outputs (papers, data, code, protocols, etc.) enables researchers to gain credit for work that matters to them and to their scholarly field. Ensuring greater transparency in research assessment promotes equity and appropriate assessment methods for researchers in diverse disciplines and at various career stages. Traditional scholarly research outputs continue to be highly valued and assessed based on their own merit, which maintains the integrity of important activities such as peer review. By considering a broader range of impact measures in research assessment, researchers who conduct important community-engaged work may look forward to clearer processes to assess and value these time-intensive research activities.
How will DORA affect early career researchers (graduate students, postdocs, new faculty hires, etc.)?
As a Declaration, DORA represents a commitment by the University of Calgary to implement the recommendations across all aspects of the University that constitute research assessment – including early career researchers. DORA's emphasis on transparency in reaching hiring, tenure, promotion and merit decisions and assessing research outputs on their own merit rather than on journal-based measures such as the Journal Impact Factor is likely to be welcome news for early career researchers.
Will funding agencies, including the Tri-Agencies (CIHR, NSERC, and SSHRC), apply DORA to their grant applications?
Major Canadian funders CIHR, NSERC and SSHRC, as well as the Canada Foundation for Innovation (CFI) and Genome Canada, have all signed DORA, stating their commitment to applying DORA recommendations to funding applications. Individual grant applications are increasingly including reference to DORA, and careful attention should be paid to the specific criteria of each call. In 2022, NSERC updated their Guidelines on the assessment of contributions to research, training and mentoring.
How are other institutions implementing DORA?
Becoming a DORA signatory is part of a broader global movement toward responsible research assessment, and several international institutions have documented their approaches to implementing the recommendations of DORA. There is considerable variation in approaches. The University of Calgary was the first university in Canada to sign DORA, in January 2021. We have since been joined by McGill University, Université du Québec à Trois-Rivières Université du Québec à Montréal - UQAM and Université de Montréal. The University of Calgary is consulting with other institutions in Canada and internationally to align with best practices nationally and worldwide.
How does DORA benefit our community?
DORA encourages institutions to consider a broad range of impact measures, including qualitative indicators of research impact, such as influence on policy and practice. Although traditional research assessment practices do not always recognize connection to and impact on community, positive impacts of research on local communities are expected to be encouraged and promoted by research assessment following DORA recommendations.
Frequently Asked Questions | Traditional Metrics
The Journal Impact Factor has been used as an indicator for research assessment for several decades. Why does DORA recommend not using it?
The Journal Impact Factor (JIF) was developed to measure the quality of scholarly journals for the purpose of selecting journal subscriptions, but it has been mistakenly used to assess the quality of individual research papers in the past decades. Considering the skewed distribution of citations to journals, the JIF of a scholarly journal is attributed to very few highly cited papers; it does not accurately represent the quality of individual research papers published in a given scholarly journal on average (Larivière & Sugimoto, 2019). Thus, DORA highly recommends not using the JIF to assess individual research papers. In consideration of disciplinary differences, DORA does not provide a replacement to the JIF but recommends using a combination of various indicators (quantitative or qualitative) for research assessment across different disciplines.
Does DORA recommend against using traditional metrics in research assessment?
No. DORA objects to the abusive use of metrics (i.e. a single bibliometric indicator) in research assessment but recommends the responsible use of metrics. Since each bibliometric indicator has its applicable scope and limitations, DORA recommends selecting the most appropriate indicators in research assessment.
Will DORA apply non-traditional metrics in research assessment?
Yes. Although traditional metrics (e.g., number of publications, number of citations, etc.) could represent the scholarly impact of research, they cannot be used to measure non-scholarly impact of research, including impact on economic development, society, community, policy and so on. Some non-traditional metrics (i.e., altmetrics) could be used in research assessment as a supplement.
How does DORA apply to research assessment across disciplines, particularly those in which the JIF is less used?
Although originally written to address concerns in the field of cell biology, DORA principles have much to offer in disciplines in which the JIF or journal publications are not the primary mode of research output or assessment. Importantly, DORA highlights the importance of assessing research on its own merits and applies a broad definition for research outputs and impact, not limited to journal publications.
Frequently Asked Questions | Communicating Research Impact
How can a UCalgary researcher demonstrate their research performance in a DORA-aligned way?
Specific requirements for hiring, tenure, promotion and merit will be determined by the appropriate policies and procedures in the researcher's Faculty and the University of Calgary GFC Academic Staff Criteria & Processes Handbook. According to the DORA Recommendations for Researchers, researchers should focus on the content of the scholarly contribution rather than simply publication metrics or the identity of the journal in which it was published. DORA also recommends researchers be recognized for research outputs in various formats; not only publications but also patents, software, license, dataset, reagents, intellectual property, highly trained new researchers, and so on. Meanwhile, researchers could demonstrate their different types of research impact, including not only the scholarly impact (citations, reach, uptake within the scholarly community) but also broader impact on policy, practice, community, etc. For journal publications, researchers are reminded to focus on article-level metrics rather than journal-level. For more details, please contact us at DORA@ucalgary.ca.
What tools or resources are available for UCalgary researchers to report their research impact?
Various resources (bibliometric databases such as Scopus, Web of Science, etc.) or tools (e.g., various metrics, toolkits, etc.) may be used to help researchers report their research impact. We recommend UCalgary researchers use the tools and resources supported by Libraries and Cultural Resource to compile metrics regarding their research performance. Please visit the Manage Your Research Identity and Track Your Impact guide to learn more. The Knowledge Engagement Impact Assessment Toolkit provides a framework for evaluating the impact of partnered research both quantitatively and qualitatively, and can be adapted as desired.
How can a UCalgary researcher describe their research contributions to society, community, policy or practice?
Researchers could use either altmetric indicators or narratives (e.g., narrative CV) to describe their non-scholarly impact including their research contribution to society, community, or policies. Consider combining both numbers and stories in order to provide a complete picture of your research contributions. Watch this page for new tools and resources for communicating research impact.
What kind of research outputs are recognized by DORA recommendations?
DORA applies a broad definition of research outputs, not limited to publications (journal articles, books, monographs, and book chapters). DORA encourages considering research outputs in various formats such as patents, software, licenses, datasets, training, reagents, public engagement, oral presentation, intellectual property, etc. that are relevant to the scholarly discipline.
We are currently working on engagement with each of the existing university strategies and will have future updates about intersections.
- Faculty Library Support Guide – Scholarly Publishing
Navigating DORA webinar series
Please find recordings from our past sessions below.
- The Evolution of Research Assessment and Responsible Use of Metrics, Dr. Vincent Larivière View Recording.
- Stop Using the h-index: Educating Researchers About Scholarly Metrics, Dr. Stefanie Haustein View Recording