Use these ten principles to guide research evaluation, urge Diana Hicks, Paul Wouters and colleagues.

This review of the international literature on evaluation systems, evaluation practices, and metrics (mis)uses was written as part of a larger review commissioned by the Higher Education Funding Council for England (HEFCE) to inform their independent assessment of the role of metrics in research evaluation (2014–5). The literature on evaluation systems, practices, and effects of indicator uses is extremely heterogeneous: it comprises hundreds of sources published in different media, spread over disciplines, and with considerable variation in the nature of the evidence. A condensation of the state-of-the-art in relevant research is therefore highly timely. Our review presents the main strands in the literature, with a focus on empirical materials about possible effects of evaluation exercises, ‘gaming’ of indicators, and strategic responses by scientific communities and others to requirements in research assessments. In order to increase visibility and availability, an adapted and updated review is presented here as a stand-alone—after authorization by HEFCE.

Given the increased role of bibliometric measures in research evaluation, it is striking that studies of actual changes in research practice are rare. Most studies and comments on ‘a metric culture’ in academia focus on the ideological and political level, and there is a clear shortage of empirical studies that analyze how researchers handle demands for accountability in context. In adopting a mixed-methods approach involving both bibliometric data and answers form questionnaires, we provide an in-depth study of how researchers at the faculty of Arts at Uppsala University (Sweden) respond to the implementation of performance-based research evaluation systems. Publication patterns from 2006 to 2013 show that journal publications, especially English-language ones, are increasing, and the proportion of peer-reviewed publications has doubled. These changes are in line with the incentives of the evaluation systems under study. Answers to the survey confirm that scholars are conscious about this development, and several respondents articulate a disagreement between disciplinary norms and external demands. However, disciplinary background as well as career stage or academic age appears to have a significant influence on how individual researchers react to the instigation of evaluation systems. Finally, responses to national and local evaluation regimes are complex, localized, and dependent on many factors. In-depth contextualized studies of research practices are needed in order to understand how performance-based funding systems influence academic research on the ground.

While quantitative performance indicators are widely used by organizations and individuals for evaluative purposes, little is known about their impacts on the epistemic processes of academic knowledge production. In this article we bring together three qualitative research projects undertaken in the Netherlands and Austria to contribute to filling this gap. The projects explored the role of performance metrics in the life sciences, and the interactions between institutional and disciplinary cultures of evaluating research in these fields. Our analytic perspective is focused on understanding how researchers themselves give value to research, and in how far these practices are related to performance metrics. The article zooms in on three key moments in research processes to show how ‘thinking with indicators’ is becoming a central aspect of research activities themselves: (1) the planning and conception of research projects, (2) the social organization of research processes, and (3) determining the endpoints of research processes. Our findings demonstrate how the worth of research activities becomes increasingly assessed and defined by their potential to yield high value in quantitative terms. The analysis makes visible how certain norms and values related to performance metrics are stabilized as they become integrated into routine practices of knowledge production. Other norms and criteria for scientific quality, e.g. epistemic originality, long-term scientific progress, societal relevance, and social responsibility, receive less attention or become redefined through their relations to quantitative indicators. We understand this trend to be in tension with policy goals that seek to encourage innovative, societally relevant, and responsible research.

The range and types of performance metrics has recently proliferated in academic settings, with bibliometric indicators being particularly visible examples. One field that has traditionally been hospitable towards such indicators is biomedicine. Here the relative merits of bibliometrics are widely discussed, with debates often portraying them as heroes or villains. Despite a plethora of controversies, one of the most widely used indicators in this field is said to be the Journal Impact Factor (JIF). In this article we argue that much of the current debates around researchers’ uses of the JIF in biomedicine can be classed as ‘folk theories’: explanatory accounts told among a community that seldom (if ever) get systematically checked. Such accounts rarely disclose how knowledge production itself becomes more-or-less consolidated around the JIF. Using ethnographic materials from different research sites in Dutch University Medical Centers, this article sheds new empirical and theoretical light on how performance metrics variously shape biomedical research on the ‘shop floor.’ Our detailed analysis underscores a need for further research into the constitutive effects of evaluative metrics.

This document presents the Bonn PRINTEGER Consensus Statement: Working with Research Integrity—Guidance for research performing organisations. The aim of the statement is to complement existing instruments by focusing specifically on institutional responsibilities for strengthening integrity. It takes into account the daily challenges and organisational contexts of most researchers. The statement intends to make research integrity challenges recognisable from the work-floor perspective, providing concrete advice on organisational measures to strengthen integrity. The statement, which was concluded February 7th 2018, provides guidance on the following key issues: § 1. Providing information about research integrity § 2. Providing education, training and mentoring § 3. Strengthening a research integrity culture § 4. Facilitating open dialogue § 5. Wise incentive management § 6. Implementing quality assurance procedures § 7. Improving the work environment and work satisfaction § 8. Increasing transparency of misconduct cases § 9. Opening up research § 10. Implementing safe and effective whistle-blowing channels § 11. Protecting the alleged perpetrators § 12. Establishing a research integrity committee and appointing an ombudsperson § 13. Making explicit the applicable standards for research integrity