Research Excellence Framework (REF)

Introduction

The Research Excellence Framework (REF) is the new system for assessing the quality of research in UK higher education institutions (HEIs). In previous years, research quality has been assessed periodically through the Research Assessment Exercise (RAE). The Higher Education Funding Councils are currently consulting on proposals for the new framework. The first REF exercise is due to be completed in 2013.

The REF will focus on three elements, which are:

Outputs - The primary focus of the REF will be to identify 'excellent research' of all kinds, as demonstrated by the quality of research publications and other outputs.

Impact - Significant additional recognition will be given to researchers who are able to demonstrate the 'economic and social impact' of their research.

Environment - REF will also assess how far the research environment supports a continuing flow of 'excellent research and its effective dissemination and application.'

The funding councils are proposing the following weightings between the three elements:

What are the problems with the proposals?

We continue to have major reservations about a research assessment process based on universities selecting particular academic staff for inclusion or non-inclusion. The 2008 RAE resulted in a significant amount of unfair and punitive treatment of academic staff and we fear that similar practices will occur in the 2013 REF.

However, the biggest problem with the HEFCE consultation document is the proposal to base 25 per cent of the REF on an assessment of the 'economic and social impact' of research. Over seventeen and a half thousand academics, higher education professionals and researchers have signed a UCU petition calling for the UK funding councils to withdraw the 'impact' proposals.

Academics are concerned that the proposals will:

• undermine support for basic research across all disciplines as well as disproportionately disadvantaging research in the arts and humanities
• lead to the further commercialisation, and therefore narrowing, of the research agenda.

The impact proposals also:

• appear to have been drawn up in haste and lack the evidence necessary to gain the support and confidence of the university community; and
• will add greatly to the bureaucratic nature of the research assessment process.

From a constitutional perspective, we are very concerned about how the impact proposals have emerged. Although HEFCE began its review of the RAE in early 2007, impact assessment did not feature until the Secretary of State's funding letter of 22 January 2009 in which he emphasised that the REF should take better account of the impact research makes on the economy and society and gave examples of the sort of activities that he favoured.

We believe that this represents an unprecedented, direct interference by government in the content, nature and direction of the research carried out in our universities. It has been accepted for some years that government indicates broad priority areas of research which are taken into account by the research councils when they develop their funding programmes. However, the fundamental principle at the heart of the other side of the dual support system is that research is assessed through peer review on the basis of its intrinsic quality, not the changing policies of governments or the perceived needs of business. The implications of the Secretary of State's intervention are a matter of public concern that have not been adequately examined and debated.

International approaches to research impact

One of the most extraordinary aspects of the current proposal is that this kind of impact assessment does not seem to have been tried or tested anywhere in the world.

The Australian Research Quality and Accessibility Framework (RQF) is the most worked up example of measuring impact in a higher education context. However, it was abandoned in 2007 before it was properly implemented. At the time, Senator Kim Carr, the new Minister for Innovation, Industry, Science and Research, said the 'The RQF is poorly designed, administratively expensive and relies on an 'impact' measure that is unverifiable and ill-defined'. A recent report from RAND Europe suggests that the 'impact indicators' developed within the RQF are 'not sufficiently developed and tested to be used to make funding decisions' (p.55). In addition, only 10% of funding decisions were meant to be influenced by the RQF impact assessment (p.4).

Nor is the new Dutch-based system - Evaluating Research in Context (ERiC) - a straightforward model for UK higher education. For example, the RAND Europe report has highlighted the administrative complexity of the ERiC system and has raised doubts about whether a scheme developed for a relatively small HE system can be adapted for a much larger one. In addition, unlike the HEFCE proposals, the Dutch evaluation system is not tied to the funding allocation.

Consequently, HEFCE's proposal to allocate potentially 25% of Quality-Related (QR) research funds on an untested and experimental assessment of 'impact', seems foolhardy in the extreme.

What are we calling for instead?

We are calling for the abolition of the 'impact' element within the REF. Instead, UCU recommends the following weightings:

Outputs: 80%

Environment: 20%

Is there a role for assessing economic and social impact?

We believe that economic indicators such as 'creating new businesses' and 'commercialising new products or processes' are best assessed through the Higher Education Innovation Fund (HEIF) and comparable funding streams in other parts of the UK.

For core research funding it is counterproductive for government to make funding for the best research conditional on its perceived economic and social benefits. It is often difficult to predict which research will create the greatest practical impact. History shows us that in many instances it is curiosity-driven research that has led to major advances in the sciences and the arts.

We believe that engagement with 'research users' should remain part of the assessment of 'research environment'.

What other changes are we proposing for the REF?

UCU recommends:

• union involvement in the development of the equalities and diversity framework
• making the 'support for research staff' a specific REF policy aim
• greater encouragement for institutions to submit applied outputs (the applicability of applied research should part of its quality assessment, rather than a separate 'impact' measure)
• reducing the number of assessed outputs from four to three.

Research concentration and the future of research

While the current debates over the REF focus on assessment methodology, there is a need for a wider debate on the funding for research. Research funding in UK higher education is already one of the most concentrated in the world. We question the current campaign for 'more research concentration where institutions are strongest' (Higher Ambitions, p.4). Further concentration of research funding would carry with it the risk of reduced research capacity for some regions, greater differential experiences for students and a reduction in the diversity of the UK's research base.

Investment in future and potential success is just as important a part of the process of funding research as is investment in existing excellence. In order to counterbalance to some extent the concentration effect of research selectivity, we believe there should be a 'seedcorn' fund available to university researchers for starting up new programmes.

A new Research Development fund should be established by the funding councils. It should be available on application to any researchers in departments with a low proportion of RAE funding, or in departments which did not enter into the 2008 RAE. It should be available for all areas of research, across all disciplines, and applications should be assessed by established peer review procedures. In order to have a real impact on the quality of research and teaching in our universities, the size of the fund would need to be at least 10 per cent of the total amount currently provided by the three funding councils for research. Given the importance for the future of the research base we believe that the new fund should be funded through additional public resources rather than by reallocating existing funds.