Research Excellence Framework Evaluation Essay

Christmas is coming, the goose is getting fat and, after a long wait, your REF results are on their way. Will you be merrily stuffing your face with turkey at the faculty Christmas meal, or shamefully struggling through a slice of humble pie? Only time will tell, but meanwhile, I’ll distract you with my idiot’s guide to REF success.

REF, which stands for Research Excellence Framework, is the way that the UK assesses academics’ research output. It works like this: you submit four publications from the preceding five years to a panel of experts who will then skim-read them and judge you mercilessly. Your future funding depends on their judgment.

If you think that seems like a rather incomprehensible and misguided evaluation process, you’d probably be right, which is why the REF has received so much flak. It also seems that unscrupulous universities have been able to game the system. One commentator managed to muster a “partial, qualified, cautious defence” of the process, but the overall picture hardly inspires confidence.

Research, research, research

At its core, REF is all about research. The best advice therefore is to make sure that you do lots of it. While elsewhere time is money, in academia time is research, so we need to buy you some time.

First and foremost, cease all non-impactful activities, particularly teaching. Ideally you should abandon your students completely, but you can always palm your commitments off on gullible colleagues, praising their expertise and inviting them to give guest lectures to your classes. Otherwise, simply skip the teaching, require submission of a paper only, and give all your students As or Bs regardless of quality (this will also save you time on marking). At the very least you should stop responding to student emails and cancel all office hours.

If you absolutely must teach, at least amuse yourself and make it as low-effort as possible. One US-based academic, for example, offers a course entitled Wasting time on the internet, the syllabus for which mandates “distraction, multi-tasking, and aimless drifting”.

In the event that you find yourself stranded without any publications, you can still take a stab at REF success by simply churning out a few semi-incomprehensible stream of consciousness pieces. There is always a small chance that the panellists will be so baffled that they assume that you are a bona fide four-star academic whose writing is beyond the comprehension of their meagre minds.

Be strategic

Don’t write a book or extended monograph: the REF makes no distinction between research outputs, so there is no incentive to undertake long-term projects. Also don’t bother with risky, visionary or imaginative projects unless you can be absolutely certain that you will get a publication out of it. No publication means no impact.

Publish only in the best known journals. Aim high: Nature, Science, the American Journal of Potato Research or similarly excellent journals are good choices.

Choose the panel that you submit to wisely. It turns out that the interdisciplinarians may have an advantage here as they can pick and choose: geography department showing you no love? Send your stuff to sociology instead. If you are a legal researcher, why not throw the chemists a curveball and send your research their way?

Have an impact

Once you’ve done your research, you should develop an unhealthy obsession with impact. Most of us have reflected on our h-index at some point, but if you want to be truly influential you need to work on your Kardashian index. This means amassing a huge Twitter following, thereby being ostensibly influential while actually contributing very little to society. Of course, you shouldn’t neglect your h-index completely just artificially inflate it by citing yourselves 50-odd times in every paper.

Finally, you need to stand out from the crowd. I’m not sure how this helps, but you’ll at least have a bit of fun. Get yourself a TV show or write a cookery column for an aspirational magazine. Wear a snazzy hat, get a badass tattoo, and remember to dress for academic success.

There you have it. Now all there is to do is make a cup of tea, grab yourself a snack, and wait… In the meantime, send your amusing #REFtips to @AcademiaObscura.

Enter the Guardian university awards 2015 and join the higher education network for more comment, analysis and job opportunities, direct to your inbox. Follow us on Twitter @gdnhighered.

The Research Excellence Framework is the successor to the Research Assessment Exercise. It is an Impact evaluation; assessing the research of British higher education institutions. It was used in 2014 to assess UK research during the period 2008–2013.[1][2]


In June 2007 the Higher Education Funding Council for England (HEFCE) issued a circular letter announcing that a new framework for assessing research quality in UK universities would replace the Research Assessment Exercise (RAE), following the 2008 RAE.[3] The following quote from the letter indicates some of the original motivation:

Our key aims for the new framework will be:

  • to produce robust UK-wide indicators of research excellence for all disciplines which can be used to benchmark quality against international standards and to drive the Council's funding for research
  • to provide a basis for distributing funding primarily by reference to research excellence, and to fund excellent research in all its forms wherever it is found
  • to reduce significantly the administrative burden on institutions in comparison to the RAE
  • to avoid creating any undesirable behavioural incentives
  • to promote equality and diversity
  • to provide a stable framework for our continuing support of a world-leading research base within HE.

The letter also set out a timetable for the development of the REF. HEFCE undertook a consultation exercise during September–December 2009, soliciting responses from stakeholders on the proposals.[4] These include for example the response from Universities UK,[5] and the response from the University and College Union.[6]

In July 2010 (following the May 2010 general election), the Universities and Science minister David Willetts announced that the REF will be delayed by a year in order to assess the efficacy of the impact measure.[7]

In July 2016, Lord Nicholas Stern's review was published, drafting general guidelines for the next REF in 2021.[8] In general, the review was supportive with the methodology used in 2014 to evaluate universities' research, however it emphasised the need for more engagement with the general public and the increase of number of case studies that undertook interdisciplinary approach.[8] The team at Loughborough University Business and Economic School have been experimenting with crowdfunding for research in order to increase the university's researchers' public engagement.[9]

Research Impact[edit]

REF's impact was defined as "an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia".[10]

Grading criteria[edit]

Submissions are assessed according to the following criteria:[11]

  • Four star: Quality that is world-leading in originality, significance and rigour.
  • Three star: Quality that is internationally excellent in originality, significance and rigour but which falls short of the highest standards of excellence.
  • Two star: Quality that is recognised internationally in originality, significance and rigour.
  • One star: Quality that is recognised nationally in originality, significance and rigour.
  • Unclassified Quality: that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment.

Performance rankings[edit]

Two publishers, The Guardian[12] and Times Higher Education,[13] produce overall rankings of multidisciplinary universities based on power and quality(GPA).

Power rankings aim to show universities with a breadth of quality, while Quality rankings aim to show the depth of quality.

The Guardian Power rankings only consider rankings graded at Four and Three star. While Times Higher Education Power rankings consider rankings across all gradings.

An additional Quality ranking is the one ranking institutions according to the proportion of their research graded as "Four star". That is, submitted researches graded as "Quality that is world-leading in originality, significance and rigour".[14]

Controversies and criticism[edit]

Criticism has focused on the element of the REF that addresses the "impact" of research. The articles below raise two objections. The main one is that "impact" has been defined to mean impact outside the academy. If researchers were required to pursue this form of impact, it would undermine academic freedom. The other is that impact—as currently construed—is hard to measure in any way that would be regarded as fair and impartial.[15][16][17]

The Higher Education Funding Council for England argue that their measure of "impact" is a broad one which will encompass impact upon the "economy, society, public policy, culture and the quality of life".[15]. However, the assessment structure does make what impact practically can be claimed rather narrow (4 page limit, no method section, 10 impact references, 10 research references and only 1 page to summarize the research and the impact respectively). These strict discursive guidelines alongside the REF's dated notion of how research impact functions (teaching research impact excluded, linear model, etc.) does restrict what impact is suited practically more for the assessment.

Another area of criticism, which the REF inherited from the structure of the RAE, is that for most full-time staff members submission normally consists of four published 'research output items'. There is no recognition of the difference between a book and an article in terms of research value. Therefore, the REF system discourages long term projects that strive for excellence. This problem is particularly evident in the humanities, where most of the ground-breaking research is traditionally not published in articles. Therefore, many researchers are pushed towards a relatively mediocre activity, which will allow them to produce one or two books during the assessment period, but not the kind of monograph that normally would need four or five years of research and writing.

Moreover, the system of the four published items discourages long-term projects with relatively high research risk in the sciences as well, since researchers are reluctant to engage in projects or experiments that may not be successful and may not lead to a publication. Since most of the ground-breaking research in the sciences takes place with precisely such risky and imaginative projects, the type of research activity that is encouraged by the REF structure is quite conservative. Also, in terms of the impact of the examined research, in the history of the sciences and the humanities it is not unusual to take some time until the full impact of a discovery is made. The present system has a vista of only four or five years.

The Times Higher Education also revealed that some universities appeared to be "gaming" the REF system. This included "REF Poaching", in which staff with established research records were headhunted from their universities immediately before the REF, giving the poaching institution full credit for their publications without having taken the risk of supporting the researcher. It also included employing large numbers of staff on 0.2 FTE contracts, the lowest level of employment that qualifies them for REF submission.[18]

In addition to such concerns about what really can be measured by four research output items, and how impact may be measured, the whole system is often criticized as unnecessarily complex and expensive, whereas quality evaluation in the digital age could be much simpler and effective.[19]

The system, with its associated financial implications, has also been criticised for diverting resources from teaching. As such, increases in student fees may often not have resulted in more staff time being spent on teaching.

In July 2016, Lord Nicholas Stern's review was published, drafting general guidelines for the next REF in 2021.[8] One of the recommendation was to increase research public engagement. Research engagement means enhancing delivery of the benefits from research. It also means making the public more aware of the research findings and their implications. One mechanism for public engagement is crowdfunding for research, where dedicated platforms host crowdfunding campaigns for university research, in a range of topics. Crowdfunding for research has two advantages: one, it is a source for a relatively high guaranteed funding, with a rate of around 50%, second, it is a very effective tool to engage with the general public.[9]

One problem that the Stern review did not address in relation to the research impact assessment, is that the structure of case study design template on which impact is assessed, does not contain a method section, and thereby making the assessment of what type of impact was claimed a rhetoric game of who can claim the most. In general, grand claims are incentivized by the assessment structure.[citation needed] Furthermore, when checking the reference of current claims, these where either not accessible (e.g. the relevant websites were taken down),[citation needed] referenced in such a way that it didn't reflect self-authorship[citation needed] or testimonials of individuals connected to the researcher.[citation needed]. Sayer (2014)[20] criticizes the overall peer review of the REF process, stating it is only a 'poor simulacrum of standard academic quality' and that the assessment process is further complicated by the sheer workload of the assessment. Now a RAND study found that the majority of the references were never consulted, certain assessment panels were discouraged from using the internet and the reference help structure of the REF took sometimes two weeks to produce associated references [21]. Thereby, the external impact focus disciplines the assessment into focusing on external values[22]. This ossifyies contemporary societal values into academic culture, due to the missing method section where rigor in impact claim that challenge current social values potentially could have been shown. In an assessment of current impact submissions none of the reported impacts include such claims.[citation needed]

See also[edit]


  1. ^"Results & submissions : REF 2014". Retrieved 22 December 2014. 
  2. ^Atkinson, Peter M. (11 December 2014). "Assess the real cost of research assessment". World View. Nature (paper). 516 (7530): 145. doi:10.1038/516145a. 
  3. ^Eastwood, David (6 March 2007). "Future framework for research assessment and funding". HEFCE. circular letter number 06/2007. Archived from the original on 2 February 2010. 
  4. ^"Research Excellence Framework: Second consultation on the assessment and funding of research". HEFCE. September 2009. 2009/38. Retrieved 10 January 2015. 
  5. ^"Universities UK response to HEFCE consultation on the Research Excellence Framework (REF)". Universities UK. 13 December 2009. Archived from the original(.doc) on 16 July 2011. 
  6. ^"Response to the Research Excellence Framework: Second consultation on the assessment and funding of research"(PDF). University and College Union. December 2009. 
  7. ^Baker, Simon (8 July 2010). "REF postponed while Willetts waits for impact 'consensus'". Times High. Educ. 
  8. ^ abcStern, Lord Nicholas; et al. (July 2016). "Building on Success and Learning from Experience"(PDF). UK Government. Retrieved 3 January 2017. 
  9. ^ abRubin, Tzameret (2017). "Is it possible to get the crowd to fund research, isn't it the government's role?". AESIS. Retrieved 2016-12-23. 
  10. ^England, Higher Education Funding Council for. "REF impact - Higher Education Funding Council for England". Retrieved 2017-01-09. 
  11. ^"Assessment framework and guidance on submission"(PDF). Research Excellence Framework. July 2011. p. 43. REF 02.2011. 
  12. ^
  13. ^
  14. ^
  15. ^ abShepherd, Jessica (13 October 2009). "Humanities research threatened by demands for 'economic impact'". Education. The Guardian. London. 
  16. ^Oswald, Andrew (26 November 2009). "REF should stay out of the game". The Independent. London. 
  17. ^Fernández-Armesto, Felipe (3 December 2009). "Poisonous Impact". Times Higher Education. 
  18. ^Jump, Paul (26 September 2013). "Twenty per cent contracts rise in run-up to REF". Times Higher Education. 
  19. ^Dunleavy, Patrick (10 June 2011). "The Research Excellence Framework is lumbering and expensive. For a fraction of the cost, a digital census of academic research would create unrivalled and genuine information about UK universities' research performance". London School of Economics. 
  20. ^Sayer, D. (2014). Rank hypocrisies: The insult of the REF. Sage.
  21. ^
  22. ^

External links[edit]

0 thoughts on “Research Excellence Framework Evaluation Essay

Leave a Reply

Your email address will not be published. Required fields are marked *