Friday, July 31, 2009

"Impact" may count for 30% in REF

Readers may know that the "RAE" (e.g., Research Assessment Exercise) is being replaced by a "REF" (e.g., Research Excellence Framwork) in the UK. All "units of assessment" (in essence, all departments) are assessed on the quality of their research. This is based upon the quality of publications, graduate student numbers, and other factors.

Previously, the RAE gave marks for the "esteem" each department held. This appears to be replaced now by "impact" -- so what's in a name?

The latest from the Times Higher Education is that "impact" will account for 20-30% of a subject's final result. This will be assessed during a five year period: 2008-12. The details are here. Departments would submit a narrative with a few "case studies" --- these would then be judged by "panels of academics and users."

There are several concerns . First, the academics who will judge impact may well be members of the relevant REF panel. If so, then this seems unproblematic. What does seem problematic is the selection of "users": who are they are (or, more specifically, who are they not)? If the "user" is, say, society in general, then who speaks to "impact" then? MPs? Much could hinge on which non-academics are chosen. Of course, the UK has a body that is thought to "represent" in some broad sense considered public opinion by experienced, distinguished persons. This body is the House of Lords and several hundred sit on its benches. This is surely too large a group for a research assessment. However, a group of five, ten, even fifteen may be better to "manage," but many critical voices will be left out --- and this may be a real concern for some departments.

My second concern is more serious. Almost one-third of a department's success in the REF will be judged upon the "impact" of its research over a five year period. Surely, departments wanting to score high will aim to produce immediate impact research with less regard for longer lasting impact. My reasons:

1. One reason why there may be less regard for longer lasting impact is that government policy changes so often. How can a department make plans for the future when -- in one assessment (RAE2008) - "impact" is not directly measured and a few years later -- in a different assessment (REF2013) -- it is. It is difficult to make long term plans when frequent changes are on hand.

2. A second reason is that, if the REF were to continue beyond 2013, the impact of research prior to each five year period may be relevant. We may think then that - eureka! - there is an incentive here to aim for long lasting impact. I doubt it. The system prioritizes making a big impact now in a current time period. An impact that is large overall, but measured over time may be less attractive for departments aiming to score high. Moreover, the impact of past research is research by department members: they may well leave. Departments prioritizing longer lasting impact may make more likely the poaching of staff who produce this research. While there will be incentives for departments to poach "stars" likely to score impact in the short term, this may be more difficult to coordinate given the more brief time period. Thus, again, there is an incentive for short term impact for maximum results.

Together, these concerns should be troubling. The government reminds us that universities should demonstrate "impact" on account of receiving public funds. If this is so, then why do universities seem singled out? Do we see MP's working harder to demonstrate their positive "impact" on account of their being 100% funded by tax payers? Hardly. (Instead, we had the expenses fiasco.)

I cannot help but think the current drive for universities to demonstrate "impact" is more about universities doing more for less. The pressure here is deliver even more bang for the government's buck. Of course, there can hardly be any problem with strongly encouraging -- even demanding -- that universities deliver the very best service possible. But this is not about that.

If the government wants British universities to perform even better, then the trick is not writing "case studies" and narratives about "impact" that will appear on yet another website the public may not read. Instead, it should invest far more in higher education than it does --- and with fewer strings attached. Politicans may be like the REF2013 and given to short term thinking, but a highly educated public is a great public good with long term benefits. If the "impact" of such benefits is not readily discernible to ministers, then I highly doubt case study narratives will do the trick . . . although I hope to be surprised.

No comments: