Thursday, October 22, 2009

On measuring the "impact" of research

There have been various debates, such as here, and petitions, such as here, concerning the role of "impact" in the forthcoming Research Excellence Framework (REF) in the UK. British departments (or "units of assessment") are assessed every few years to measure their research quality. The better a department's score, the greater funding it would earn via funding councils. Thus, research excellence pays and highly respected work means ££ in UK higher education circles.

In the past, these evaluations have been somewhat different. For example, the last evaluation was a "Research Assessment Exercise" (RAE) that measured research quality from January 1, 2001 through December 31, 2007. Results were announced in 2008. Departments were measured by the quality of their publications, research environment, and "esteeem" --- and given g.p.a. out of a possible perfect 4.0. Before RAE 2008, departments were ranked in categories of "5" or less.

The current REF plans look likely to assess departments between January 1, 2008 - December 31, 2012 with the results announced in 2013. Publications and environment will remain as categories, but "esteem" will now be replaced by "impact" --- and impact will have a greater weight than esteem. The current proposal is publications 60%, impact 25%, environment 15%.

These plans may look far from unreasonable. After all, why shouldn't impact have some importance? There are several important concerns worth raising:

1. We don't know what "impact" is. What counts as "impact"? Is a popular blog impact? This has not yet been defined.

2. We don't know how "impact" will be measured. It is proposed that per every five-ish department members every unit will offer a "narrative" of "an impact case study": how do we know one narrative is better than another? This is left very unclear.

3. "Impact" in the short-term is bad for UK research in the long-term. The impact period is 2008-12. One problem with this is that departments will do what they can to demonstrate impact for this period because impact in this period will benefit scores. This may be counterproductive to producing long-term impact.

4. There are doubts this is genuinely about justifying higher education funding to the public. For one thing, we may doubt how a department's "impact case study" narrative submitted in REF 2013 will sway the public. One reason to have this doubt is information is readily provided concerning RAE2008 submissions, but most in the public I speak with don't know of the relevant sites, etc.

For another thing, the Times Higher reports today this story:

"[. . .] Academics have reacted angrily to an internal research council document that says that the Government - not the academic community or the public - is the "primary audience" for its campaign to improve the economic and social impact of the research it funds.

The document, leaked to Times Higher Education, outlines the contents of a presentation made to senior staff at the Engineering and Physical Sciences Research Council. It says: "We need to show (the Government) the importance and relevance of the research we fund to current and future global challenges in order to secure future funding."

Academics are named a "secondary audience" with the public given third priority.

The document says that the campaign "is deliberately blatant in its attempt to address the immediate demand" from the Government to demonstrate impact.

Researchers said the leaked document showed that the research councils were playing politics instead of protecting the interests of scholarship. [. . .]"

5. The REF is already under way! Despite these major concerns, there is a pilot study know on impact . . . that may not be concluded until next year. Thus, it may not be until the REF2013 is halfway finished that we will have any knowledge about how we'll assessed up to 25% of our final scores . . . including the last few years! Thus, today what we do or do not do may or may not count as "impact": we'll be assessed all the same according to a criteria to be announced later that will be applied retrospectively to work that has already happened.

What is the solution? Well, one way forward would be to postpone the measurement of "impact" until post-REF2013. It does not make much sense using an indicator that is still not defined nor tested . . . and won't be so until literally a year or two before the end of the current assessment period. It's important to get these decisions correct. This will not happen by rushing through this proposal.

NOTE: There are echoes here of where the government previously pushed for bibliometric measures to assess all departments, only to withdraw these measures. I would not be surprised if "impact" had the same fate.

Wednesday, October 21, 2009

New Poll: 48% Of Republicans Say Obama Does Not Love America -- 27% Say He Does

Details are here. The Party of No is fast becoming the Party of Don't Take Us Seriously.

Petition against "impact" in Research Excellence Framework

. . . can be found here. Please do sign it, if you have not done so already. In less than week, it has attracted over 1,500 signatures. Let us hope it will have an "impact" . . . .

Tuesday, October 20, 2009

Friday, October 16, 2009

Publishing Advice for Graduate Students - revised

Readers may know of my "Publishing Advice for Graduate Students" (now downloaded over 4,300 times) which is a revised version of an earlier piece here (downloaded 2,855 times). Needless to say, I have been genuinely overwhelmed at the popularity of these papers. Over the past year or two, I have received much helpful feedback that I have used to revise the more recent paper.

You can find the newly revised "Publishing Advice for Graduate Students (Revised" here. The abstract remains:

"Graduate students often lack concrete advice on publishing. This essay is an attempt to fill this important gap. Advice is given on how to publish everything from book reviews to articles, replies to book chapters, and how to secure both edited book contracts and authored monograph contracts, along with plenty of helpful tips and advice on the publishing world (and how it works) along the way in what is meant to be a comprehensive, concrete guide to publishing that should be of tremendous value to graduate students working in any area of the humanities and social sciences."

Please feel free to distribute the new link widely --- this piece is available for free --- and please let me know how it might be further improved in future.

Thursday, October 15, 2009

Academy of Social Sciences announces new Academicians

. . . in today's Times Higher Education here. The full list:

John Allen, professor of economic geography, The Open University
Michael Anyadike-Danes, head of research, Economic Research Institute of Northern Ireland
Vernon Bogdanor, professor of government, University of Oxford
Martyn Bond, visiting professor of European politics and policy, Royal Holloway, University of London
Sophie Bowlby, senior lecturer in geography, University of Reading
Thom Brooks, reader in political and legal philosophy, Newcastle University
Jacquelin Burgess, professor of environmental risk, University of East Anglia
Tim Butler, professor of geography, King's College London
Timothy Clark, professor of organisational behaviour, Durham University
Alistair Cole, professor of European politics, Cardiff University
Diana Coole, professor of political and social theory, Birkbeck, University of London
Douglas Davies, professor in the study of religion, Durham University
Lorraine Dearden, professor of economics and social statistics, Institute of Education
John Dunn, emeritus professor of political theory, University of Cambridge
Lewis Elton, emeritus professor of higher education, University of Surrey
Anthony Forster, pro vice-chancellor for learning and teaching, Durham University
Keith Glaister, dean of the Management School, University of Sheffield
Keith Grint, professor of public leadership and management, University of Warwick
Alexander Haslam, professor of psychology, University of Exeter
Colin Hay, professor of political analysis, University of Sheffield
Robert Hetherington, chief economist, Devon County Council
Celia Hoyles, professor of mathematics education, Institute of Education
Janet Hunter, professor of economic history, London School of Economics
Chris Huxham, professor of management, University of Strathclyde
Ron Iphofen, retired director of postgraduate studies, Bangor University
Michael Keating, professor of politics, University of Aberdeen
Emil Kirchner, professor of European studies, University of Essex
Saville Kushner, professor of public evaluation, University of the West of England
John Leach, professor of science education and dean of education, University of Leeds
Kevin Lee, professor of economics and graduate dean, University of Leicester
Robert Leonardi, director-general of the Sicilian Regional Government's Brussels office
Peter Malpass, professor of housing and urban studies, University of the West of England
John Mingers, professor of operational research and information systems, University of Kent
Michael Moran, professor of government, University of Manchester
Elizabeth Murphy, professor of sociology and pro vice-chancellor, University of Leicester
Emma Murphy, professor of political economy, Durham University
David Nelken, distinguished research professor of law, Cardiff University
Frank Peck, professor of regional economic development, University of Cumbria
John Peterson, professor of international politics, University of Edinburgh
Judith Phillips, professor of gerontology, Swansea University
Bob Picciotto, visiting professor in the department of war studies, King's College London
Laurence Ray, professor of sociology, University of Kent
John Richardson, professor of student learning and assessment, The Open University
Anne Rogers, professor of the sociology of healthcare, University of Manchester
James Rollo, professor of European economic integration, University of Sussex
Thomas Scharf, professor of social gerontology, Keele University
Andrew Scott, professor of European Union studies, University of Edinburgh
Michael Shackleton, extraordinary professor of European institutions, Maastricht University
David Shanks, professor of psychology, University College London
Michael Sheppard, professor of social work, University of Plymouth
David Simon, professor in development geography, Royal Holloway, University of London
Iram Siraj-Blatchford, professor of early-childhood education, Institute of Education
Maria Slowey, vice-president for learning innovation, Dublin City University
Guy Standing, professor of social and economic security, University of Bath
John Stewart, professor of health history, Glasgow Caledonian University
Gerry Stoker, professor of politics and governance, University of Southampton
Michael Swan, freelance writer
Howard Thomas, dean and professor of strategic management, Warwick Business School
Claire Wallace, professor of sociology, University of Aberdeen
Paul Whiteley, professor of government, University of Essex
Allan Williams, emeritus professor in occupational psychology, City University London
David Wilson, deputy dean and professor of strategy at Warwick Business School
Fiona Wishlade, reader in the European Policies Research Centre, University of Strathclyde
Cecilia Wong, professor of spatial planning, University of Manchester.

Monday, October 12, 2009

200,000 hits . . . and counting!

Today, the Brooks Blog reaches a special milestone: the blog now has over 200,000 hits since June 15, 2006. I have really been amazed at how the blog has taken off and become so popular. It's largely been a collection of any number of things from job and conference announcements to noting important papers and a place to note my own work, as well as various bits of humour and music. I remain eternally grateful to Brian Leiter for his warm encouragement over many years to start blogging and, most especially, to my readers. Thank you all!

Tuesday, October 06, 2009

The Academy of Social Sciences and Me

Today, I have just received confirmation that I have been accepted into the Academy of Social Sciences. The announcement is here. The Academy's website notes here the following:

The Academy is delighted to announce the award of Academician status to 64 senior social scientists. The award is for making a significant contribution to the social sciences and is by nomination and peer group review. Speaking about the conferment, the Chair of Council, Professor Cary Cooper CBE AcSS, said:

“I am delighted that we are able to confer the award on such a significant number of distinguished new Academicians. The social sciences have a vital contribution to make towards some of the big issues facing society today, such as sustainability, crime, communities and individual fulfilment and well-being. Conferment of the award will give confidence that the contributions made by Academicians to knowledge production and transfer are well founded, because that are based on rigorous peer group review of the available evidence."

I was nominated by Political Studies Association. Those sharing the honour with me include Vernon Bogdanor CBE (Oxford), Diana Coole (Birkbeck), John Dunn (Cambridge), Colin Hay (Sheffield), Michael Moran (Manchester), Gerry Stoker (Southampton), and Paul Whiteley (Essex) amongst many others.

Saturday, October 03, 2009

Further evidence that Republicans -- the party of "no" -- care more for ideology than country

. . . with the recent evidence widespread delight at Chicago failing to host the 2016 Olympics, as reported here.

Thursday, October 01, 2009

Introducting the new UK Supreme Court!

Details here. The 11 judges were sworn in today. The story:

"[. . .] The Supreme Court, housed at Middlesex Guildhall, replaces the Law Lords as the last court of appeal in all matters other than criminal cases in Scotland.

The court is independent of Parliament and will hear the most important cases.

Lord Phillips, President of the Supreme Court, said the change in form was important for judicial openness.

The judicial function of the House of Lords, whose powers had evolved over centuries, ended with the swearing in of Lord Phillips.

The justices wore black robes threaded with gold, replacing the full-bottomed wigs, robes and breeches of the lords.

The £59m Supreme Court has opened six years after it was first announced. Its first members were - until last month - the Law Lords who would have otherwise heard the same cases in the House of Lords.

But the constitutional change that led to the Supreme Court's creation means that Parliament's lawmakers and the judges charged with overseeing legislation have been separated.

Scotland's supreme criminal court remains the High Court of Justiciary.

The swearing-in saw Lord Phillips of Worth Matravers become the first President of the Court.

He was joined by 10 other colleagues in taking an oath of allegiance to uphold the law. A final 12th member of the court will be appointed at a later date.

Lord Phillips said: "This is the last step in the separation of powers in this country. We have come to it fairly gently and gradually, but we have come to the point where the judges are completely separated from the legislature and executive.

"The change is one of transparency. It's going to be very much easier for the public to come to our hearings.

"I would hope that the court is still sitting in 100 years' time and that when people look back at this step that they see it as a very significant step in the constitution of this country."

While only the Law Lords' judgements were televised from Parliament, all of the Supreme Court's hearings will be open to the public.

Its building, in Westminster, includes a public cafe and education facilities. For the first time, television cameras have been fitted into the courtrooms meaning that many hearings will be available to broadcasters.

The Supreme Court sits for the first time later on Thursday to deal with a relatively minor issue relating to legal costs.

Its first major appeal hearing follows next week in a case concerning terrorist suspects whose assets have been frozen.

Although the actual business and workings of the justices will be essentially the same as those of the Law Lords, it will be watched closely to see if the move across Parliament Square will affect the way its decides cases.

One group of influential solicitors and barristers is launching a blog to monitor the Supreme Court's decision-making.

But others have criticised the change, arguing that it is largely a cosmetic exercise."