tag:blogger.com,1999:blog-1274716328603234407.post5482961251841983569..comments2022-03-31T20:47:23.431+01:00Comments on Psychology, computing and geekery: The problem with performance measures (e.g. REF and NSS)Anonymoushttp://www.blogger.com/profile/11823365707092158487noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-1274716328603234407.post-62761000511232520132012-05-30T14:55:50.675+01:002012-05-30T14:55:50.675+01:00Allowing people to talk about the impact their wor...Allowing people to talk about the impact their work has had above and beyond the paper citations is a good thing, as Dorothy points out above. And at least, in this instance, we are talking about impacts that have occurred, unlike in grant applications where we have to discuss impacts that might occur (which appears to be fairly close to science fiction writing).<br />Does it cause people to go for the quick study? I'm not sure. It might cause scientists to work on simple or sexy topics, rather than some important topics that are difficult for the general public to understand or engage with. But if it's just that a scientist can get credit for creating something that turned out to be useful as well as scientifically novel, then that's surely good.Anonymoushttps://www.blogger.com/profile/11823365707092158487noreply@blogger.comtag:blogger.com,1999:blog-1274716328603234407.post-19399702241912920712012-05-30T10:45:08.017+01:002012-05-30T10:45:08.017+01:00The construction of the REF somewhat reminds me of...The construction of the REF somewhat reminds me of the creation of the first intelligence tests. Inclusion of items on those tests was based on the principle that kids who were deemed intelligent by others should come out intelligent in the test; test construction and inclusion of items were atheoretical and largely included and excluded to fit the shape of the curve. <br />The REF seems quite similar to that. Including our perception that institutions we *think* are top should come out on top, and if they don't than the measurement is deemed not accurate. If we are to have a system that is probably not reliable, perhaps not valid, constructed on somewhat arbitrary grounds, mainly used to confirm hunches we have about institutions etc. then perhaps we could look for a process that at least takes less time and money.Fenjahttps://www.blogger.com/profile/04158425267768214788noreply@blogger.comtag:blogger.com,1999:blog-1274716328603234407.post-15597898669799143482012-05-30T10:24:59.006+01:002012-05-30T10:24:59.006+01:00Yes, I didn't distinguish between the differen...Yes, I didn't distinguish between the different measures. But on this issue they are all the same. /None/ tell you about their confidence intervals. We know a lot, statistically, about how to measure such things, even for unusual tools such as this. For example, Mori's political polls tell you the expected error in their measurements. I just believe REF could do the same. Or, at least, highlight that there /is/ measurement error.Anonymoushttps://www.blogger.com/profile/11823365707092158487noreply@blogger.comtag:blogger.com,1999:blog-1274716328603234407.post-69648541422184389872012-05-30T10:20:58.007+01:002012-05-30T10:20:58.007+01:00On behalf of Andrew Derrington (cruelly bounced by...On behalf of Andrew Derrington (cruelly bounced by blogger):<br /><br />I think it is easy to be imprecise when talking about the RAE and the REF. I think we should distinguish three different things.<br /><br /> - The RAE itself, which is a way of assessing the quality and quantity of research in UK Universities. RAE scores are based mostly on the quality of publications, and partially on assessments of the research environment and esteem. REF will discard the esteem component and introduce a component based on impact. The last RAE was in 2008 and cost £12M, it will be replaced by the REF in 2013.<br /> - League tables and rankings based on RAE results.<br /> - All of the activities in universities that prepare for and respond to, the RAE and now the REF. HEFCE commissioned a survey that estimated the cost of these as nearly £50million for the last RAE.<br /><br />I think that most arguments against the RAE and the REF are based on bad qualities of the league tables and on the apparently excessive and counterproductive preparations for theREF and responses to the RAE in some universities. <br /><br />I think that it would be a tragedy if we moved away from a system in which we assess university research mainly by the quality of its outputs.Anonymoushttps://www.blogger.com/profile/11823365707092158487noreply@blogger.comtag:blogger.com,1999:blog-1274716328603234407.post-40484604853166961552012-05-29T20:13:16.938+01:002012-05-29T20:13:16.938+01:00Though the ideas and considerations evoked by acti...Though the ideas and considerations evoked by actively thinking about impact can be useful (formation of explicit end-goals and research direction), the REF's impact section is as-yet poorly defined. It also encourages more concrete, shorter-term research as opposed to innovative work, which has been shown to result in demonstrably higher impact despite initially appearing as somewhat of a 'gamble' (Azoulay, Zivin, and Manso, 2011).Danihttps://www.blogger.com/profile/10128159415516788214noreply@blogger.comtag:blogger.com,1999:blog-1274716328603234407.post-27689224250463570152012-05-29T19:19:47.670+01:002012-05-29T19:19:47.670+01:00I think you make a good point here. I also think t...I think you make a good point here. I also think that concerns about reliability and validity of the measures is one reason why the assessments get ever more complex: 'impact' is now included because it's realised that importance of work can't be measured by publications alone. People want the system to be fair and so they try harder and harder to improve breadth and precision of the measures, but they're doomed because you can't reduce a complex multifactorial system to a number. And at the end of the day, it's down to the subjective judgement of committees. Andrew D points out we need some system to distribute funds and asks what alternative system would be better. I have to say I hanker after the old pre-RAE system, but it's true that also contained arbitrary decisions and was less transparent than REF, where the rules are at least explicit. But it was highly efficient - and perhaps no less unfair than the REF. <br />I don't think there were that many surprises in the last round, but agree they sometimes happen and when there ARE discrepancies between outcomes and people's expectations, then most people conclude that it's likely to reflect better gamesmanship, not better scholarship, in the institutions that get surprisingly good results.deevybeehttps://www.blogger.com/profile/15118040887173718391noreply@blogger.com