League tables and connecting slides: objective ways of assessing knowledge exchange strategies

League tables and connecting slides: objective ways of assessing knowledge exchange strategies

As we move from the initial reactions to the REF 2014 results to analysis of its impact we should start praising benefits of the REF other than league tables. One of these ought to be demonstrating like never before the variety of ways in which academic work makes life easier or better for others, but acknowledging that there was a lot done before and there will be more done beyond. The flow of knowledge from academia to elsewhere (and “elsewhere” need not be as defined by REF) is a process, not an end, and it happens back and forth as a by-product of science and research activities.

"Smaller universities with broad KE portfolios have been more likely to grow their KE income sustainably than the larger universities."

Beyond breadth, the standardization of the case studies should also help demonstrate the difficulty of controlling this by-product of academic work. In some cases the process of knowledge flow will be replicable, but in others it will not. And rightly so, if we could fully map the ways in which all new knowledge will flow to its uses we would have destroyed the essence of innovation - namely, uncertainty.

And yet, uncertainty poses a problem for those who have to allocate resources to their “best” uses, this being the role of public and private managers of activities that have uncertain outcomes, such as those performed with and in academic institutions.

To evaluate the outcomes of uncertain activities we have to allow for a certain degree of subjectivity, be that expert opinion, peer-review or out-of-sample prediction. Some public funders have sought to reduce subjectivity by allocating funds using formulas sourcing parameter values from peer review and expert opinion, but increasingly these are being complemented with less subjective indicators of performance. However, and because of the same uncertainty, objective assessments of research performance are always partial and should not fully substitute judgement in the allocations; otherwise, we risk funding only the certain areas of research that are observable, thereby limiting the scope for unforeseen discovery.

Even using objective assessments alone does not guarantee that different assessments will obtain the same allocations, as illustrated by the latest NCUB research report published today.

Seeking a better understanding of how universities make decisions on knowledge exchange (KE), NCUB teamed up with HEFCE using the Higher Education Business and Community Interactions survey (HEBCI) as an objective source of information.

Among the many indicators the survey captures, we focused on KE income, in particular, how much Higher Education Institutions are earning from consultancy and contract research activities, career development programmes and use of facilities and equipment by non-academic users. This is an objective performance indicator as it reflects the monetary value of these activities and services for their users, and HEFCE have used KE income as an objective anchor in their allocations.

"it is better to have a responsive system of knowledge where resources can flow where they are needed"

However, the distribution of KE income across HEIs in the UK is highly skewed, with most income concentrated in the hands of a few, and persistent, showing limited income mobility over time. As a result, performance evaluations based on the actual distribution will inevitably reflect the inequality and the stickiness, and allocations made on the basis of these evaluations will contribute to perpetuating the situation.

The NCUB report picks on this particular side effect of an objective allocation and illustrates how the same data can be used to defend the role of breadth, as opposed to concentration, in KE activities.

Instead of assessing performance using the total amount of income drawn, we use the growth of income over the 6 years from 2007 to 2013. Because scale and growth are unrelated, using growth as a performance indicator shines a new light on some of the smaller universities, whose efforts to engage with business and the community are commensurate to their size but dwarfed in absolute volume by the larger players.

The report shows that the breadth of KE activity has been a more important driver of sustained growth of KE income than scale, that is, the smaller universities with broad KE portfolios have been more likely to grow their KE income sustainably than the larger universities. The larger HEIs have access to more varied resources, and this is what gives them an advantage in totals, but the best performing smaller HEIs have been catching up with their direct competitors by growing income across a wider range of activities.  

Why does this matter? Because when it comes to knowledge flows, it is not large but broad that makes a difference in the long run. If the only goal is more money, then concentrating resources where the demand is brings the most cash in the short term. But this strategy does not guarantee that demand will persist over time. Increasing income from a range of activities means you are more likely to see growth year on year than big jumps in one area.

The report argues that it is better to have a responsive system of knowledge where resources can flow where they are needed, bearing in mind that needs change in uncertain ways.

Knowledge flows have to be managed as a multi-dimensional system of connecting slides not as a scale-driven flat league table.

Dr Rosa Fernandez is Director of Research at NCUB.

The NCUB Research Report Strategies for Sustaining Growth of Income from Knowledge Exchange across Higher Education Institutions (HEIs) in the UK is available to download now.

Have an opinion? Comment below, tweet us @NCUBtweets or submit your own blog

Other posts that may interest you:

Can the finance sector put a price on researchers?

Reforming Employability Measures

Value of university-business interactions rises

Expand for more