We are presenting an interview with one of the world's leading experts in the field of university rankings Dr. Richard Holmes. Professor Holmes is an author and editor of a special blog dedicated to the topic of rankings and world trends in the development of higher education. Interview by Round Univesity Ranking editor - Oleg Solovyev.
Part I. University rankings: what’s next?
What is your opinion on further development of university rankings?
How university rankings are going to look like in the nearest future? This is one of the most discussed and acute topic in university rankings sphere.
There are approximately three approaches in this domain:
1) Rankings will develop in framework of evaluation systems like U-Map, U-Multirank. E.g. in the future rankings will become and mostly serve as comparison platforms for decision making for students, faculty and policy makers.
2) League tables will retain their position as a dominant approach to measure HEI performance.
3) Rankings will go further than just serving as evaluation platforms. They will become complete search analytical systems with artificial intelligence elements that would help applicants to choose appropriate courses among millions and would also help academics to find most appropriate positions.
In your opinion, which way of development is the most promising? Or is there any other alternative way of development? Furthermore shall we see any merge between the three approaches?
Answer:
I think that there is a demand or a potential demand for all three approaches. There will almost certainly be more and better programmes like U-multirank, which may begin to start evolving into something more complex.
We are already seeing signs of these approaches merging. QS and THE are introducing interactive elements into their rankings. This could be frightening if one or two organizations acquire complete control over everything connected with comparative education data.
League tables: why so important?
Since 2003 almost all university rankings follow the standards identified by the Shanghai Rankings. This is a so-called “League tables” approach.
In general, is it correct to merge the data of different origins in one composite score? As ranking providers we publish our rankings based on the “league tables” approach as major part of society is used to see rankings as composite tables. Moreover, it is quite hard to explain to the community that there are many approaches in measuring university’s performance and everybody can choose its own set of metrics to generate a unique personalized ranking.
However, there is a “golden standard” - league tables. Can we say that the "league tables" approach lead the society and key education stakeholders to a false understanding of excellence in higher education? In other words, if the first raking appeared as a multidimensional platform, how could rankings look like nowadays?
Answer:
If rankings are focused on a single concept, say research excellence, and this can be demonstrated, for example with a correlation matrix, then the league table approach is not a bad idea. It is clear that the Shanghai rankings measure research primarily in the natural sciences and this is supported by correlational analysis that shows a close but not identical association.
It could be argued that excellence in different activities is likely to spread to others. A reputation for research will attract talented graduate students who will boost ranking scores which will attract able undergraduates and so on. If the various indicators in rankings can be shown to be related to one another there is nothing wrong in principle with amalgamating them into a single score.
There is of course a problem with the weighting that is assigned to the indicators. So far, it seems that this is decided by the wishes of data providers or the convenience of rankers.
Who is the primary target audience of the rankings: applicants/parents and students, academics, policy makers?
Answer:
QS is pretty clearly aiming at students, in particular internationally mobile graduate students. THE has a close relationship with “elite” universities in the Russell Group and LERU. The Shanghai rankings command respect among academics for their consistency even though their methods may be rather unsophisticated.
Do rankings actually serve as navigators for students and academics, as ranking providers position themselves? What is the real sense for student to compare university X in 172-th position and university Y in 196-th position? There is an opinion that rankings are more likely to be a political instrument - an evaluation instrument for the government but not for applicant’s parents, academics. Do you agree with this position?
Answer:
Not much although there might be some sense in comparing 2nd and 24th. But you should always look at the indicators to see exactly what is contributing to the overall score. A lot of skepticism is in order if a high score results from one indicator with a disproportionate weighting.
Part II. Round University Ranking: a new stage in ranking development?
RUR: league tables ranking or multidimensional platform?
This is mostly about high-level approaches rather than details on how to improve one or another particular indicator or to replace some indicators totally. Currently RUR is closer to the traditional league tables rather than to multidimensional systems. RUR includes 30 different rankings on one platform, from 5 to 20 indicators in each of these tables. We also show expanded university profiles and offer a variety of different online services, which allows users to compare intuitions. However, RUR still sticks to the league tables approach.
Since the first RUR edition published in 2013, one of our major goals was to merge the league tables approach and multidimensional approaches in development of our rankings. In your opinion, how well have we succeed in this effort?
Answer:
As a league table, RUR was an improvement in that it included more indicators, although some such as the reputation indicators did not add any new information, and did not hide them in groups of indicators.
As a multidimensional comparison tool, there are some technical issues that need to be dealt with – e.g. how to compare more than six universities at a time – but it is promising.
Recommendations in further development of Round University Rankings.
What is your professional advice for the RUR development? Should we transform the ranking into multi-dimensional system or limit our efforts by producing more and more different league tables (like some other rankings do)? On one hand, the “league tables” are considered an “old school, old fashion” approach to evaluation the performance of higher education intuitions. On the other hand, the “league tables” is what the society expects to see i.e. a simple evaluation instrument. Where lies the balance between scientific correctness in measuring quality of higher education and simplicity of presentation forms?
Answer:
I think there is a definite need for rankings that include several non-research indicators and are transparent and reproducible. A multidimensional system could develop gradually out of a set of international ranking.
Ranking of rankings?
If you could rank rankings, where would you place RUR? It’s a question of rankings clustering and classifications rather than “Ranking X > Ranking Y > Ranking Z”. If one could classify all the rankings in some groups and indicate, for example “three zones” (green, yellow, red), in which group and zone RUR would be?
In other words, how RUR meets IREG Ekaterinburg recommendations, which you proposed in Ekaterinburg’s conference of the Russian 5-100 project?
Could you please state 2-3 key strengths and weaknesses of RUR?
Answer:
Answer to both 6 and 7.
RUR could be categorized as a holistic ranking along with QS and THE but not Shanghai in that it includes indicators of things other than research.
It is inclusive in that it has about the same number of institutions as QS and THE, more than Shanghai but fewer than URAP and Webometrics.
It is probably rather unstable with universities rising and falling a lot. This may be because of the inclusion of financial and reputational data.
RUR would score low on for reproducibility – it is easy to check the awards indicator in the Shanghai rankings but how to check field normalized citations using TR data?
Comprehensive – RUR scores fairly high here It includes data on finance as THE does and reputation as THE and QS do. It does not have a measure of student or graduate quality – QS is the only ranking to do this although not very well.
Suggestions for the future for RUR
For all rankings, the most significant challenge is to find measures of student/graduate quality and teaching quality. This might be done through surveys or through the development of instruments like AHELO. In addition, it might become feasible to tap into the developing social media like LinkedIn.
There are some indicators that measure inputs such as income and teaching resources. It might be interesting to have separate rankings for inputs and outputs such as publications or graduates. Some indicators correlate so closely that they could be removed or at least have their weighting reduced.