Round University Ranking Methodology

Round University Ranking (RUR) is an international ranking of leading world universities published by the RUR Ranking Agency (Moscow, Russia). The ranking provides comparison of 930 leading universities from 80 countries around the world for 8 years (2010-2017) according to 20 indicators distributed into 4 areas: teaching, research, international diversity, and financial sustainability. Such a wide coverage, both geographically and temporally, makes RUR ranking a unique tool for choosing universities for study and work, as well as comparing higher education institutions on a global scale.


RUR ranking is:

•  a navigator in the world of higher education enabling applicants and their parents to choose the appropriate higher institution and the way of training;

•  a tool for students helping to choose a higher education institutions for their short-term programs or change their main place of study;

•  a tool for teachers and professors enabling them to find suitable vacancies and make decisions about cooperation with their colleagues in the other universities;

•  an assessment tool for the management of universities evaluating the competitiveness of a university on the national and global scale and making the appropriate management decisions aimed at comprehensive improvement of the university’s international competitiveness;

•  one of the assessment tools for business evaluating the university and making decisions about cooperation with a particular higher education institution;

•  a tool for the state to comprehensively assess the country's higher education system as well as a way of maintaining national prestige.


BASIC PRINCIPLES IN METHODOLOGY DEVELOPMENT:


The following basic questions were formulated before the methodology development:

•  What basic principles should be used as a basis for the methodology?

•  Should the ranking be presented in the format of a traditional system of leagues or a system of comparison/correlation of the universities?

•  What data should the ranking be based on?


1.   RUR rankings basic principles


A)   Inclusiveness


•  There are no restrictions for any higher educational institution in the world to limit their participation in the RUR ranking such as the number of publications, the availability of undergraduate programs, etc. The only exception effective as on 2016 is a number of restrictions concerning the initial indicators of universities: the number of academic staff, students, publications, etc. If a university does not reach these indicators, it is assigned the average value of the universities of the host country then. These restrictions are implemented to remove a number of anomalies from the ranking, which will be discussed below. It is worth noting that a university can participate in the ranking in any case, even though it does not overcome the threshold values.


B)   Balanced Methodology


•  Initially, we proceeded from the concept that weights should be the same for all indicators of the ranking, i.e. 5% each. As testing made it clear, such approach led to anomalies in the ranking results (for more details refer to International Diversity chapter). Thus, we found a compromise solution saying that the weights inside the measuring directions (groups of indicators) should be the same, so that no indicator had an advantage over the others. In so doing, the presence of a large number of indicators and small weights should minimize the volatility of the ranking.


C)   Stable Methodologyy


•  Methodology of ranking, i.e. the number of indicators, the distribution of weights between them, and the initial data source should undergo minimal changes. Eight versions of the 2010-2017 RUR rankings had the changes but they were related to data processing methods and the elimination of the found anomalies rather than changes in the methodology basics.


2.  Ranking type: a league table or a comparison system


Traditionally, the rankings of the universities are represented by the so-called «League tables». Within the framework of such type of rankings the data for individual metrics are reduced to a single score. This is a «classical» ranking, in which universities are evaluated by a number of parameters and are ranked according to the decrease of the final score. The final score is an aggregation (i.e. compilation) of scores for a number of indicators.


There is an alternative to league tables, i.e. the systems of comparison and correlation, which provide the clustering of higher education institutions according to similar characteristics instead of ranking. Thus, the ranking is no longer a ranking in the strict sense of the word. It rather becomes a set of indicators, which customers use to compare universities in accordance with their goals and objectives.


Finally, it was decided to stick with the first option - a league table. First off, most of the rankings were built precisely on this principle, which makes this format familiar and understandable to users. Secondly, the system of user comparison in the strict sense is not a ranking because in this case each user has his/her own ranking. Thirdly, the team of the RUR Ranking Agency counts to combine both approaches in the long term so that on the one hand, there will be the ranking as a league table, on the other, there will be introduced elements of comparison and correlation.


3.  Ranking Data Sources


The procedure of choosing the data types used for the assessment of universities has become the key factor determining the methodology structure.


The main types of information to create a ranking are:


•  Bibliometric data, which are based on information obtained from bibliometric systems such as Web of Science Core Collection, Scopus, Google Scholar, etc. As a rule, the term «bibliometric» means the number of publications in scientific journals, the number of citations for these publications and all derivatives of those indicators. Scientometric data refer to the type of data most widely used in university rankings.


•  Statistics, which include the information provided by the universities, i.e. staff indicators (number of academic staff, researchers, etc.), data on the student population (total number of students, number of various diplomas/degrees awarded, etc.), financial data (total budget, R&D budget, etc.).


•  Reputational data, which are collected from specially conducted reputational surveys showing which universities respondents choose considering them the best in terms of the quality of teaching, research, etc.



After a careful evaluation we have chosen Institutional Profiles by Clarivate Analytics, the former Thomson Reuters IP & Science Solutions.


Institutional Profiles are an annually updated database containing all three abovementioned types of data. There are more than a hundred of unique indicators for each higher education institution within this database so it allows choosing appropriate indicators for any area of university activity.


The next task after selecting the data source is to define the methodology model, which is the basis for defining specific indicators and distributing the weights between them. This task is divided into three sub-tasks:


•  Selection of measuring directions (groups of indicators);


•  Definition of the minimum and maximum limits of indicators for both the entire ranking and certain groups of indicators;


•  Determination of the weights of the groups and indicators.



Selecting groups of measurement


The modern university is an extremely complex phenomenon and can include several campuses, hundreds of structural divisions, thousands and tens of thousands of employees, tens and hundreds of thousands of students. Universities are becoming the basis of scientific and technological clusters, driving the growth of cities and regions. The mission of the university has long gone beyond the limits of teaching or presenting the knowledge by means of its translation, as it was in the Middle Ages.


Therefore, a natural question arises: how can we sort out and choose the key aspects of the university's activity among the variety of the others and reflect them in the ranking methodology? It is worth taking into account that there is no ranking to be able to cover all aspects of the university's activities since each university has its own unique features. Not to mention that some areas of activity are more amenable to quantitative accounting (scientific activity), the others are less (educational activities).


After a careful analysis of all possible areas of activity, we chose two main and two auxiliary directions of the university's assessment. The main directions constitute the meaning of the institution's activity and its basis. The auxiliary ones are supposed to measure additional, background aspects of activities.


The main activities are:

•  Teaching;

•  Research.


The auxiliary activities are:

•  International Diversity;

•  Financial Sustainability.



Determination of the number of indicators


Based on the availability of the Institutional Profiles database, it was decided to choose 5 indicators for each of the 4 measured groups.


Determination of weights


Taking into account there are no scientifically based and universally accepted criteria for selecting weights of indicators, it was decided to assign equal weights within the indicator groups. Hence «Round», the name of the ranking, comes, which denotes the symmetry of the methodology.


«International Diversity» and «Financial Sustainability» groups were assigned 10% of weights. Weight of each criterion was 2% because there were 5 evaluation parameters.


Thus, 20% were distributed between two areas. The rest of 80% left for «Teaching» and «Research» groups.


We consider both «Teaching» and «Research» as two primary missions of every university. Therefore, the remaining 80% were equally divided: 40% for each of the two groups, and 8% for each of 10 indicators within these groups.


Summarizing the above we came to the conclusion that the RUR ranking had the following structure:

•  20 indicators;

•  4 measured groups;

•  5 indicators in each measured group;

•  Equal weights in the measured groups.


Description of measured groups and indicators of the ranking


I.   Teaching (40%)


Teaching is one of the key missions of the higher education institution and at the same time it is the hardest activity to measure. For example, what is the correct way of evaluating professor’s talent as a lecturer? Therefore, the only way to assess the quality of teaching in the global ranking is to use such formalized indicators as the number of academic staff divided by the number of students, etc. After careful selection we have chosen the following five indicators to be included in the Teaching area:


1.   Academic staff per students (8%)


This indicator shows how many faculty there are per student in a university. This is a standard indicator used in world university rankings due to objective reasons. As already mentioned, measuring the quality of education is one of the most difficult tasks in ranking universities. Using this indicator, we proceed from the assumption that the more lecturers there are per one student, the smaller the size of the student group is, and therefore the more attention a lecturer can pay to each student. This implies the conclusion that the quality of teaching is better in a group where there are 5 students per a lecturer than in another group, in which there are 25 students per a lecturer.


The initial values of the number of academic staff and students are calculated as reduced to the full time equivalent (for academic staff) or to the standard workload of the full-time student. Therefore, the number of academic staff and students in RUR University Profiles may not correspond to similar values («headcount») found on the other resources.


2.   Academic staff per bachelor degrees awarded (8%)


This indicator specifies the number of academic staff per undergraduate degrees awarded in a given year. In fact, this indicator is a special case of «Academic staff per students» ratio and it measures undergraduate level programs.


The reason for allocating this indicator in a separate metric is that the level of baccalaureate is the basis of higher education in the world.


3.   Doctoral degrees awarded per academic staff (8%)


This indicator shows the number of PhD degrees awarded or its equivalent per academic staff. In other words, the indicator shows how many successfully defended dissertations are per a faculty on average. The more dissertations per academic staff are defended in the university in question, the higher level of teaching we can expect at the third stage of higher education, i.e. PhD.


4.  Doctoral degrees awarded per bachelor degrees awarded (8%)


This indicator shows the ratio of PhD degrees awarded to bachelor degrees awarded. This ratio reflects the proportion of the PhD programs compared to the programs of the 1st level, i.e. bachelor programs. The higher the proportion of PhD graduates is, the more the university is accordingly focused on the most complex and expensive level of training. A high percentage of PhD students in the total number of students shows that the university has a sufficient number of academic staff and researchers capable of preparing the PhDs, as well as a sufficient amount of equipment and resources.


5.  World teaching reputation (8%)


The international reputation of the university in the field of teaching shows the degree of university’s fame in the international academic community. The initial data on reputation represent the distribution of respondents' votes among universities. A special survey - Academic Reputation Survey - is conducted annually among representatives of the academic community to collect the reputational data. Participation in this survey is possible only by invitation. In other words, it is the survey operator who determines the initial sample of potential respondents who will receive an invitation letter to participate in the survey.


As part of the survey, a respondent is asked to select up to 15 universities, which he/she considers the most powerful in teaching. Annually about 10 thousand representatives of the international academic community participate in this survey around the world.


II.   Research (40%)


The quality of the research conducted at the university is the second important group considered in the RUR ranking. Therefore, «Research» has as much weight in the overall ranking of RUR as «Teaching». Most of «Research» metrics are scientometric and are based on data from Web of Science Core Collection’s bibliometric system.


6.  Citations per academic and research staff (8%)


This indicator helps to evaluate the intensity of the research conducted at the university. For this purpose, the number of citations of all University’s scientific publication authors for a certain period of time (different for the 2010-2015 and 2016-2017 rankings) is divided by the number of academic staff and researchers who worked at the university for a certain year. When counting the number of publications, there are taken into account only «Article» and «Notes» indexed by Web of Science Core Collection’s bibliometric system.


«Scientists» are understood to be only those employees of the university who are engaged exclusively in research and development without being involved in teaching.


The version of indicator in RUR 2010-2015


The RUR 2010-2015 rankings counted only one-year publications, and the citation periods for the same publications were two years.


For example, the RUR 2015 ranking used data on the number of university publications for 2012 and citations for the same publications for 2012-2013. The number of teachers and researchers was counted for 2012.


The choice of such a short time interval was determined by the need to give the ranking a greater dynamism due to a shorter period of counting bibliometric information. In 2016, it was decided to abandon the annual periods of counting publications and biennial ones for citations since the volatility of indicator scores was significantly higher than the periods when longer periods of counting bibliometric information were used.


The version of indicator in RUR 2016


The 2016’s version has 5-year periods of accounting for publications and 6-year periods for citations.


2016 ranking has:

•  Period of accounting for publications: 2009-2013’

•  Period of accounting for citations: 2009-2014’

•  Year of accounting for academic staff and researchers: 2013.


7.  Doctoral degrees per admitted PhD (8%)


This indicator assesses the quality of the student preparation for the PhD level. For this purpose, the number of dissertations defended at a given university during the year under review is divided by the number of PhD programs and their analogues accepted for the first year. Thus, the average percentage of students defended PhD degrees is determined relative to those taken for the first year of education for educational programs at this level. A higher percentage, therefore, shows the organizational level of PhD program and motivation of the students. In addition, a high percentage of defended dissertations shows how a work system between academic staff and PhD students is built at the university. Conversely, the lower the defended/enrolled ratio is, the worse the motivation of students and the conditions of work at the PhD level are.


The version of the indicator in the subject rankings of 2016-2017


RUR’s 7th indicator in the subject rankings is stated as: «The share of post-bachelor students in the total number of students».


This indicator measures the proportion of students enrolled in the first year of the master’s program, PhD (and equivalent programs) out of the total number of students in this higher education institution. Since the master's programs and the PhD’s in particular are predominantly research programs, then there is assessed university’s contribution into the science of the future. The higher the percentage of students at the post-bachelor level is, the more significant contribution to world science this university can make. Also, this indicator helps evaluate the human resources of the university. Accordingly, the higher the percentage of masters and graduate students is in the university, the more highly qualified scientific cadres the university should possess.


The change in the wording of the 7th indicator is due to the fact that since 2015 Clarivate Analytics has not collected data on the number of students admitted to the 1-st year of PhD level programs at the subject area level.


8.  Normalized citation impact (8%)


Normalized Citation Impact (NCI) shows the ratio of average citation of university publications per average citation in the world, type of publication and subject area for a similar time interval. For example, «the value of NCI = 1» means that the university's scientific publications are cited in the same way as at the world average (100%). The value of 2.15 will show that the authors of the university are quoted as 215% of the world average, etc


Differences between NCI for different years of the RUR ranking:


The version of indicator in RUR 2010-2015


The 1-year periods of accounting for publications and 2-year periods of citations were used to calculate NCI in 2010-2015 versions.


Such short periods were associated with the RUR analytical team’s original concept claiming that short periods would provide a certain dynamism to universities. On the one hand, it really happened. On the other hand, in a number of cases just a few publications could have an unjustifiably high influence on the final position of the university in the ranking. In this regard, it was decided to increase the counting intervals for both publications and citations in the following versions of the ranking.


The version of indicator in RUR 2016


2016 version used the NCI sum for 5 periods: 5-year periods for accounting for publications and 6-year periods for citations according to the following scheme:


NCI Publications Citations
2013 2009 - 2013 2009 - 2014
2012 2008 - 2012 2008 - 2013
2011 2007 - 2011 2007 - 2012
2010 2006 - 2010 2006 - 2011
2009 2005 - 2009 2005 - 2010

The version of indicator in RUR 2017


For the calculation of the NCI indicator, publications for a 5-year time interval and 6-year period of citations of the same publications are calculated.


This scheme of calculating NCI seems to be the most stable. On the one hand, the counting periods are quite long, on the other hand, there is no artificial summation of NCI for several years, as it was in the edition of 2016 ranking.


9.  Papers per academic and research staff (8%)


This indicator shows the level of scientific productivity of the university. For this purpose, the number of scientific publications is divided by the number of academic staff and researchers working at the university during the last year when the publications were rated. There are only «Article» and «Note» publications considered.


Changes of the Indicator:


The version of indicator in RUR 2010-2015


The period of accounting for publications is 1 year.

•  For example: publications of 2016 were rated in the ranking of 2015.

•  The period of accounting for citations is 2 years.

•  For example: citations of 2012-2013 were taken into account in the ranking of 2015.



The version of indicator in RUR 2016-2017


The period of accounting for publications is 5 years.

•  For example: publications of 2010-2014 are rated in the ranking of 2017.

•  The period of accounting for citations is 6 years.

•  For example: the citations of 2010-2015 are rated in the ranking of 2017.



10.  World research reputation (8%)


The university's reputation in Research shows the degree of influence that the studies conducted at the university have on the international academic community. In its original form, the reputation is the number of votes given for the university.


The reputational survey called Academic Reputation Survey is conducted by Clarivate Analytics. On average, 10,000 respondents participate in this survey every year, leaving 60,000 votes for universities participating in the RUR ranking. The initial sample of respondents is formed on the basis of authors contacts indexed in the Web of Science Core Collection logs. Participation in the survey is possible only by invitation. In other words, any third parties, including universities themselves, cannot send Clarivate Analytics a list of participants recommended for the academic survey.


In the survey each respondent:

•  Selects one or more narrow subject categories, according to the classification scheme of Web of Science Core Collection’s scientific categories;

•  Selects up to 15 higher education institutions, which he/she considers as leading in terms of the level of research conducted on the subject categories that the respondent indicated at the beginning of the survey.



The number of votes on the enlarged subject areas is used to compile the subject rankings of RUR. The total number of votes given for universities in all subject areas is used in drawing up the overall ranking, i.e. RUR World University Rankings.


The table below shows the periods of accounting for publications, and citations in rankings of different years.


III.    International Diversity (10%)


Internationalization of a university shows the degree of its involvement in the global educational and scientific process. This measurement gives 10% of weight against 40% of Teaching and Research.


The relatively low share of internationalization in the overall assessment of the university is due to the fact that, as a rule, universities are limited in their ability to influence the level of internationalization since this factor depends more on the geographical location of the university, on the overall standard of living of the country, in which the university is located, and on the policy pursued by the state in internationalization of higher education.


11.  Share of international academic staff (2%)


This indicator shows the level of academic staff internationalization at the university. To this end, the number of foreign professors and teachers (Faculty) is divided by the total number of academic staff working at the university. Faculty is considered to be foreign, if their citizenship differs from the country where the university is located. In the case of dual citizenship, the priority is given to the citizenship of the home country. For example, if a join citizenship lecturer works at a university, and one of the citizenships coincides with the country where the university is located, and is native, then such a person will not be considered a foreigner.


The number of faculty is counted as the Full Time Equivalent of this institution.


The high level of internationalization of the university characterizes the work conditions of university academic staff and indirectly affects the conditions of students' education. The higher the university internationalization level is, the more attractive conditions for work are created in this institution, and the more diversity of views in the course of educational and scientific work can be expected..


12.  Share of international students (2%)


This indicator shows the share of students of foreign origin, students in the university on programs of 3 levels (bachelor's, master's, postgraduate). Using this indicator, the attractiveness and competitiveness of the university for students from all over the world are assessed.


A student is considered foreign, if his/her citizenship does not coincide with the country where the institution is located. In the case of dual citizenship, priority is given to the country of origin. In other words, if a student studies in Country X and comes from Country X, but also has Country Y’s citizenship, such a student will not be considered foreign to any institution in Country X. And, conversely, he will be treated as a foreigner in country Y.


13.  Share of international co-authored papers (2%)


The indicator shows the proportion of publications with at least one foreign co-author in the total number of publications of the university. A foreign co-author is one of the authors of the publication who works in a country different from the country where the institution in question is located.


This indicator shows the scale of the university's external relations, the degree of its involvement in the international academic community. At the same time, this indicator largely depends on the region and institution location country


The change of Indicator:


The version of indicator in RUR 2010-2015


The period of accounting for publications is 1 year.

•  Example: publications of 2016 were rated in the ranking of 2015.


The period of accounting for the citations is 2 years.

•  Example: citations of 2012-2013 were taken into account in the ranking of 2015


The version of indicator in RUR 2016-2017


The period of accounting for publications is 5 years.

•  For example: publications of 2010-2014 are rated in the ranking of 2017.


The period of accounting for citations is 6 years.

•  For example: the citations of 2010-2015 are rated in the ranking of 2017.


14.  Reputation outside region (2%)


This indicator shows the reputation of the institution outside the geographical region of the university's location. The list of regions is as follows:

•  Asia;

•  Europe;

•  North America;

•  Oceania;

•  South America.


For example, if University X is located in Asia, then the voices given by out-of-Asia participants in the reputational poll for University X will be taken into account, i.e. the voices from the remaining five regions.


This indicator is a special case from the global reputation, which takes into account the university's votes from around the world, including the region where the institution is located.


The reputation of the university outside the region is highlighted in a separate indicator because the reputation obtained from «one's own» region can show a significant percentage of votes from the total number of votes given for the university.


The change of Indicator:


The version of indicator in RUR 2010-2015


Only reputation in Teaching outside the region is taken into account.


The version of indicator in RUR 2016-2017


The average value (the number of respondents' votes) of reputation in both Teaching and Research outside the region is taken into account.


15.  International level (2%)


This indicator shows the overall level of institution internationalization and the average for indicators 11-14.


The version of indicator in RUR 2010-2015


International bachelors (2%)


The indicator with this wording is relevant for the 2010-2015 rankings. By means of this criterion, the proportion of foreign students who are enrolled in the first year of bachelor-level programs is determined.


This indicator was singled out in a separate metric despite the seemingly similar resemblance to the «foreign students» indicator because higher education at the bachelor's level has the largest share among all levels of education.


The version of indicator in RUR 2016-2017


International level


In the 2016 ranking the previous indicator was replaced by the current version, i.e. the average score of indicators 11-14 due to Clarivate Analyticss stopped collecting information on the number of foreign students admitted to undergraduate programs and our partner in data collection and processing.


IV.    Financial Sustainability (10%)


High financial capabilities of the university are one of the indispensable conditions for quality education and research. At the same time, the financial indicators of the university largely depend on the overall economic level of the country where the university is located. That is why, despite the objective significance of financial indicators for the development of the university, this indicator has 10% of the final assessment of the university, along with «International Diversity».


When calculating information on all financial indicators, both public and private investments in the university are taken into account.


16.  Institutional income per academic staff (2%)


This indicator shows institutional income per academic staff ratio (Faculty). When calculating the total budget, all incomes that the university received from all types of sources (both state and non-state) for a fiscal or calendar year are taken into account.


As follows from the description, this indicator is normalized, i.e. it is a ratio of total budget to the faculty is taken into account rather than overall budget of the institution. This equates the opportunities of both large and small universities.


17.  Institutional income per students (2%)


This indicator calculates the level of university funding per student.


18.  Papers per research income (2%)


This indicator shows the number of publications concerning the university budget allocated for research. In other words, the average cost of a publication is measured, which, in turn, shows the financing level of the university's scientific activity per one unit of the final product – a publication in a prestigious academic publication.


The changes of the indicator:


The version of indicator in RUR 2010-2015


The period of accounting for publications is 1 year.

•  Example: publications of 2012 were rated in the ranking of 2015.


The version of indicator in RUR 2016-2017


The period of accounting for publications is 5 years.

•  Example: publications of 2010-2014 are rated in the ranking of 2017.


19.  Research income per academic and research staff (2%)


This indicator evaluates the relative funding of the scientific activity of the university. For this purpose, the total amount of the research budget is divided by the number of academic staff and researchers in the university (academic staff).


We do not use «the number of researchers» as the only one (i.e. the university staff who are involved only in research activities) because the understanding of «a researcher» differs in the countries all over the world. Thus, the denominator uses the number of both academic staff and researchers staff.


20.  Research income per institutional income (2%)


The share of the research budget in the general budget of the university shows the degree of university orientation for carrying out advanced research. That, in turn, also affects the level of teaching at the university. The higher the share of investment in research is, the higher the level of the educational process is expected at the university due to getting students and the academic community involved in research. Thus, with help of this metric, the level of both teaching in the university and research are assessed.


The stages of Round University Ranking calculation


The seemingly clear and simple values of the ranking – points and ranks of universities – are based on a long and elaborated multi-stage data processing. All the stages of data processing in the RUR ranking are listed down below.


Raw ranking data


The initial data are absolute values, which are grouped into three categories (for details on each category refer to the beginning of this review about the RUR ranking methodology). In total, there are 19 initial parameters used to calculate 20 ranking indicators.


1)  Statistical data


Statistical data are provided by universities either at the end of the calendar year (December) or at the beginning of the academic year.


Subcategory 1. Human resources:

•  Number of academic staff (faculty);

•  Number of researchers - i.e. university employees involved only in research activities without lecturing

•  Number of international academic staff.


Sub-category 2. Student Contingent:

•  Number of students;

•  Number of international students;

•  Number of undergraduate program graduates;

•  Number of master's program graduates;

•  The number of defended PhD degrees or their equivalent (including master's thesis and doctoral thesis);

•  The number of admitted PhD programs or their equivalent.


Sub-category 3. Financial indicators:

•  Total university budget;

•  University research budget.


2)  Bibliometric data:

•  Number of publications;

•  Number of international co-authored papers;

•  Number of citations;

•  Normalized citations;


3) Reputational data


The initial information on the four indicators listed below shows the proportion of the total number of respondents who voted for the university in the reputational survey. The list of initial reputation parameters consists of:

•  World teaching reputation;

•  World research reputation;

•  World teaching reputation outside region;

•  World research reputation outside region.


Step 1. Processing the initial ranking data


The initial data array is not yet the basis for calculating scores of indicators, i.e. primary information, which, in turn, is used to calculate scores of various types of rankings existing within the system of RUR rankings.


The main problems of the ranking initial data are:

•  Missed data. Not all universities provide the data in the amount required for calculating the ranking. Therefore, there are «gaps» appearing in the initial data.

•  Abnormal values. The institution can provide abnormally high or low values by mistake. For example, financial indicators can be introduced instead of the number of students.

•  The specificity of national education systems can lead to unintentional distortion of information. For example, there are the so-called «undivided degrees» in a number of countries, i.e. the educational programs lasting 5-6 years awarded with specialist degree equivalent to master degrees. Therefore, universities with dominating educational programs at the specialty level can indicate a small number of the bachelor degrees awarded, which leads to abnormally high values in the ranking indicator using the number of the bachelor degrees awarded.


Step 2. Raw data processing


The rules below are relevant for the subject rankings of RUR 2016, the general ranking of RUR 2017, and all subsequent rankings. These rules are not applied to the subject rankings of 2014-2015 and general rankings of 2010-2015 (except for the 1st one).


1)  Testing for abnormal values:

Each data array is tested for abnormal values. For this, each array of initial parameters is evaluated in the manual mode. If a higher education institution has abnormally high or low values, such values are checked again by sending inquiries to those universities, as well as by gaining data from the third parties.


2)  Clipping indicators:

The threshold values are introduced in order to minimize the effect of the anomalous values that occur at the stage of calculating the relative data from the initial data. If actual values for a number of indicators are less than the threshold values listed below, then for all relative indicators of the ranking, in which these absolute values are used, the institution is assigned 25% of the universities average in the country where the institution is located. If the only institution represents a country in the ranking, then this institution is assigned 25% of the world average by the corresponding indicator.


Thresholds by indicators:

•  Academic staff – 50;

•  Students – 200;

•  Bachelor degrees awarded – 50;

•  Degrees awarded – 20;

•  Publications – 50.


For example, if a University X has 170 students, then according to «Academic staff per students» indicator this university is assigned 25% of the average number of students in the country where it is located.


3)  Data omissions:

•  If a university does not have any initial value, then it is assigned 25% of the average value of the country. If there is only one university in the country, then this institution is assigned 25% of the world average.


4) Bachelor degrees awarded:

•  If the share of bachelor degrees awarded is less than 5% of the total number of students, the actual value of the institution according to the initial parameter of «bachelor degrees awarded» changes by 50% of the national average. This is done so that universities with undivided degrees did not receive a significant advantage in calculating the scores of those ranking indicators that use the number of bachelor degrees awarded.


5) The exclusion of universities from the ranking provided that there are no input data (this rule is relevant for ranking versions).


In the 2010-2016 rankings the university was excluded from consideration, if the data missed on more than:

•  5 initial parameters in the 2010-2015 rankings;

•  7 initial parameters in the rankings of 2016, subject rankings of 2017 and subsequent years.


In RUR World University Ranking 2017 this rule does not apply. In other words, all universities participate in the general ranking from 2017 on regardless of how much data is missing in the university initial source.


Step 3.  Relative values calculation


After preparing the initial data set, relative data are calculated from absolute ones (i.e. the number of academic staff, students, publications, citations, etc.). Relative values are a product of 2 values. For example, if we need to get the value for «Academic staff per students» indicator, the number of academic staff is divided by the number of students. But not the other way around.


Of the 20 indicators, RUR-relative values are used in 15 cases out of 20. In the remaining 5 cases, the absolute values are transferred unchanged to the relative category. These are the following values:

•  World teaching reputation;

•  Normalized citation impact

•  World research reputation;

•  World reputation outside region;

•  International level.


Step 4.  Scores calculation


After the preparation of an array of relative data, the calculation of scores on 20 ranking indicators begins. To do this, the relative data array is ranked from the largest to the smallest. After that each university has a percentile assigned on 20 arrays of relative values, i.e. a value on a 100-point scale based on the rank (number) of the university and the size of the sample.


Example. If there are 1,000 universities in the sample, then the higher education institution with the highest score gets 100 points, the second university does 99.9 points, etc. Thus, the percentile’s «step» is obtained, which depends on the size of the data sample.


Additional rules


1)  Repeating Relative Values:

If a number of relative values ⊗⊗is repeated, then all HEIs with the same values ⊗⊗are assigned average percentiles.


For example: the last 50 universities in a 1,000-institution sample have the same values. These universities are assigned from 0 to 5 points. The average value is 2.5. Thus, each of the 50 universities is assigned 2.5 points.


As a rule, reputation indicators contain a lot of identical values⊗.


2)  Calculation of international level indicator


This rule is relevant only for the rankings from 2016 on.


International level is the only indicator whose score is calculated not from relative values, but from the average score. In this case, the average of the 4 indicators of the «International Diversity» group is calculated.


Step 5.   RUR World University Rankings scores and positions calculation


The scores assigned at the 4-th stage are converted to the university final score, which is used to rank universities and finalize the universities ranking:

•  Scores of ranking indicators are multiplied by weights. In RUR this is either 8% or 2%;

•  The results are summarized;

•  Universities are ranked on the basis of the sum of universities score products and weights. At this stage, the ranking is ready;

•  University scores are listed on a 100-point scale. For this purpose, the highest absolute score is assigned to be equal to 100 points. The other institutions scores are considered in proportion to 100 points;

•  The ranking is ready.


Step 6.  Calculation of scores and positions by measured groups


Rankings by measured groups (teaching, research, international diversity, financial sustainability) are calculated as follows:

•  Each ranking by measured groups has 5 indicators from the indicator group with the same name;

•  When calculating the absolute score of the university, the common weight of 20% is used for all rankings in the measured groups.


Step 7. Calculation of subject rankings


RUR Subject rankings are calculated similar to the basic, general rankings.


The only difference is the 1-st indicator of «Research» group. There is «post-bachelor students per the total number of students» indicator used instead of «Doctoral degrees awarded per admitted PhD».


In other respects, the stages and algorithms for calculating the subject rankings are identical to the algorithms for calculating the overall rankings.


About Round University Ranking


Round University Ranking (RUR) is an international university rankings published by the Ranking Agency RUR. The rankings have evaluated the activity of 930 leading universities from 80 countries around the world for the period of 8 years (2010-2017) on 20 indicators assessing 4 groups of universities activity: quality of teaching, quality of research, level of internationalization and level of financial sustainability. Such a wide coverage, both geographically and temporally, makes RUR a unique tool for choosing universities for study and work comparing and correlating higher education institutions on a global scale. In addition to the overall ranking, RUR includes 30 rankings on 6 subject areas: humanities, life sciences, medical sciences, natural sciences, social sciences, engineering.


RUR International University Ranking is the first regularly updated global ranking of Russian universities. The ranking was developed to enable key stakeholders - university entrants and students, the academic community, university and state management - to evaluate and make decisions on what universities to choose in accordance with their personal needs and requirements.


About RUR Rankings Agency


RUR Ranking agency is a Russian ranking agency, existing since 2013. The agency publishes the University Ranking system. The data provider for the RUR system is an international company of Clarivate Analytics (formerly Thomson Reuters IP & Science Solutions).