1. bookVolumen 6 (2021): Heft 2 (April 2021)
Zeitschriftendaten
License
Format
Zeitschrift
eISSN
2543-683X
Erstveröffentlichung
30 Mar 2017
Erscheinungsweise
4 Hefte pro Jahr
Sprachen
Englisch
access type Uneingeschränkter Zugang

Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

Online veröffentlicht: 02 Mar 2021
Volumen & Heft: Volumen 6 (2021) - Heft 2 (April 2021)
Seitenbereich: 67 - 95
Eingereicht: 01 Nov 2020
Akzeptiert: 25 Jan 2021
Zeitschriftendaten
License
Format
Zeitschrift
eISSN
2543-683X
Erstveröffentlichung
30 Mar 2017
Erscheinungsweise
4 Hefte pro Jahr
Sprachen
Englisch
Abstract Purpose

Building on Leydesdorff, Bornmann, and Mingers (2019), we elaborate the differences between Tsinghua and Zhejiang University as an empirical example. We address the question of whether differences are statistically significant in the rankings of Chinese universities. We propose methods for measuring statistical significance among different universities within or among countries.

Design/methodology/approach

Based on z-testing and overlapping confidence intervals, and using data about 205 Chinese universities included in the Leiden Rankings 2020, we argue that three main groups of Chinese research universities can be distinguished (low, middle, and high).

Findings

When the sample of 205 Chinese universities is merged with the 197 US universities included in Leiden Rankings 2020, the results similarly indicate three main groups: low, middle, and high. Using this data (Leiden Rankings and Web of Science), the z-scores of the Chinese universities are significantly below those of the US universities albeit with some overlap.

Research limitations

We show empirically that differences in ranking may be due to changes in the data, the models, or the modeling effects on the data. The scientometric groupings are not always stable when we use different methods.

Practical implications

Differences among universities can be tested for their statistical significance. The statistics relativize the values of decimals in the rankings. One can operate with a scheme of low/middle/high in policy debates and leave the more fine-grained rankings of individual universities to operational management and local settings.

Originality/value

In the discussion about the rankings of universities, the question of whether differences are statistically significant, has, in our opinion, insufficiently been addressed in research evaluations.

Introduction

Classifications and rankings are based on assumptions and decisions about parameters. For example, Harvard University is listed at the top in many rankings. However, when one controls for the budget by dividing the numbers of publications and/or citations (output) by budget (input), other universities come to the fore. Leydesdorff and Wagner (2009), for example, found Eastern-European universities (Poland, Slovakia) as most efficient in terms of output/dollar, because of the relatively low costs of skilled labour in these countries at the time. The ranked order is conditioned by the choice of indicators.

In China, for example, the mathematics department of Qufu Normal University unexpectedly led the ranking of US News with a 19th position worldwide.

https://www.usnews.com/education/best-global-universities/china/mathematics; retrieved 27 October 2020.

This unexpected result generated a heated discussion about rankings. Qufu Normal University is a provincial university. How could it be ahead of Peking University, which is traditionally believed to have the strongest mathematics department in China. Shandong University of Science and Technology was ranked third, trailing Peking University by only 0.1 points (Tsinghua University ranked sixth.) However, the majors in mathematics of these two universities are rated relatively low when using other indicators including international collaborations. In sum, a rank-order is conditioned by the choice of indicators and by how the various indicators are weighed in the attribution (Shin, Toutkoushian, & Teichler, 2011).

Comparing universities may be even more complex than comparing nations. While nations can be expected to cover portfolios of standardized measurements,

See, for example, OECD’s Frascati Manual: https://www.oecd.org/publications/frascati-manual-2015-9789264239012-en.htm

universities can be specialized by discipline or by mission (e.g. agricultural universities). What counts as “output” of universities is debatable. Furthermore, universities provide higher-education, and one can argue that this task should be appreciated in a ranking, even if it is difficult to measure. University ranking services seek to compare such heterogeneous indicators by weighting, summing, or averaging partial indicators, but often without giving much value to mission-orientation and social relevance.

The reliability and reproducibility of yearly rankings is low because the models are based on specific choices in a parameter space of possible choices. Gingras (2016), for example, concluded “that annual rankings of universities, be they based on surveys, bibliometrics, or webometrics, have no foundation in methodology and can only be explained as marketing strategies on the part of the producers of these rankings.” Others have pointed to the reputational effects in rankings. Rankings can instantiate historical power structures (Bowman & Bastedo, 2011). However, numbers can also provide a new perspective because ongoing changes may have gone hitherto unnoticed. We shall provide examples of such unexpected developments below.

In our opinion, statisticians and bibliometricians have made progress during the last decade in developing arguments for making informed decisions about the data and relevant methodologies for comparing institutions. The rankings no longer have to be arbitrary and subjective combinations of surveys and partial indicators. However, they remain constructs which enable us to distinguish among universities in specific terms. For example, the Leiden Rankings (LR) of research universities stand out in terms of their transparency and clear objectives. The objective of the LR exercises is to rank research universities exclusively in terms of their publications and citations as output, without accounting for other differences.

Decisions about data

We use in this study LR and not, for example, the Shanghai Jiaotong University Ranking, because the data curation is bibliometrically transparent and easily reproducible in LR. Our analytical and statistical arguments, however, can analogously be applied to the other ranking (e.g. Peters, 2017; Shin, Toutkoushian, & Teichler, 2011). In addition to providing the rankings online (https://www.leidenranking.com/ranking/2020/list), the yearly source data for LR are freely available in an Excel file for secondary analysis. This data is derived from the Web of Science of the Institute of Scientific Information (ISI/Clarivate) in Philadelphia. The address information is disambiguated and reorganized by the Centre for Science and Technology Studies (CWTS) in Leiden.

LR does not take into account conference proceedings and book publications; it is exclusively based on research articles and reviews published in the journals included in the Web of Science.

Clarivate’s criterion for classifying papers as reviews is as follows: “In the JCR system any article containing more than 100 references is coded as a review. Articles in ‘review’ sections of research or clinical journals are also coded as reviews, as are articles whose titles contain the word ‘review’ or ‘overview’” (at http://thomsonreuters.com/products_services/science/free/essays/impact_factor/ (retrieved 8 April 2012); https://clarivate.com/webofsciencegroup/essays/impact-factor/ (retrieved on 29 August 2020)).

Consequently, the coverage may favor work in the natural and engineering sciences more than the social science and humanities (Sivertsen, 2016). However, this data can be considered as an attempt to capture the most intellectual contribution to knowledge production among the numerous outputs of research universities (Garfield, 1971). A major decision remains thereupon to attribute co-authored papers for a full count to each of the (co-)authors and their home institutions or to attribute this credit proportionally (so-called “fractional counting”); both of these methods are made available in the Excel files.

The full counting method gives a full point to each author and her institutional address. Fractional counting divides the attribution by the number of authors or institutions. Bibliometrically, fractional counting has the advantage that each publication is eventually counted as one full point and thus percentages add up to hundred (Anderson et al., 1988). Among the possible counting rules, LR uses fractional counting at the institutional level. For example, if a publication is co-authored by three researchers and two of these researchers are affiliated with a particular university, the publication has a weight of 2 / 3 = 0.67 in the attribution of the scientific impact to this university (Sivertsen, Rousseau, & Zhang, 2019).

Another important decision is the time window used to harvest publications and citations, respectively. LR uses four-year periods for cited publications. The last interval for LR 2020 is 2015–2018; the count included citations to publications in these four years cumulatively. The resulting file with input data for this study contained ranks for 1,176 research universities in 65 countries in the preceding years in intervals of four years (Table 1). Rankings are provided both fractionally and as whole counts. In terms of disciplines, data is provided for “All sciences” and five major fields on the basis of relevant journal categories: (i) biomedical and health sciences, (ii) life and earth sciences, (iii) mathematics and computer science, (iv) physical sciences and engineering, and (v) social sciences and humanities. In this study, we limit the analysis first to “All sciences,” the last available period (2015–2018), and fractional counting. However, this analysis can be repeated analogously using subsets with other parameter choices.

Number of research universities included in LR 2020 for respective countries.

Country (top 10 of 65) N of universities
China 205
United States 198
United Kingdom 58
Germany 54
Japan 53
South Korea 44
Italy 41
Spain 41
India 36
Iran 36 %

sum 766 65.1%

<55 other countries> = (1,176 – 766) = 410 34.9%

65 countries 1,176 100%

Our sample contains 205 Chinese universities (Table 2) covered by LR 2020. This data was further processed by us with dedicated routines, which are online available at http://www.leydesdorff.net/software/leiden (Leydesdorff, Bornmann, & Mingers, 2019). These routines analyze the comparisons in each nation. China and the US, however, are represented in LR 2020 with 205 and 198 universities, respectively, making comparisons possible. For example, one can compare Harvard with Stanford as US universities, and—as we shall see below—it is possible to compare them with Tsinghua or Zhejiang University.

Methods
Statistical significance

The values of indicators are almost by definition unequal between two measurements, but one can ask whether differences are statistically significant or fall within the margin of error. In a study about German, British, and US universities, Leydesdorff, Bornmann, and Mingers (2019) proposed three statistics for distinguishing among groups of universities: (i) overlapping confidence intervals, (ii) z-tests based on differences between observed and expected values of the percentage publications in the top 10% group (PP-top10%)—and (iii) effect sizes (Cohen, 1988). Although many statisticians nowadays have a preference for the latter measure (Schneider, 2015; Waltman, 2016), effect sizes are less known to practitioners. Power analysis based on effect sizes is sometimes considered to be of “practical significance” (Cumming, 2013; Wasserstein & Lazar, 2016). However, the scientometric interpretations of the effect sizes in our previous study were not convincing.

Building on Leydesdorff, Bornmann, and Mingers (2019), we elaborate the differences between Tsinghua and Zhejiang University numerically and step-by-step: (i) we address the question of whether differences are statistically significant in the rankings of Chinese universities; (ii) we propose methods for measuring statistical significance among different universities within or among countries. We limit the discussion here to z-testing and overlapping confidence intervals. Using the z-scores, one can test the differences for significance and thus generate groups of nodes with links among them indicating group membership. These results can be analyzed and visualized as clusters. The z-scores can also be used as quantitative measures of differences among nodes (universities) and links (between universities).

We limit the discussion here to z-testing and overlapping confidence intervals. When the differences between two universities are not significantly different, they can be grouped together. Using the z-scores or the relative overlaps, one can test the differences and thus generate groups of nodes with links among them indicating group membership. These results can be analyzed and visualized as clusters. The z-scores can also be used as quantitative measures of differences among nodes (universities) and links (between universities).

Observed versus Expected

Most commonly, one tests for the significance of the differences between mean values of variables or, in the case of citation analysis, between the so-called c/p-ratios—that is, the mean numbers of citations per publication. However, scientometric distributions are highly skewed; the mean is therefore not a meaningful indicator of the central tendency in the distribution.

The so-called “crown indicator” in scientometrics (van Raan et al., 2010; Waltman et al., 2011) is unfortunately the Mean Normalized Citation Score (MNCS). As noted, the mean is an unfortunate choice as a central-tendency statistic of a skewed distribution.

An alternative could be found in using the median which is by definition equal to the top 50% of the distribution. Given the skew of the distribution, however, quality indicators can also focus on the numbers of top 10% or even top 1% most-highly cited papers (Bornmann & Mutz, 2011; Leydesdorff et al., 2011; McAllister, Narin, & Corrigan, 1983; Tijssen, Visser, & van Leeuwen, 2002).

Counts allow for significance testing of the differences between expected and observed values using non-parametric statistics. For example, chi-square is formulated as follows: χ2=i=1n(Observedi-Expectedi)2Expectedi {\chi ^2} = \sum\nolimits_{i = 1}^n {{{{{(Observed_{i} - Expected_{i})}^2}} \over {Expected_{i}}}}

Significance of the resulting chi-square values can be looked-up in any table of chi-square values; for example, at the Internet.

Additionally, and in this context importantly, the individual terms before the squaring [(Observedi-Expectedi)Expectedi] \left[ {{{(Observed_{i} - Expected_{i})} \over {\sqrt {Expected_{i}}}}} \right are the so-called standardized residuals of the chi-square. Any residual with an absolute value > 1.96 is significant at the 5% level, and any residual > 2.576 is significant at the 1% level. In other words, the residuals are z-scores enabling a detailed investigation of differences at the level of each individual cell of a matrix.

Let us elaborate with a numerical example comparing Tsinghua and Zhejiang University. Table 2a shows the values obtained from LR 2020 data for these two leading universities.

Observed values of top 10% cited papers for Tsinghua and Zhejiang University during the period 2015–2019 (fractional counting, all sciences).

Observed values top 10% non-top total p
Tsinghua 2,738 17,164 19,902
Zhejiang 2,604 20,906 23,510
5,342 38,070 43,412

The expected values can be derived from the observed ones by using the margin-totals and grand-total of the cross-table as follows: Expectation(ij) = (Σ.j Σi.) / Σ..). In other words, the product of the column total and the row total is divided by the grand total of the matrix. Applying this counting rule, the expected number of papers in the top 10% category of Tsinghua University is (19902 * 5342) / 43412 = 2449.01. This value is written in the top-left cell of Table 2b.

Expected values of top 10% cited papers for Tsinghua and Zhejiang University.

Expected values top 10% non-top total p
Tsinghua 2,449.01 17,452.99 19,902.00
Zhejiang 2,892.99 20,617.01 23,510.00
5,356.09 38,055.91 43,412.00

Using Eq. 1, Table 3 shows the contributions of cell values to the chi-square. The sum of the values in Table 3 is the chi-square; in this case 71.80. The corresponding p-value is < 0.001 and thus the differences are statistically significant.

Chi-square for the comparison of Tsinghua and Zhejiang University in LR 2020.

Chi-square top 10% non-top
Tsinghua 34.10 4.79 38.89
Zhejiang 28.87 4.05 32.92
χ2 = 71.80

Table 4 adds the residuals of the chi-square for this data. Tsinghua ranks significantly above expectation in the top 10% category (z > 2.576), and non-significantly below expectation in the other publications. For Zhejiang the opposite is the case. In conclusion: these two universities cannot be considered statistically as belonging to the same group.

Standardized residuals of the chi-square values in Table 4.

Top 10% Non-top
Tsinghua 5.84 −2.19
Zhejiang −5.37 2.01
z-test

Without prior knowledge of historical or social contexts, one would expect that 10% of a university’s publications will belong to the 10% most-highly cited papers in the reference group. A university that publishes more than 10% of these top papers scores above expectation. The z-test can be used to test the significance of the observed number of papers in the top 10% segment against the expected 10%. In addition to comparing a university with the expectation, the test can be applied to the differences between any two universities. A value of z = 1.96 indicates a significance of the difference at the 5% level: z > 2.576 indicates that the chance process is only 1 in 100, and for z > 3.29, the chance rate is only one per mille. A negative z-score indicates mutatis mutandis that the score is below expectation. The resulting z values can be compared and used for ranking purposes.

It can be derived from Eq.1 that the test statistics between two proportions—percentages are proportions—can be formulated as follows (Sheskin, 2011): z=p1-p2p(1-p)[1n1+1n2] z = {{{p_1} - {p_2}} \over {\sqrt {p(1 - p)\left[ {{1 \over {{n_1}}} + {1 \over {{n_2}}}} \right]}}} where n1 and n2 are the numbers of all the papers published by institutions 1 and 2 (under the column “P” in the LR); and p1 and p2 are the values of PPtop 10% of institutions 1 and 2. The pooled estimate for proportion p is defined as: p=t1+t2n1+n2 p = {{{t_1} + {t_2}} \over {{n_1} + {n_2}}} where: t1 and t2 are the numbers of top 10% papers of institutions 1 and 2. These numbers can be calculated on the basis of the values for “P” and “PPtop 10%” in LR. When testing values for a single university, n1 = n2, p1 is the value of the PPtop 10%, p2 = 0.1, and t2 = 0.1 * n2 (that is, the expected number in the top 10%).

Using the same numerical example as above, n1 = 19,902 for Tsinghua and n2 = 23,510 for Zhejiang, respectively; t1 = 2738 and t2 = 2604 (Table 2a). The pooled estimate p is in this case: p=2738+260419902+23510=534243412=0.123054p(1-p)=0.1234*0.8766=0.1081 \matrix{{p = {{2738 + 2604} \over {19902 + 23510}} = {{5342} \over {43412}} = 0.123054} \cr {p(1 - p) = 0.1234*0.8766 = 0.1081} \cr}

Using Eq. 2, one can fill out as follows: z=13.8-11.11000.1081[119,902+123,510]=8.525 z = {{{{13.8 - 11.1} \over {100}}} \over {\sqrt {0.1081\left[ {{1 \over {19,902}} + {1 \over {23,510}}} \right]}}} = 8.525

The z-test indicates that the scores of these two universities are statistically different above the 0.001 level.

It can be shown that if both the z-test and the chi-square are applied to the same set of data, the square of the z-value is equal to the chi-square value (Sheshkin, 2011). The residuals to the chi-square are standardized as a z-statistics as well. The z-test for two independent proportions (e.g. percentages) provides an alternative large-samples procedure for evaluating contingency tables. The z-test is the most appropriate test given the research questions and the design (Sheshkin, 2011).

At http://www.leydesdorff.net/leiden11/index.htm the user can retrieve a file leiden11.xls which allows for feeding values harvested from the LR for the comparison of any two universities. The effect sizes are additionally provided in the template.

Confidence intervals

The LR additionally provides confidence intervals

The confidence intervals are based on bootstrapping; that is, random drawings that are sufficiently repeated to provide stable patterns. In case of the LR, one draws thousand times a sample from each university’s set of publications. In order to obtain a 95% stability interval, the lower and upper bounds of the stability intervals are taken as the 2.5th and the 97.5th percentiles of the thus generated distribution of PPtop 10% values (Waltman et al., 2012).

which can be used as another statistic for grouping universities.

Bornmann and Leydesdorff’s (2013) analysis of LR 2011 compared the stability intervals with other possible ways to calculate standard errors (e.g. the standard errors of a binary probability). They found a perfect correlation between stability intervals and these other possible ways which are based on less data-intensive computing procedures.

When the confidence intervals of two universities overlap, the distinction between them can be ignored in terms of the indicator (e.g. Colliander & Ahlgren, 2011). The words confidence and stability intervals can be used interchangeably.

Since each of two universities may be indistinguishable from other universities, one thus obtains a so-called “weak” component in terms of network analysis. If both the upper and lower bounds of university A are contained within the stability interval of university B, the performance of the former can be seen as similar to the latter; network analysis would place them into the same cluster. In this case, we have a strong component since both arcs are valued.

Using the same example of comparing Tsinghua with Zhejiang, Figure 1 shows that there is no overlap between their confidence intervals. This accords with our previous conclusion that the output of these two universities in terms of PP-top10% is significantly different. For didactic reasons, we added Peking University to the comparison in Figure 1 and Table 7 so that we can draw Figure 2 with the z-values as a further illustration of the options for visualization.

Figure 1

Potentially overlapping confidence intervals of the PP-Top10% for three leading Chinese universities.

Figure 2

Grouping of significantly different and non-different values for PP-top 10% among three Chinese universities; z-values (provided in the figure) are used for sizing the nodes.

Zhejiang and Peking University both have a lower bound of 10.5% of top 10% articles. The upper bound of Zhejiang, however, is higher than for Peking University. Table 5 shows the z-values for these three nodes on the main diagonal and the links off-diagonal. The z-values are also written into Figure 2.

z-values for nodes and links among three leading Chinese universities.

Peking University Tsinghua University Zhejiang University
Peking University 2.689 **
Tsinghua Univ. 8.460 *** 11.005 ***
Zhejiang University 0.638 8.533 *** 3.800 ***

Significance levels:

p < .05;

p < .01;

p < .001

The difference between Peking and Zhejiang University is not significant, leading to an arc between these two universities (z = 0.64) in Figure 2 and placing them into a cluster together. This weak component does not include Tsinghua University which scores significantly different on this performance indicator (PP-Top10%).

Results
Results based on using the z-test

Aggregating universities into networked groups based upon their z-score similarities creates clusters. Figure 3 shows the resulting clusters among 205 Chinese universities. The z-values are used as input to the sizes of the nodes and fonts, and the z-values between two universities determine the lines insofar as z < 2.576 (p < .0.01) since universities which are not significantly different, are considered as part of the same group. Note that this grouping is on the basis of (structural) similarity and not on actions. We use VOSviewer only for the layout and visualization. The grouping is based on the above statistics. The file is first organized as partitions in Pajek which are exported to VOSviewer. We chose this procedure in order to prevent parameters in VOSviewer (e.g., resolution) to affect the grouping. (In other words, the clustering is not updated in VOSviewer.)

Figure 3

Grouping of 205 Chinese universities in terms of z-values (p <.01); VOSviewer used only for the visualization; modularity Q = 0.165. The figure can be web-started from here (or with a black background from here).

Three major groups of universities and two isolates are distinguished in the analysis and shown in Figure 3: A top-group of 32 universities is listed in Table 9 (Appendix). As could be expected, Tsinghua leads this group with a z-score of 11.0, followed by Hunan University (z = 10.2).

The second group of 69 universities is headed by Zhejiang University and listed in Table 10 (Appendix). As shown above, Peking University is not significantly different from Zhejiang University; it is ranked in the 24th position overall but, following Zhejiang, Peking University is in the 5th position within the second group. Only the top 30 (43.5%) of these 69 universities score on the PP-10% above expectation. A third group of 102 universities are listed in Table 11 (Appendix). None of these universities score above expectation.

Two universities—Chang’an and Shanxi—form a fourth cluster with both negative z-values. Table 6 provides the top 20 universities in each of the three groups in decreasing order (on the basis of the z-values). Note that five of the 20 universities listed on the top-list are from an address in Hong Kong, China.

Top-20 universities in each of the three clusters. (See for a full list in the Annexes.)

top group (top 20) z middle group (top 20) z bottom group (top 20) z
1 Tsinghua Univ. 11.005 Zhejiang Univ. 3.800 Hubei Univ. −0.498
2 Hunan Univ. 10.193 Harbin Institute of Technology 3.226 Northwest Univ. −1.609
3 Hong Kong Univ. of Science and Technology 6.566 Huazhong Univ. of Science and Technology 2.907 South China Agricultural Univ. −1.631
4 Univ. of Science and Technology of China 6.482 Peking Univ. 2.689 Zhejiang Univ. of Technology −1.676
5 City Univ. of Hong Kong 6.454 Nanjing Univ. 1.973 China Univ. of Mining and Technology −1.678
6 Shandong Univ. of Science and Technology 6.444 Xiamen Univ. 1.954 Nanjing Univ. of Aeronautics and Astronautics −1.816
7 Hong Kong Polytechnic Univ. 6.406 Wuhan Univ. 1.765 Second Military Medical Univ. −1.844
8 South China Univ. of Technology 6.049 Tianjin Univ. 1.738 Zhejiang Sci-Tech Univ. −1.886
9 Chinese Univ. of Hong Kong 5.993 Dalian Univ. of Technology 1.553 Taiyuan Univ. of Technology −1.934
10 Nankai Univ. 4.860 East China Normal Univ. 1.333 Shanghai Univ. of Traditional Chinese Medicine −1.950
11 Univ. of Hong Kong 4.418 Soochow Univ. 1.310 Qingdao Univ. of Science and Technology −1.961
12 Shenzhen Univ. 4.089 Univ. of Electronic Science and Technology of China 1.154 Nanjing Univ. of Chinese Medicine −1.981
13 Qufu Normal Univ. 3.724 Nanjing Univ. of Science and Technology 1.140 Harbin Engineering Univ. −2.033
14 Fuzhou Univ. 3.487 Southwest Jiaotong Univ. 1.088 Lanzhou Univ. −2.110
15 Beihang Univ. 3.432 China Univ. of Geosciences 1.035 Northeast Normal Univ. −2.128
16 Univ. of the Chinese Academy of Sciences 3.354 Huazhong Agricultural Univ. 1.030 China Jiliang Univ. −2.130
17 Central China Normal Univ. 3.090 Tongji Univ. 0.954 Jiangnan Univ. −2.163
18 Univ. of Macau 3.046 Univ. of Science and Technology Beijing 0.932 Jinan Univ. −2.227
19 Northwestern Polytechnical Univ. 2.915 China Agricultural Univ. 0.704 Hangzhou Normal Univ. −2.245
20 Wuhan Univ. of Technology 2.914 Beijing Institute of Technology 0.587 Southern Medical Univ. −2.263

The results are sometimes counterintuitive. However, Brewer, Gates, & Goldman (2001) drew attention to the difference between prestige and reputation; prestige is sticky, whereas the citation windows are only four years in LR. For example, Fudan University is considered a prestigious university in China. In the period under study, however, Fudan was listed as an address in 15,442 papers in journals included in the ISI-list. Only 1,395 of these papers (or 9.0%) belonged to the top 10% most-cited papers. This profile is not significantly different from other universities in the third group.

The reason for this relatively low rank of Fudan University is not a decline of publications with this university among the addresses, but a relative decline of papers in the top 10% in this university’s publications. The number of publications with Fudan University among the author-addresses shows an increase of 1,127 papers over the consecutive four-year periods and on the basis of fractional counting (Figure 4). However, the yearly increase in the number of publications in the top 10% segment is only 6.9% (775 papers/year). The relative decline can be cumulative as a composed interest rate (Figure 4). It may be caused by all kind of effects in the data or in the model. Ceteris paribus, for example, an increase in fractional counting leads to a decline in the share of publications and citations (e.g. Anderson et al., 1988).

Figure 4

Development of the values of P, P-top10%, and PP-Top10% for Fudan University during 2016–2020.

Mutatis mutandis, one can be puzzled by the high status of Shandong University of Science and Technology at the sixth position and Qufu Normal University at the 13th on the top-list. Shandong University published 1,576 articles during the period under study, of which 299 belong to the top 10%. This is almost 19% and thus far exceeds the expectation of 10%. Further analysis may enable us to understand these counterintuitive results.

Results based on confidence intervals

Figure 5 provides the resulting figure using the overlaps in confidence levels for the delineation of groups: 75 universities are classified as top-universities. These universities are listed in Table 12 (Appendix).

Figure 5

Grouping of 205 Chinese universities based on overlapping confidence intervals; VOSviewer used for the decomposition and clustering; font- and node-sizes based on z-values. (The map can be web-started from here.)

A measure for the correspondence between the two classifications—the one above based on z-scores (Figure 2) and this one (Figure 5)—is provided by Cramèr’s V, which is based on chi-square statistics, but which conveniently varies between 0 and 1. Cramèr’s V between these two classifications is significant (V = 0.48; p < 0.01).

An alternative measure is phi; phi = 0.831 (p < 0.01). The Spearman rank-order correlation between the clustering based on the two methods is .6 (p < 0.01).

Comparison among Chinese and American universities

In the LR2020, universities are grouped by nation-states, but it is possible to draw samples of universities across nations or within nations. Classification of universities at lower levels of aggregation can be relevant for the study of regional innovation systems. Whereas universities remain the units of analysis in LR, relevant samples can be drawn from the database on the basis of a specific research question.

For example, if the research question is about comparing universities in the EU, one can study the EU as a single unit internally different from the USA or China. The organization at the level of EU is probably different from the sum of the national perspectives. As noted, LR also contains information for six major fields so that one can cross-tabulate nations with these disciplinary categories, where one could test in principle the conjecture that “China is strong in the basic sciences and the US in the biomedical sciences.” We will leave this for a later study, but focus here first on how to compare US and Chinese universities in a single framework.

Both the US and China happen to be represented with approximately 200 universities in LR 2020 (Table 8). Among the 198 American universities, Rockefeller University is an extreme outlier with more than 30% of the papers in the top 10% most-highly cited group. The following analysis is based on 205 Chinese and (198 – 1 (Rockefeller University) =) 197 American universities. The clustering in the network among these (197 + 205 =) 402 universities is visualized in Figure 6. Three clusters and a few isolates are distinguished by the statistical analysis. The isolates are: George Mason University and the University of Toledo in the USA, and the Hangzhou Dianzi University in China. The cross-tabulation in Table 7 is significant at the one-percent level (χ2 = 93.40; p < .01).

Figure 6

Grouping of 205 Chinese and 197 US universities in terms of z-values; differences among groups are significant at the 1% level; VOSviewer used for the decomposition and clustering. (The map can be web-started from here.)

Descriptive statistics of the comparison among 197 American and 205 Chinese universities.

Row Labels low middle High Isolates Grand Total
China 116 67 21 1 205
USA 36 60 99 2 197
Grand Total 152 127 120 402

Table 8 shows the top 20 Chinese universities juxtaposed to the 20 American universities with highest z-scores at different scales. This table reveals that the highest ranked among the Chinese universities (Tsinghua with z = 11.005) does not reach the z-level of the lowest among the 20 most-highly ranked American universities in the right column (z = 12.23). This may be due in part to historical factors where Chinese authors are not as well integrated into the network of science as others and thus struggle to gain citations, and it may also reflect some quality issues, as well. Figure 7 shows that the distribution of z-scores is systematically lower for Chinese universities than for their US counterparts. In other words, using these parameters, China is still far behind the US in terms of the quality of its universities.

Highest ranked universities in top-groups in the combined American-Chinese set of 402 universities in LR 2020 with their respective z-scores.

rank Top 20 Chinese universities z Top 20 American Universities z
1 Tsinghua Univ. 11.005 Harvard Univ. 36.632
2 Hunan Univ. 10.193 Stanford Univ. 26.028
3 Hong Kong Univ. of Science and Technology 6.566 Massachusetts Institute of Technology 24.504
4 Univ. of Science and Technology of China 6.482 Univ. of California, Berkeley 20.319
5 City Univ. of Hong Kong 6.454 Yale Univ. 16.576
6 Shandong Univ. of Science and Technology 6.444 Univ. of California, San Francisco 16.524
7 Hong Kong Polytechnic Univ. 6.406 Princeton Univ. 16.522
8 South China Univ. of Technology 6.049 Columbia Univ. 16.348
9 Chinese Univ. of Hong Kong 5.993 Univ. of California, San Diego 16.061
10 Nankai Univ. 4.860 Univ. of Pennsylvania 15.975
11 Univ. of Hong Kong 4.418 Cornell Univ. 14.934
12 Shenzhen Univ. 4.089 Univ. of Washington, Seattle 14.620
13 Qufu Normal Univ. 3.724 Johns Hopkins Univ. 14.540
14 Fuzhou Univ. 3.487 Northwestern Univ. 14.488
15 Central China Normal Univ. 3.090 Univ. of California, Los Angeles 14.450
16 Univ. of Macau 3.046 Univ. of Michigan 13.951
17 Wuhan Univ. of Technology 2.914 California Institute of Technology 13.695
18 Southern Univ. of Science and Technology 2.332 Univ. of Chicago 13.561
19 GuangZhou Univ. 2.120 Duke Univ. 12.605
20 Hong Kong Baptist Univ. 2.094 Washington Univ. in St. Louis 12.123

Figure 7

Distribution of z values in decreasing order for 205 Chinese and 197 American universities.

Dynamic effects of changes in the model

The methodology for the normalization in terms of different fields of science is continuously improved by CWTS and the database is expanded with new universities. In LR 2016, for example, the number of universities covered was 842 compared to 1,076 in this study based on LR2020. The expansion of the database from year to year may have an effect on the rankings because universities may enter the comparison with higher or lower scores on the relevant parameter.

To address the problem of changes in the methodology, LR values are recalculated each year for the historical values of the indicators based on the latest methodology. Thus, we have two time series: one based on the yearly series of LR 2016 to LR2020 and one based on the reconstruction of the data using the method of LR 2020. The two series for Fudan university are graphed in Figure 8.

Figure 8

The participation of Fudan University in the top 10% class of papers using the Leiden Rankings for subsequent years as a time series versus the reconstruction using the 2020-model.

In LR 2016, Fudan University had 9.81% of its papers in the top 10% class (all fields). In 2020, the position of Fudan University has dropped to 9.03 % of its papers in the top 10% class. This is a decline of 0.78% (9.81–9.03). Would Fudan University have been able to keep the value of 9.81%, it would still have been in the second tier in 2020. Using the 2020 model, however, the value for 2016 is reconstructed as 9.54%, so that we can conclude that the difference in the data is only 0.37% (that is, 9.81–9.54%). The remaining decline (9.54–9.03 =) 0.51% is an effect of changes in the model. In other words, the model accounts for almost two-thirds of the decline (0.51/0.78 = 65.4%) and the citation data themselves for (0.27/0.78 =) 34.6%. These relative declines are of the same order of magnitude as the ones shown in Leydesdorff, Wouters, and Bornmann (2017) for Carnegie Mellon University in the USA during the period 2012–2016.

Discussion and conclusions

In summary, both changes in the data and changes in the model can result in differences in the rankings. Whereas scientometric indicators are meant to serve “objectivization” of the discussion about quality, the quality of the indicators themselves is also an issue in the discussion of the results which requires attention. Differences may be due to changes in the data, the models, or the modeling effects on the data.

Our main argument has been that differences among universities can be tested for their statistical significance. This allows for a classification, since universities which are not significantly different can be grouped together, regardless of geographic location. The scientometric groupings, however, should not be reified. The groupings are not stable when we use different methods. Particularly at the margins, the attribution may be sensitive to parameter choices.

The statistics relativize the values of decimals in the rankings. One can operate with a scheme of low/middle/high in policy debates and leave the more fine-grained rankings of individual universities to operational management and local settings. Further analysis can reveal points which merit attention and discussion. Is a decline due to a parameter choice, or is there a reason for concern? An alternative view can foreground unseen relationships in the background; the resulting insights can be made the subject of managerial and political interventions. One can expect that qualitative assessments lag behind the ongoing developments. Cultural expectations are conservative, while all universities are under the pressure to change their position in relationship to one another in a competitive environment. One can zoom in and organize follow-up investigations in these cases. The result may initially be unwelcome, but can also induce asking urgent questions.

At the macro level, our results show that Chinese universities do not yet (?) operate at the same levels of performance as those in the USA. Despite concerns in the US about the “competition” from Chinese universities, the latter do not rank in the same elite categories as American universities at this time. As in other national systems, we found three or four groups of universities at national levels. In the case of China, the top-list includes five universities with an address in Hong Kong, perhaps partly because of the historical British tradition operating in the background.

Normative implications

University ranking services contribute to the hierarchy of reputation and status that has been the subject of critique and scholarship. Bourdieu (1998) examined the political, intellectual, bureaucratic, and economic forms of power that influence governance and the allocation of resources based upon elite reputation, claiming a link among them. This line of scholarship views international status and side-by-side comparisons, such as those found in rankings, as operating within a framework of broader geopolitical struggles (Heilbron, Boncourt, & Sorá, 2018). From this perspective, placing Chinese universities into international rankings is a political act rather than simply social science. Rankings, then, become an artifact of globalization reflecting political posturing.

It is important to tread lightly here because a vast literature is available addressing globalization. Certainly, Chinese universities are actively widening the scope of cross-border communication, intensifying transnational mobility, and, apparently, growing dependency on global structures—all features of globalization. Further, geopolitical power struggles may be served in different ways by rankings and their underlying assumptions, such as ones that present China’s universities as challenging those of the West for preeminence in knowledge creation for competitiveness or military hegemony. The rankings can also be considered as instantiating an unequal distribution of power that are at the root of asymmetrical power relations. These measures of symmetry or asymmetry, elite status, or reputation, then become important to challenge from several angles, of which we addressed one here.

Figure 1

Potentially overlapping confidence intervals of the PP-Top10% for three leading Chinese universities.
Potentially overlapping confidence intervals of the PP-Top10% for three leading Chinese universities.

Figure 2

Grouping of significantly different and non-different values for PP-top 10% among three Chinese universities; z-values (provided in the figure) are used for sizing the nodes.
Grouping of significantly different and non-different values for PP-top 10% among three Chinese universities; z-values (provided in the figure) are used for sizing the nodes.

Figure 3

Grouping of 205 Chinese universities in terms of z-values (p <.01); VOSviewer used only for the visualization; modularity Q = 0.165. The figure can be web-started from here (or with a black background from here).
Grouping of 205 Chinese universities in terms of z-values (p <.01); VOSviewer used only for the visualization; modularity Q = 0.165. The figure can be web-started from here (or with a black background from here).

Figure 4

Development of the values of P, P-top10%, and PP-Top10% for Fudan University during 2016–2020.
Development of the values of P, P-top10%, and PP-Top10% for Fudan University during 2016–2020.

Figure 5

Grouping of 205 Chinese universities based on overlapping confidence intervals; VOSviewer used for the decomposition and clustering; font- and node-sizes based on z-values. (The map can be web-started from here.)
Grouping of 205 Chinese universities based on overlapping confidence intervals; VOSviewer used for the decomposition and clustering; font- and node-sizes based on z-values. (The map can be web-started from here.)

Figure 6

Grouping of 205 Chinese and 197 US universities in terms of z-values; differences among groups are significant at the 1% level; VOSviewer used for the decomposition and clustering. (The map can be web-started from here.)
Grouping of 205 Chinese and 197 US universities in terms of z-values; differences among groups are significant at the 1% level; VOSviewer used for the decomposition and clustering. (The map can be web-started from here.)

Figure 7

Distribution of z values in decreasing order for 205 Chinese and 197 American universities.
Distribution of z values in decreasing order for 205 Chinese and 197 American universities.

Figure 8

The participation of Fudan University in the top 10% class of papers using the Leiden Rankings for subsequent years as a time series versus the reconstruction using the 2020-model.
The participation of Fudan University in the top 10% class of papers using the Leiden Rankings for subsequent years as a time series versus the reconstruction using the 2020-model.

75 top universities with overlapping confidence values.

University Z rank within group
Tsinghua University 11.005 1 1
Hunan University 10.193 2 2
Hong Kong University of Science and Technology 6.566 3 3
University of Science and Technology of China 6.482 4 4
City University of Hong Kong 6.454 5 5
Shandong University of Science and Technology 6.444 6 6
Hong Kong Polytechnic University 6.406 7 7
South China University of Technology 6.049 8 8
Chinese University of Hong Kong 5.993 9 9
Nankai University 4.860 10 10
University of Hong Kong 4.418 11 11
Shenzhen University 4.089 12 12
Zhejiang University 3.800 13 13
Qufu Normal University 3.724 14 14
Fuzhou University 3.487 15 15
Beihang University 3.432 16 16
University of the Chinese Academy of Sciences 3.354 17 17
Harbin Institute of Technology 3.226 18 18
Central China Normal University 3.090 19 19
University of Macau 3.046 20 20
Northwestern Polytechnical University 2.915 21 21
Wuhan University of Technology 2.914 22 22
Huazhong University of Science and Technology 2.907 23 23
Peking University 2.689 24 24
Southern University of Science and Technology 2.332 25 25
Beijing University of Chemical Technology 2.273 26 26
GuangZhou University 2.120 27 27
Hong Kong Baptist University 2.094 28 28
Nanjing University 1.973 29 29
Xiamen University 1.954 30 30
Wuhan University 1.765 31 31
Tianjin University 1.738 32 32
Dalian University of Technology 1.553 33 33
Shandong Normal University 1.478 34 34
East China Normal University 1.333 35 35
Jiangsu Normal University 1.313 36 36
Soochow University 1.310 37 37
Zhejiang Normal University 1.238 38 38
University of Jinan 1.195 39 39
University of Electronic Science and Technology of China 1.154 40 40
Nanjing University of Science and Technology 1.140 41 41
Nanjing University of Information Science and Technology 1.093 42 42
Southwest Jiaotong University 1.088 43 43
China University of Geosciences 1.035 44 44
Huazhong Agricultural University 1.030 45 45
Tongji University 0.954 46 46
University of Science and Technology Beijing 0.932 47 47
China Agricultural University 0.704 48 48
Beijing Institute of Technology 0.587 49 49
Sun Yat-sen University 0.572 50 50
Hangzhou Dianzi University 0.449 51 51
North China Electric Power University 0.414 52 52
China University of Petroleum Beijing 0.389 53 53
Guangdong University of Technology 0.332 54 54
Shanghai University 0.332 55 55
Fourth Military Medical University 0.257 56 56
Beijing Normal University 0.178 57 57
Chongqing University 0.000 58 58
Hefei University of Technology 0.000 59 59
Xi’an Jiaotong University 0.000 60 60
Nanjing University of Posts and Telecommunications −0.110 61 61
Heilongjiang University −0.158 62 62
Nanjing Agricultural University −0.171 63 63
China Pharmaceutical University −0.259 64 64
Central South University −0.280 65 65
China University of Petroleum East China −0.407 66 66
Qingdao Agricultural University −0.422 67 67
Beijing Jiaotong University −0.448 68 68
Chang’an University −0.465 69 69
Hubei University −0.498 70 70
Dalian Medical University −0.643 72 71
Chongqing University of Posts and Telecommunications −0.656 73 72
Huaqiao University −0.795 74 73
Anhui Agricultural University −0.862 75 74
Army Medical University −0.952 80 75

z-values for nodes and links among three leading Chinese universities.

Peking University Tsinghua University Zhejiang University
Peking University 2.689 **
Tsinghua Univ. 8.460 *** 11.005 ***
Zhejiang University 0.638 8.533 *** 3.800 ***

Standardized residuals of the chi-square values in Table 4.

Top 10% Non-top
Tsinghua 5.84 −2.19
Zhejiang −5.37 2.01

Bottom group of 102 Chinese universities.

University z rank within-group
Hubei University −0.498 70 1
Northwest University −1.609 100 2
South China Agricultural University −1.631 101 3
Zhejiang University of Technology −1.676 102 4
China University of Mining and Technology −1.678 103 5
Nanjing University of Aeronautics and Astronautics −1.816 106 6
Second Military Medical University −1.844 108 7
Zhejiang Sci-Tech University −1.886 110 8
Taiyuan University of Technology −1.934 111 9
Shanghai University of Traditional Chinese Medicine −1.950 112 10
Qingdao University of Science and Technology −1.961 113 11
Nanjing University of Chinese Medicine −1.981 114 12
Harbin Engineering University −2.033 115 13
Lanzhou University −2.110 117 14
Northeast Normal University −2.128 118 15
China Jiliang University −2.130 119 16
Jiangnan University −2.163 120 17
Jinan University −2.227 121 18
Hangzhou Normal University −2.245 122 19
Southern Medical University −2.263 123 20
Hunan Normal University −2.282 124 21
Henan University −2.328 125 22
Beijing University of Chinese Medicine −2.346 126 23
Tianjin Medical University −2.348 127 24
Shantou University −2.383 128 25
Xi’an University of Architecture and Technology −2.384 129 26
Hainan University −2.417 130 27
Hohai University −2.453 131 28
Beijing University of Posts and Telecommunications −2.458 132 29
Qingdao University −2.465 133 30
Jiangsu University −2.467 134 31
Jiangxi Normal University −2.486 135 32
Shaanxi Normal University −2.505 136 33
Shandong University of Technology −2.506 137 34
Nanchang University −2.528 138 35
Henan Agricultural University −2.584 139 36
Guizhou University −2.675 140 37
Guangxi Normal University −2.709 141 38
Northeast Forestry University −2.751 142 39
Northeastern University −2.761 143 40
Henan Normal University −2.775 144 41
Shandong Agricultural University −2.783 145 42
Anhui Normal University −2.834 146 43
Lanzhou University of Technology −2.835 147 44
Ningbo University −2.838 148 45
Southwest University −2.855 149 46
Yunnan University −2.940 150 47
Shanghai Ocean University −2.954 151 48
Northwest Normal University −2.962 152 49
Inner Mongolia University −2.968 153 50
Beijing University of Technology −2.969 154 51
Fudan University −3.065 155 52
Yanshan University −3.100 156 53
Tianjin University of Science and Technology −3.150 157 54
Beijing Forestry University −3.151 158 55
People's Liberation Army University of Science and Technology −3.152 159 56
Capital Normal University −3.188 160 57
Southwest University of Science and Technology −3.433 161 58
Hebei University of Technology −3.558 162 59
Shenyang Pharmaceutical University −3.713 163 60
Xinjiang University −3.741 164 61
Kunming University of Science and Technology −3.832 165 62
Nantong University −3.935 166 63
Xiangtan University −4.012 167 64
Northeast Agricultural University −4.025 168 65
Wenzhou Medical University −4.062 169 66
Chongqing Medical University −4.106 170 67
Guangxi University −4.119 171 68
Xi’An University of Technology −4.203 172 69
Zhengzhou University −4.244 173 70
Ocean University of China −4.246 174 71
Henan Polytechnic University −4.348 175 72
Shihezi University −4.400 176 73
Kunming Medical University −4.506 177 74
Xuzhou Medical College −4.604 178 75
Tianjin Polytechnic University −4.775 179 76
South China Normal University −4.827 180 77
Nanjing Medical University −4.941 181 78
Chengdu University of Technology −4.958 182 79
Xinxiang Medical University −4.962 183 80
Harbin University of Science and Technology −5.001 184 81
North University of China −5.068 185 82
Yangtze University −5.093 186 83
Guangzhou Medical University −5.200 187 84
Sichuan Agricultural University −5.212 188 85
Sichuan University −5.247 189 86
National University of Defense Technology −5.257 190 87
Harbin Medical University −5.331 191 88
Tianjin Normal University −5.358 192 89
Shanxi Medical University −5.378 193 90
Henan University of Science and Technology −5.416 194 91
Shandong University −5.619 195 92
Anhui Medical University −5.951 196 93
Hebei University −6.307 197 94
Guangxi Medical University −7.267 198 95
Peking Union Medical College −7.621 199 96
Xinjiang Medical University −7.703 200 97
China Medical University −8.064 201 98
Fujian Medical University −8.169 202 99
Hebei Medical University −8.966 203 100
Jilin University −9.607 204 101
Capital Medical University −12.944 205 102

Number of research universities included in LR 2020 for respective countries.

Country (top 10 of 65) N of universities
China 205
United States 198
United Kingdom 58
Germany 54
Japan 53
South Korea 44
Italy 41
Spain 41
India 36
Iran 36 %

sum 766 65.1%

<55 other countries> = (1,176 – 766) = 410 34.9%

65 countries 1,176 100%

Top group of 32 Chinese universities.

University Z overall rank within-group rank
Tsinghua University 11.005 1 1
Hunan University 10.193 2 2
Hong Kong University of Science and Technology 6.566 3 3
University of Science and Technology of China 6.482 4 4
City University of Hong Kong 6.454 5 5
Shandong University of Science and Technology 6.444 6 6
Hong Kong Polytechnic University 6.406 7 7
South China University of Technology 6.049 8 8
Chinese University of Hong Kong 5.993 9 9
Nankai University 4.860 10 10
University of Hong Kong 4.418 11 11
Shenzhen University 4.089 12 12
Qufu Normal University 3.724 14 13
Fuzhou University 3.487 15 14
Beihang University 3.432 16 15
University of the Chinese Academy of Sciences 3.354 17 16
Central China Normal University 3.090 19 17
University of Macau 3.046 20 18
Northwestern Polytechnical University 2.915 21 19
Wuhan University of Technology 2.914 22 20
Southern University of Science and Technology 2.332 25 21
Beijing University of Chemical Technology 2.273 26 22
GuangZhou University 2.120 27 23
Hong Kong Baptist University 2.094 28 24
Shandong Normal University 1.478 34 25
Jiangsu Normal University 1.313 36 26
Zhejiang Normal University 1.238 38 27
University of Jinan 1.195 39 28
Nanjing University of Information Science and Technology 1.093 42 29
Hangzhou Dianzi University 0.449 51 30
Heilongjiang University −0.158 62 31
Qingdao Agricultural University −0.422 67 32

Chi-square for the comparison of Tsinghua and Zhejiang University in LR 2020.

Chi-square top 10% non-top
Tsinghua 34.10 4.79 38.89
Zhejiang 28.87 4.05 32.92
χ2 = 71.80

Middle group of 69 Chinese universities.

University z overall rank within-group
Zhejiang University 3.800 13 1
Harbin Institute of Technology 3.226 18 2
Huazhong University of Science and Technology 2.907 23 3
Peking University 2.689 24 4
Nanjing University 1.973 29 5
Xiamen University 1.954 30 6
Wuhan University 1.765 31 7
Tianjin University 1.738 32 8
Dalian University of Technology 1.553 33 9
East China Normal University 1.333 35 10
Soochow University 1.310 37 11
University of Electronic Science and Technology of China 1.154 40 12
Nanjing University of Science and Technology 1.140 41 13
Southwest Jiaotong University 1.088 43 14
China University of Geosciences 1.035 44 15
Huazhong Agricultural University 1.030 45 16
Tongji University 0.954 46 17
University of Science and Technology Beijing 0.932 47 18
China Agricultural University 0.704 48 19
Beijing Institute of Technology 0.587 49 20
Sun Yat-sen University 0.572 50 21
North China Electric Power University 0.414 52 22
China University of Petroleum Beijing 0.389 53 23
Guangdong University of Technology 0.332 54 24
Shanghai University 0.332 55 25
Fourth Military Medical University 0.257 56 26
Beijing Normal University 0.178 57 27
Chongqing University 0.000 58 28
Hefei University of Technology 0.000 59 29
Xi’an Jiaotong University 0.000 60 30
Nanjing University of Posts and Telecommunications −0.110 61 31
Nanjing Agricultural University −0.171 63 32
China Pharmaceutical University −0.259 64 33
Central South University −0.280 65 34
China University of Petroleum East China −0.407 66 35
Beijing Jiaotong University −0.448 68 36
Xidian University −0.529 71 37
Dalian Medical University −0.643 72 38
Chongqing University of Posts and Telecommunications −0.656 73 39
Huaqiao University −0.795 74 40
Anhui Agricultural University −0.862 75 41
Wuhan University of Science and Technology −0.899 76 42
Nanjing Tech University −0.919 77 43
Donghua University −0.923 78 44
East China University of Science and Technology −0.943 79 45
Army Medical University −0.952 80 46
Anhui University −1.004 81 47
Tianjin University of Technology −1.047 82 48
Renmin University of China −1.064 83 49
Nanjing Normal University −1.189 84 50
Southeast University −1.213 85 51
Fujian Agriculture and Forestry University −1.218 86 52
Northwest Agriculture and Forestry University −1.226 87 53
University of Shanghai for Science and Technology −1.232 88 54
North China University of Science and Technology −1.249 89 55
Wenzhou University −1.306 90 56
Southwest Petroleum University −1.311 91 57
Shaanxi University of Science and Technology −1.314 92 58
China Three Gorges University −1.333 93 59
University of South China −1.345 94 60
Shanghai Normal University −1.369 95 61
Dalian Maritime University −1.380 96 62
Yangzhou University −1.392 97 63
Changzhou University −1.544 98 64
Nanjing Forestry University −1.581 99 65
Guangzhou University of Chinese Medicine −1.685 104 66
Fujian Normal University −1.781 105 67
China Academy of Chinese Medical Sciences −1.840 107 68
Shanghai Jiao Tong University −1.876 109 69

Expected values of top 10% cited papers for Tsinghua and Zhejiang University.

Expected values top 10% non-top total p
Tsinghua 2,449.01 17,452.99 19,902.00
Zhejiang 2,892.99 20,617.01 23,510.00
5,356.09 38,055.91 43,412.00

Observed values of top 10% cited papers for Tsinghua and Zhejiang University during the period 2015–2019 (fractional counting, all sciences).

Observed values top 10% non-top total p
Tsinghua 2,738 17,164 19,902
Zhejiang 2,604 20,906 23,510
5,342 38,070 43,412

Top-20 universities in each of the three clusters. (See for a full list in the Annexes.)

top group (top 20) z middle group (top 20) z bottom group (top 20) z
1 Tsinghua Univ. 11.005 Zhejiang Univ. 3.800 Hubei Univ. −0.498
2 Hunan Univ. 10.193 Harbin Institute of Technology 3.226 Northwest Univ. −1.609
3 Hong Kong Univ. of Science and Technology 6.566 Huazhong Univ. of Science and Technology 2.907 South China Agricultural Univ. −1.631
4 Univ. of Science and Technology of China 6.482 Peking Univ. 2.689 Zhejiang Univ. of Technology −1.676
5 City Univ. of Hong Kong 6.454 Nanjing Univ. 1.973 China Univ. of Mining and Technology −1.678
6 Shandong Univ. of Science and Technology 6.444 Xiamen Univ. 1.954 Nanjing Univ. of Aeronautics and Astronautics −1.816
7 Hong Kong Polytechnic Univ. 6.406 Wuhan Univ. 1.765 Second Military Medical Univ. −1.844
8 South China Univ. of Technology 6.049 Tianjin Univ. 1.738 Zhejiang Sci-Tech Univ. −1.886
9 Chinese Univ. of Hong Kong 5.993 Dalian Univ. of Technology 1.553 Taiyuan Univ. of Technology −1.934
10 Nankai Univ. 4.860 East China Normal Univ. 1.333 Shanghai Univ. of Traditional Chinese Medicine −1.950
11 Univ. of Hong Kong 4.418 Soochow Univ. 1.310 Qingdao Univ. of Science and Technology −1.961
12 Shenzhen Univ. 4.089 Univ. of Electronic Science and Technology of China 1.154 Nanjing Univ. of Chinese Medicine −1.981
13 Qufu Normal Univ. 3.724 Nanjing Univ. of Science and Technology 1.140 Harbin Engineering Univ. −2.033
14 Fuzhou Univ. 3.487 Southwest Jiaotong Univ. 1.088 Lanzhou Univ. −2.110
15 Beihang Univ. 3.432 China Univ. of Geosciences 1.035 Northeast Normal Univ. −2.128
16 Univ. of the Chinese Academy of Sciences 3.354 Huazhong Agricultural Univ. 1.030 China Jiliang Univ. −2.130
17 Central China Normal Univ. 3.090 Tongji Univ. 0.954 Jiangnan Univ. −2.163
18 Univ. of Macau 3.046 Univ. of Science and Technology Beijing 0.932 Jinan Univ. −2.227
19 Northwestern Polytechnical Univ. 2.915 China Agricultural Univ. 0.704 Hangzhou Normal Univ. −2.245
20 Wuhan Univ. of Technology 2.914 Beijing Institute of Technology 0.587 Southern Medical Univ. −2.263

Descriptive statistics of the comparison among 197 American and 205 Chinese universities.

Row Labels low middle High Isolates Grand Total
China 116 67 21 1 205
USA 36 60 99 2 197
Grand Total 152 127 120 402

Highest ranked universities in top-groups in the combined American-Chinese set of 402 universities in LR 2020 with their respective z-scores.

rank Top 20 Chinese universities z Top 20 American Universities z
1 Tsinghua Univ. 11.005 Harvard Univ. 36.632
2 Hunan Univ. 10.193 Stanford Univ. 26.028
3 Hong Kong Univ. of Science and Technology 6.566 Massachusetts Institute of Technology 24.504
4 Univ. of Science and Technology of China 6.482 Univ. of California, Berkeley 20.319
5 City Univ. of Hong Kong 6.454 Yale Univ. 16.576
6 Shandong Univ. of Science and Technology 6.444 Univ. of California, San Francisco 16.524
7 Hong Kong Polytechnic Univ. 6.406 Princeton Univ. 16.522
8 South China Univ. of Technology 6.049 Columbia Univ. 16.348
9 Chinese Univ. of Hong Kong 5.993 Univ. of California, San Diego 16.061
10 Nankai Univ. 4.860 Univ. of Pennsylvania 15.975
11 Univ. of Hong Kong 4.418 Cornell Univ. 14.934
12 Shenzhen Univ. 4.089 Univ. of Washington, Seattle 14.620
13 Qufu Normal Univ. 3.724 Johns Hopkins Univ. 14.540
14 Fuzhou Univ. 3.487 Northwestern Univ. 14.488
15 Central China Normal Univ. 3.090 Univ. of California, Los Angeles 14.450
16 Univ. of Macau 3.046 Univ. of Michigan 13.951
17 Wuhan Univ. of Technology 2.914 California Institute of Technology 13.695
18 Southern Univ. of Science and Technology 2.332 Univ. of Chicago 13.561
19 GuangZhou Univ. 2.120 Duke Univ. 12.605
20 Hong Kong Baptist Univ. 2.094 Washington Univ. in St. Louis 12.123

Alsmadi, I., & Taylor, Z. (2018). Examining university ranking metrics: Articulating issues of size and web dependency. In Proceedings of the 2018 International Conference on Computing and Big Data (ICCBD ’18). Association for Computing Machinery, New York, NY, USA, 73–77. doi: https://doi.org/10.1145/3277104.3277111 AlsmadiI. TaylorZ. 2018 Examining university ranking metrics: Articulating issues of size and web dependency In Proceedings of the 2018 International Conference on Computing and Big Data (ICCBD ’18) Association for Computing Machinery, New York, NY, USA 73 77 https://doi.org/10.1145/3277104.3277111 DOI öffnenSearch in Google Scholar

Anderson, J., Collins, P.M.D., Irvine, J., Isard, P.A., Martin, B.R., Narin, F., & Stevens, K. (1988). On-line approaches to measuring national scientific output: A cautionary tale. Science and Public Policy 15(3), 153–161. AndersonJ. CollinsP.M.D. IrvineJ. IsardP.A. MartinB.R. NarinF. StevensK. 1988 On-line approaches to measuring national scientific output: A cautionary tale Science and Public Policy 15 3 153 161 Search in Google Scholar

Bornmann, L., & Leydesdorff, L. (2013). The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000. Journal of Informetrics, 7(2), 286–291. BornmannL. LeydesdorffL. 2013 The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000 Journal of Informetrics 7 2 286 291 10.1016/j.joi.2012.12.003 Search in Google Scholar

Bornmann, L., & Mutz, R. (2011). Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics, 5(1), 228–230. BornmannL. MutzR. 2011 Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization Journal of Informetrics 5 1 228 230 10.1016/j.joi.2010.10.009 Search in Google Scholar

Bourdieu, P. (1998). The state nobility: Elite schools in the field of power. Stanford University Press. BourdieuP. 1998 The state nobility: Elite schools in the field of power Stanford University Press Search in Google Scholar

Bowman, N.A., & Bastedo, M.N. (2011). Anchoring effects in world university rankings: Exploring biases in reputation scores. Higher Education, 61(4), 431–444. BowmanN.A. BastedoM.N. 2011 Anchoring effects in world university rankings: Exploring biases in reputation scores Higher Education 61 4 431 444 10.1007/s10734-010-9339-1 Search in Google Scholar

Brewer, D.J., Gates, S.M., & Goldman, C.A. (2001). In Pursuit of Prestige: Strategy and Competition in U.S. Higher Education. Piscataway, NJ: Transaction Publishers, Rutgers University. BrewerD.J. GatesS.M. GoldmanC.A. 2001 In Pursuit of Prestige: Strategy and Competition in U.S. Higher Education Piscataway, NJ Transaction Publishers, Rutgers University Search in Google Scholar

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum. CohenJ. 1988 Statistical Power Analysis for the Behavioral Sciences 2nd ed. Hillsdale, NJ Lawrence Erlbaum Search in Google Scholar

Colliander, C., & Ahlgren, P. (2011). The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments. Journal of Informetrics, 5(1), 101–113. CollianderC. AhlgrenP. 2011 The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments Journal of Informetrics 5 1 101 113 10.1016/j.joi.2010.09.003 Search in Google Scholar

Cumming, G. (2013). Understanding the New Statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY/Hove, East Sussex, UK: Routledge. CummingG. 2013 Understanding the New Statistics: Effect sizes, confidence intervals, and meta-analysis New York, NY/Hove, East Sussex, UK Routledge 10.4324/9780203807002 Search in Google Scholar

DiMaggio, P.J., & Powell, W.W. (1983). The Iron Cage Revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological Review, 48, 147–160. DiMaggioP.J. PowellW.W. 1983 The Iron Cage Revisited: Institutional isomorphism and collective rationality in organizational fields American Sociological Review 48 147 160 10.2307/j.ctv1f886rp.7 Search in Google Scholar

Garfield, E. (1971). The mystery of the transposed journal lists—wherein Bradford’s Law of Scattering is generalized according to Garfield’s Law of Concentration. Current Contents, 3(33), 5–6. GarfieldE. 1971 The mystery of the transposed journal lists—wherein Bradford’s Law of Scattering is generalized according to Garfield’s Law of Concentration Current Contents 3 33 5 6 Search in Google Scholar

Gingras, Y. (2016). Bibliometrics and research evaluation: An overview. Cambridge MA: MIT Press. GingrasY. 2016 Bibliometrics and research evaluation: An overview Cambridge MA MIT Press 10.7551/mitpress/10719.001.0001 Search in Google Scholar

Halffman, W., & Leydesdorff, L. (2010). Is Inequality Among Universities Increasing? Gini Coefficients and the Elusive Rise of Elite Universities. Minerva, 48(1), 55–72. HalffmanW. LeydesdorffL. 2010 Is Inequality Among Universities Increasing? Gini Coefficients and the Elusive Rise of Elite Universities Minerva 48 1 55 72 10.1007/s11024-010-9141-3285052520401157 Search in Google Scholar

Heilbron, J., Boncourt, T., & Sorá, G. (2018). Introduction: The Social and Human Sciences in Global Power Relations. In The Social and Human Sciences in Global Power Relations (pp. 1–25). Pal. grave Macmillan, Cham. HeilbronJ. BoncourtT. SoráG. 2018 Introduction: The Social and Human Sciences in Global Power Relations In The Social and Human Sciences in Global Power Relations 1 25 Pal. grave Macmillan, Cham. 10.1007/978-3-319-73299-2_1 Search in Google Scholar

Leydesdorff, L., & Wagner, C. (2009). Macro-level indicators of the relations between research funding and research output. Journal of Informetrics, 3(4), 353–362. doi: 10.1016/j.joi.2009.05.005 LeydesdorffL. WagnerC. 2009 Macro-level indicators of the relations between research funding and research output Journal of Informetrics 3 4 353 362 10.1016/j.joi.2009.05.005 DOI öffnenSearch in Google Scholar

Leydesdorff, L., Bornmann, L., & Mingers, J. (2019). Statistical significance and effect sizes of differences among research universities at the level of nations and worldwide based on the leiden rankings. Journal of the Association for Information Science and Technology, 70(5), 509–525. LeydesdorffL. BornmannL. MingersJ. 2019 Statistical significance and effect sizes of differences among research universities at the level of nations and worldwide based on the leiden rankings Journal of the Association for Information Science and Technology 70 5 509 525 10.1002/asi.24130 Search in Google Scholar

Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the Tables on Citation Analysis One More Time: Principles for Comparing Sets of Documents. Journal of the American Society for Information Science and Technology, 62(7), 1370–1381. LeydesdorffL. BornmannL. MutzR. OpthofT. 2011 Turning the Tables on Citation Analysis One More Time: Principles for Comparing Sets of Documents Journal of the American Society for Information Science and Technology 62 7 1370 1381 10.1002/asi.21534 Search in Google Scholar

Leydesdorff, L., Wouters, P., & Bornmann, L. (2016). Professional and citizen bibliometrics: Complementarities and ambivalences in the development and use of indicators-a state-of-theart report. Scientometrics, 109(3), 2129–2150. LeydesdorffL. WoutersP. BornmannL. 2016 Professional and citizen bibliometrics: Complementarities and ambivalences in the development and use of indicators-a state-of-theart report Scientometrics 109 3 2129 2150 10.1007/s11192-016-2150-8512404427942086 Search in Google Scholar

McAllister, P.R., Narin, F., & Corrigan, J.G. (1983). Programmatic Evaluation and Comparison Based on Standardized Citation Scores. IEEE Transactions on Engineering Management, 30(4), 205–211. McAllisterP.R. NarinF. CorriganJ.G. 1983 Programmatic Evaluation and Comparison Based on Standardized Citation Scores IEEE Transactions on Engineering Management 30 4 205 211 10.1109/TEM.1983.6448622 Search in Google Scholar

Peters, M.A. (2017). Global university rankings: Metrics, performance, governance. Educational Philosophy & Theory, 51(1), 1–9. PetersM.A. 2017 Global university rankings: Metrics, performance, governance Educational Philosophy & Theory 51 1 1 9 10.1080/00131857.2017.1381472 Search in Google Scholar

Schneider, J. (2015). Null hypothesis significance tests. A mix-up of two different theories: The basis for widespread confusion and numerous misinterpretations. Scientometrics, 102(1), 411–432. SchneiderJ. 2015 Null hypothesis significance tests. A mix-up of two different theories: The basis for widespread confusion and numerous misinterpretations Scientometrics 102 1 411 432 10.1007/s11192-014-1251-5 Search in Google Scholar

Sheskin, D.J. (2011). Handbook of Parametric and Nonparametric Statistical Procedures (5th Edition). Boca Raton, FL: Chapman & Hall/CRC. SheskinD.J. 2011 Handbook of Parametric and Nonparametric Statistical Procedures 5th Edition Boca Raton, FL Chapman & Hall/CRC Search in Google Scholar

Shin, J.C., Toutkoushian, R.K., & Teichler, U. (2011). University Rankings: Theoretical Basis, Methodology and Impacts on Global Higher Education. Dordrecht: Springer. ShinJ.C. ToutkoushianR.K. TeichlerU. 2011 University Rankings: Theoretical Basis, Methodology and Impacts on Global Higher Education Dordrecht Springer 10.1007/978-94-007-1116-7 Search in Google Scholar

Sivertsen, G. (2016). Patterns of internationalization and criteria for research assessment in the social sciences and humanities. Scientometrics, 107(2), 357–368. SivertsenG. 2016 Patterns of internationalization and criteria for research assessment in the social sciences and humanities Scientometrics 107 2 357 368 10.1007/s11192-016-1845-1483382227122643 Search in Google Scholar

Sivertsen, G., Rousseau, R., & Zhang, L. (2019). Measuring scientific contributions with modified fractional counting. Journal of Informetrics, 13(2), 679–694. SivertsenG. RousseauR. ZhangL. 2019 Measuring scientific contributions with modified fractional counting Journal of Informetrics 13 2 679 694 10.1016/j.joi.2019.03.010 Search in Google Scholar

Tijssen, R.J.W., Visser, M.S., & van Leeuwen, T.N. (2002). Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? Scientometrics, 54(3), 381–397. TijssenR.J.W. VisserM.S. van LeeuwenT.N. 2002 Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? Scientometrics 54 3 381 397 10.1023/A:1016082432660 Search in Google Scholar

van Raan, A.F.J., van Leeuwen, T.N., Visser, M.S., van Eck, N.J., & Waltman, L. (2010). Rivals for the crown: Reply to Opthof and Leydesdorff. Journal of Informetrics, 4(3), 431–435. van RaanA.F.J. van LeeuwenT.N. VisserM.S. van EckN.J. WaltmanL. 2010 Rivals for the crown: Reply to Opthof and Leydesdorff Journal of Informetrics 4 3 431 435 10.1016/j.joi.2010.03.008 Search in Google Scholar

Waltman, L. (2016). Conceptual difficulties in the use of statistical inference in citation analysis. Journal of Informetrics, 4(10), 1249–1252. WaltmanL. 2016 Conceptual difficulties in the use of statistical inference in citation analysis Journal of Informetrics 4 10 1249 1252 10.1016/j.joi.2016.09.012 Search in Google Scholar

Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E., Tijssen, R.J., Eck, N.J., . . . Wouters, P. (2012). The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, 63(12), 2419–2432. WaltmanL. Calero-MedinaC. KostenJ. NoyonsE. TijssenR.J. EckN.J. WoutersP. 2012 The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation Journal of the American Society for Information Science and Technology 63 12 2419 2432 10.1002/asi.22708 Search in Google Scholar

Waltman, L., Van Eck, N.J., Van Leeuwen, T.N., Visser, M.S., & Van Raan, A.F.J. (2011). Towards a New Crown Indicator: Some Theoretical Considerations. Journal of Informetrics, 5(1), 37–47. WaltmanL. Van EckN.J. Van LeeuwenT.N. VisserM.S. Van RaanA.F.J. 2011 Towards a New Crown Indicator: Some Theoretical Considerations Journal of Informetrics 5 1 37 47 10.1016/j.joi.2010.08.001 Search in Google Scholar

Wasserstein, R.L., & Lazar, N.A. (2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. WassersteinR.L. LazarN.A. 2016 The ASA’s statement on p-values: Context, process, and purpose The American Statistician 70 2 129 133 10.1080/00031305.2016.1154108 Search in Google Scholar

Whitley, R.D. (1984). The Intellectual and Social Organization of the Sciences. Oxford: Oxford University Press. WhitleyR.D. 1984 The Intellectual and Social Organization of the Sciences Oxford Oxford University Press Search in Google Scholar

Empfohlene Artikel von Trend MD

Planen Sie Ihre Fernkonferenz mit Scienceendo