1. bookVolume 2 (2017): Issue 1 (February 2017)
Journal Details
License
Format
Journal
First Published
30 Mar 2017
Publication timeframe
4 times per year
Languages
English
access type Open Access

Patent Citations Analysis and Its Value in Research Evaluation: A Review and a New Approach to Map Technology-relevant Research

Published Online: 18 Feb 2017
Page range: 13 - 50
Received: 23 Nov 2016
Accepted: 03 Dec 2016
Journal Details
License
Format
Journal
First Published
30 Mar 2017
Publication timeframe
4 times per year
Languages
English

Ton (Anthony F.J.) van Raan is Professor of Quantitative Studies of Science. Founder and until 2010 Director of the Centre for Science and Technology Studies (CWTS), Leiden University, the Netherlands. After his retirement as Director of CWTS, he remained research professor. He studied mathematics, physics, and astronomy at Utrecht University. PhD in Physics, Utrecht (1973). Post-doctoral fellow (1973–1977) at the University of Bielefeld (Germany), visiting scientist in the US, UK, and France. Work in atomic physics, laser-physics, astrophysics, and in science policy and research management. From 1977 senior research fellow physics in Leiden, in 1985 “field switch” from physics to science and technology studies, 1991 Full Professor. His research focuses on design, construction, and application of quantitative indicators on important aspects of science and technology. Contract research for the government of the Netherlands, other European Union member states, the European Commission, national and international research organizations, and the business sector. Main research interests: application of bibliometric indicators in research evaluation, science as a “self-organizing” cognitive ecosystem, statistical properties of bibliometric indicators, ranking and benchmarking of universities, mapping of science, and scaling of universities and cities. In 1995 he received together with the American sociologist Robert K. Merton, the Derek de Solla Price Award, the highest international award in the field of quantitative studies of science. He published (as author and co-author) around 30 articles in physics and 200 in science and technology studies. He is editor of the Handbook of Quantitative Studies of Science and Technology and member of the editorial board of the international journals Scientometrics, Research Evaluation, and Journal of Informetrics. Prof. van Raan set up a small spin-off company for advice on research evaluation and science policy issues. On the occasion of his retirement as CWTS Director he was awarded by the Queen of the Netherlands with the royal distinction of Knight in the Order of the Dutch Lion. More info: http://www.cwts.nl/tvr/.

Introduction

Given the increasing emphasis on the “societal impact” of scientific research it is important to analyze the role of patents in the monitoring and evaluation of research groups and programs (see for instance Chowdhury, Koya, & Philipson (2016) in the context the UK’s Research Excellence Framework 2014) and particularly the analysis of the link between patents and research. Obviously, in this way a specific part of societal impact of research is analyzed, namely technological, possibly followed by economic relevance.

This paper consists of two parts. We first present in Section 2 a review of the research on patent analysis, particularly on characteristics and context of patent citations to scientific publications (scientific non-patent references, SNPRs), on the problem of time lags between scientific achievements and inventions, and on the economic value of patents. Section 3 reviews the research on the role of academic scientific work in technological developments, in particular the role of universities as sources of technological knowledge and the importance of inventor-author relations. We conclude the review part of this paper with Section 4 by an overview of the recent research on mapping and network analysis of the science and technology interface and of the monitoring of technological progress in interaction with science. After these review sections, we discuss in Section 5 a novel approach to identify publications with technological importance by analyzing and mapping the citation links and conceptual relations of SNPRs with other publications. Finally, in Section 6 the potential of this approach for evaluation practices, particularly the assessment of the technological relevance of research is addressed.

Review of Patent Citation Analysis Research
Patents and Their Citations: Basic Properties

Patents are documents with a legal status to describe and claim technological innovations. Figure 1 presents as an example the front page of a recent patent (patent publication year is 2014) of a membrane fuel cell. In this front page we find the data related to the patent publication date, the patent number, the international patent classification (IPC) codes which indicate the relevant fields of technology, and the names and affiliations of the inventors. Figure 2 shows the first page of the list of claims.

Figure 1

Front page of patent WO2014/009721A1 as an example.

Figure 2

Claims of patent WO2014/009721A1; the fi rst page of the claims section is shown as an example.

Similar to scientific publications, also in patent documents references are given. These references mainly concern earlier patents (patent-to-patent citations) in order to prove novelty in view of the existing technological development (“prior art”) and, generally to a lesser extent, to non-patent items (non-patent references, NPRs), particularly scientific publications (scientific non-patent references, SNPRs). References in scientific publications are the sole responsibility of the authors. References in patents, however, can be given by both the inventors as well as by the patent examiners. Figures 3 and 4 show the “international search report” part of the patent document in which the references to earlier patents as well as to scientific publications are given. One of the cited publications is the paper of Zarrin et al. (2011) on a functionalized graphene oxide nanocomposite membrane for low humidity and high temperature proton exchange membrane fuel cells, an issue strongly related to the invention described in the patent.

Figure 3

International search report of patent WO2014/009721A1, page 1. The SNPRs are indicated with a light blue box, and the references to earlier patents with a red box.

Figure 4

International search report of patent WO2014/009721A1, page 2. The SNPRs are indicated with a light blue box, and the references to earlier patents with a red box.

Characteristics of Patent Citations to Scientific Literature

Pioneering work on patent citations to scientific literature (SNPRs) was done by Narin and colleagues (Carpenter, Cooper, & Narin, 1980; Carpenter & Narin, 1983; Narin & Noma, 1985; Narin, Rosen, & Olivastro, 1989). The number of SNPRs was considered to be a measure of the “science intensity” of technological fields. In the follow-up work of Narin it was found that about three quarters of the papers cited by US industry patents were public science, authored at academic, governmental, and other public institutions; only about a third was authored by industrial scientists. The SNPRs showed a strong national component. The cited US papers are from the mainstream of modern science; quite basic, in influential journals, often authored at top research universities and laboratories, relatively recent, and heavily supported by NIH, NSF

NIH refers to the US National Institutes of Health and NSF refers to the US National Science Foundation.

, and other public agencies (Narin, Hamilton, & Olivastro, 1997). This work shows the importance of a well-developed public research system for the technological development and the economy of a country.

After the pioneering work of Narin the number of studies on patent citations to scientific literature rapidly increased. We mention the early work at ISI

ISI: Institute for Scientific Information, the original producer of the Science Citation Index (SCI). The Web of Science (WoS) is the successor of the Science Citation Index, until recently part of Thomson Reuters, now owned by Clarivate Analytics.

to match patent data to a bibliometric model (Coward & Franklin, 1989); the early Leiden (CWTS) work on patent citations (van Vianen, Moed, & van Raan, 1990) where it was found that over half of the NPRs in Dutch patents in the first half of the 1980s were journal document citations, mostly SCI-(WoS) covered journals, and the other references were mainly books, abstracting services, and meeting abstracts. The Fraunhofer Institute in Karlsruhe started at the beginning of the 1990s a research program on the dynamics of science-based innovation with patent analysis studies (Grupp, 1992; Schmoch, 1993). CWTS continued its work by Tijssen and colleagues with patent-citation analysis focused on global and domestic utilization of industrial relevant science (Tijssen, Buter, & van Leeuwen, 2000; Tijssen, 2001).

Particularly the Leuven group developed a broad research program on patent analysis and the relation between science and technology. For instance Verbeek et al. (2002) found a skewed distribution of NPRs in patents, with a majority of patents containing no references, and only a small number with numerous references. The majority of NPRs are journal references and thus SNPRs (Callaert et al., 2006). Furthermore it was found that the SNPRs are published in a small group of scientific journals. Van Looy et al. (2006) found that national technological performance positively correlates with the scientific strength of a country, particularly when this strength is present in a wide variety of companies as well as knowledge institutes. In new and emerging fields of technology the number of SNPRs in patents is higher, which indicates that new technological developments are generally more science intensive (van Looy, Magerman, & Debackere, 2007).

The study of patent citations and particularly patent citations to scientific literature is not a piece of cake. SNPR information is often incomplete and not unified, and may contain multiple distinct references pointing to the same scientific publication. It requires, after the necessary “data cleaning” work, a careful merging of a patent database such as PATSTAT with the WoS or Scopus

Scopus is the citation index published by Elsevier.

in a bibliometrically advanced way in order to match with high precision SNPRs with publications covered by the WoS or Scopus. Different data analytical techniques are necessary to successfully perform such patent-publication data matching. For instance, Magerman, van Looy, & Song (2010) discuss text mining techniques to detect similarity between patents and scientific publications. In their recent studies the Leuven researchers remark that caution is needed in the interpretation of SNPRs as indicators of direct knowledge flows from published research to patented technology. It matters whether the patents with SNPRs originate from knowledge (R&D performing) institutions such as universities and other public research institutes, or from companies. Also there are substantial differences between countries, most probably related to the scientific strength of a country. Moreover, there are also differences between patent systems regarding the use of SNPRs: the United States Patent and Trademark Office (USPTO) requires more SNPRs than the European Patent Office (EPO) or the World Intellectual Property Organization (WIPO) (Callaert, Grouwels, & van Looy, 2012; Callaert et al., 2014).

Patent citations to scientific literature can also provide insight into the globalization of technological developments. Ribeiro et al. (2014) analyze 167,315 USPTO patents granted in 2009 and the papers cited by these patents to identify the “scientific footprints of technology” that cross national boundaries, and particularly how multinational enterprises interact globally with universities and other firms.

Context of Patent Citations to Scientific Publications

Not every SNPR is a central reference to underlying research. SNPRs can also be meant as general background information and not necessarily as a source of inspiration. In quite a substantial number of cases the inventors regard the SNPRs as less important or even trivial (Callaert, Pellens, & van Looy, 2014). This latter situation is related to the differences between inventor- and examiner-given SNPRs. Examiners play an important role in adding citations to patents (63% for an average patent), there is a strong firm-specific (mostly technology-field specific) variation, and the highest proportion of citations added by examiners is found for foreign applicants to USPTO (Criscuolo & Verspagen, 2008; Alcácer, Gittelman, & Sampat, 2009). Examiner-given references, however, are not without problems. Wada (2016) analyzes obstacles to prior-art searching by examiners and found evidence of a negative effect of geographical distance on the probability to capture prior patents. Although this relates to problems in referencing to earlier patents, similar difficulties may also arise in proper referencing to scientific literature. Furthermore, the number of SNPRs in patents will depend on the stage of development of a technological field. A number of studies deal with patent citation analysis in developing and emerging fields, for instance nanotechnology (Hu et al., 2007; Meyer, 2000, 2001) and genetic engineering research (Lo, 2010). A rapidly developing technological field will generally be more based on recent scientific knowledge than a mature field.

From the above it is clear that the number of SNPRs in patents, and with that the probability that a publication may be used as an SNPR, depends on several different factors: the role of inventors versus examiners; characteristics of the patent office; characteristics of firms and of technology fields. Furthermore, the distribution of citations in patents to non-patent literature (of which SNPRs are the major part) is skew (see for instance Squicciarini et al., 2013, pp. 26–30). The number of SNPRs is also influenced by the large differences in the economic values of patents (see Section 2.5). Therefore it is sensible to focus specifically on the important patents, particularly patents in a patent family

Patent families are datasets consisting of patent publications that are equivalent and relate to one and the same invention.

with at least one US patent. As remarked above, with a US patent in a patent family many more SNPRs are obtained. It is a legal requirement in the US to send in as much possible relevant information in the patent-application procedure. Nevertheless, the real importance of SNPRs can only be determined by analyzing the patent’s main text and/or by querying the inventors. A general observation is that SNPRs indeed form a bridge between science and technology, but more in a broader sense, i.e. at a macro-level such as the “science intensity” of technological fields or the science-technology interaction at the level of countries.

Time Lags between Scientific Breakthroughs and Inventions

As always, there is the time dimension. In the relation between science and technology particularly the speed of transfer of scientific knowledge into the patenting process is important. This time lag, mostly defined as the time lapse between the publication year of a paper and the year this paper is cited in a patent, may differ substantially between the various fields of technology. This time lapse defines the age of the SNPRs. In emerging and developing fields this time lag is mostly relatively short. Finardi (2011) finds that for nanotechnology the time lag is between three and four years. Some authors find time lags of more than 20 years, see for instance the study on the technological impact of library science research (Halevi and Moed, 2012). Information about the age of SNPRs is important in order to know when scientific results are used in a technological innovation. Mehta, Rysman, and Simcoe (2010) discuss the problems involved in the determination of the time lag and the precise definition of the age of SNPRs.

Directly related to the time lags between scientific work and technological developments is the more fundamental problem to identify the “real” scientific basis of a technological innovation. Ground-breaking work was done in the 1960s with the Hindsight (Sherwin & Isenson, 1967; Isenson, 1969) and the TRACES studies (IIT, 1968, 1969; Heilbron, 1972) in the US

See for instance https://en.wikipedia.org/wiki/Project_Hindsight, https://marchofscience.wordpress.com/2013/03/12/project-hindsight-and-project-traces-2/, and http://scimaps.org/mapdetail/tracing_of_key_event_4.

. Quite often it takes more than a generation before fundamental scientific discoveries can be used in new technologies. Grant, Green, and Mason (2003) found a time lag of about 20 years between clinical advances in neonatal intensive care and the underpinning basic research. The SNPRs may represent important recent scientific research but this research on its turn may be based on even more important, earlier breakthrough work, not cited in the patent but perhaps cited in the SNPRs. We come back to this important issue in Section 5. Another, more technical time-related problem is the delay between patent application and patent publication, mostly 18 months. Within this period the patent application is not accessible for analysis. Also a substantial amount of patent applications are never published, so these inventions are “invisible” but they may influence the development of a technological field.

Economic Value of Patents

In order to make patent analysis a valuable part of monitoring and evaluating research, we need to know how the economic value of patents can be assessed. The reason is clear: just as in the case of publications, also patents show a wide variety of impact. Only a relatively small amount of patents represents important technological breakthroughs (Albert et al., 1991). Therefore, specific patent indicators are necessary to assess the importance of patents (Carpenter, Narin, & Woolf, 1981; Hall, Jaffe, & Trajtenberg, 2005). In analogy to publications, patent-to-patent citations are often regarded as an indicator of patent quality (Trajtenberg, 1990).

Patent-to-patent citations provide a first indication of the importance of the cited patents, particularly if they are highly cited and belong to, for instance, the top 10% cited patents in their field. From the perspective of the cited patent, the citing patents (which are later in time than the cited patent) provide the “forward citations.” The “backward citations” are the citations in a patent to earlier patents or to non-patent literature such as the SNPRs. Harhoff et al. (1999) obtained through a survey private economic value estimates on nearly 1,000 US and German inventions. These authors found that patents renewed to full-term (which is the maximum duration of the patent protection, mostly 20 years) were significantly more highly cited than patents allowed to expire before their full term. The higher an invention’s economic value estimate was, the more the patent was subsequently cited.

Patent-to-patent citation analysis is also used to trace the evolution of new fields of technology. Bruck et al. (2016) show that laser-inkjet printer technology started from the merging of two existing technologies: sequential printing and static image production. Sternitzke (2010) investigated both radical as well as incremental pharmaceutical innovations. He finds that public sector scientific knowledge is important for all innovations. But radical innovations are based on a higher degree of basic research and on a significantly higher share of own prior scientific research than incremental innovations. Arts, Appio, and van Looy (2012) show that biotechnology patents representing important technological innovations have a high number of, particularly recent, citations to scientific publications, and these “radical” patents connect patent subclasses which were so far unconnected. The authors caution that these indicators are ex ante. In line with the earlier research discussed in the beginning of this section, the authors state that also ex post indicators such as forward citations (patent-to-patent citations) are necessary to identify breakthrough innovations more accurately.

Squicciarini, Dernis, and Crisculo (2013) assessed patent quality on the basis of a combination of several indicators of the economic value of patents: number of backward citations particularly SNPRs, patent claims which determine the boundaries of the exclusive rights of a patent owner, number of forward citations (up to five years after patent publication), patent renewal, and patent-family size. The authors develop a generality, originality, and radicalness index for patents on the basis of differences in IPC

IPC: International Patent Classification, for more information see http://www.wipo.int/classifications/ipc/en/.

patent classes between cited versus citing patents. Earlier work with partly similar approaches can be found in a preliminary report (PATVAL study) for the European Commission (European Commission, 2005). Benson and Magee (2015) find that patents contain significant information relevant to the quantitative assessment of technological improvement rates. In particular, these authors show that the importance of patents, the recency of patents, and the immediacy of patents are all strongly correlated with increases in the rate of performance improvement in the technology field of interest. These indicators appear to have good predictive power for more than 10 years into the future. A new measurement of technological novelty is developed by Verhoeven, Bakker, and Veugelers (2016). These authors characterize inventions ex ante along two dimensions of technological novelty: novelty in recombination of different technology fields, and novelty in technological as well as scientific knowledge origins. They use patent classification and citation information to operationalize the two dimensions.

More research is necessary on the economic value of patents. But also here, like in the case of scientific publications, it may take a long time before the true value of a patent for the socio-economic progress of our society becomes apparent. This means that in the determination of the top 10% patents a relatively long citation window, but at least several windows of different lengths are necessary. Moreover, patent mapping and network analysis are increasingly used to assess the impact of patents. We will discuss this further in Section 4 where we review recent research on the mapping of technological development and of the science-technology interface.

Summary of the Findings

On the basis of our review in this section we draw the following conclusions:

Majority of NPRs are SNPRs;

Majority of the SNPRs are published in a relatively small group of journals;

SNPRs show a strong national component;

SNPRs show a strong public science component;

The distribution of SNPRs over patents is skewed;

Patents in emerging fields have more SNPRs;

SNPRs are not a direct indicator of knowledge flows;

Patent office differences: USPTO requires more SNPRs than EPO;

SNPRs are not necessarily central references to underlying research;

There are inventor- and examiner-given SNPRs, and examiners play an important role;

Number of SNPRs is field- and developmental stage dependent;

Time lag between the SNPR publication year and citation in a patent can be 3–20 years;

Earlier breakthrough work not cited in patent but perhaps cited in SNPR;

Real importance of SNPRs can only be found by querying the inventors;

Only a small fraction of patent-relevant publications are SNPRs;

For university-industry collaboration papers the number of SNPRs is much higher;

Only a small amount of patents represents important, “radical” technological breakthroughs;

Patent-to-patent citations are regarded as an indicator of patent quality if patents are highly cited, particularly the top 10% patents;

Patents renewed to full-term are significantly more highly cited than patents allowed to expire;

The higher an invention’s economic value estimate, the more the patent was subsequently cited;

Radical innovations are based on a higher degree of basic research and on a significantly higher share of own prior research as compared to incremental innovations;

Radical patents connect patent classes so far unconnected;

Patent quality assessment can best be based on a combination of indicators of economic value of patents: number of SNPRs; patent claims that determine the boundaries of the exclusive rights; number of forward citations (up to five years after patent publication); patent renewal; patent-family size;

Generality, originality, and radicalness index for patents can be based on differences in IPC patent classes between cited versus citing patents;

The discussed work shows the importance of a combined patent-and publication-citation index system that covers both patents and publications as sources, enabling both patent-to-patent and patent-to-publication citation analyses. For instance, patent impact distribution functions based on patent-to-patent citations can be established in order to determine the top 10% patents, an important indicator of patent value. Furthermore, with a combined patent-and publication-citation index system it will be possible to measure more accurately, and on a larger scale, what fraction of all WoS covered publications is an SNPR, what the differences are between the fields of science, the changes in the course of time, and whether these publications are an SNPR in a top 10% patent and whether highly cited papers are more likely to become an SNPR.

Review of the Role of Universities and Researchers in Technology
Universities as Sources of Technological Knowledge

We already mentioned the pioneering work of Narin and colleagues on the important role of the public research system, particularly universities, in technological development. Recent work supports these early findings. In order to analyze the role of universities in technology-relevant knowledge production, Hung et al. (2015) study growth trajectories of the cumulative patent citations to scientific publications produced by individual universities. Their results indicate that not all top 300 research universities in the world perform well in knowledge utilization for patented inventions, and that university-industry collaboration plays an important role. In studies on the role of universities in technological development patent citations to scientific literature as well as patent citation to earlier patents (patent-to-patent citations) are analyzed. For instance, Guerzoni et al. (2014) investigate the creation of a new industry on the basis of funding sources of university patents. The authors argue that patent citations provide insight into the originality of patents. With data on patented cancer research they find that university researchers have a higher propensity to generate more original patents when they are partly funded by their own university in contrast to university researchers funded either by industry or other non-university organizations. Other research on funding is the study of Chai and Shih (2016) in which these authors focus on the transformation of new scientific knowledge from academic research into commercialized products of private firms. They assess the effect of funded partnerships between universities and private companies on the innovative performance of the participating firms. The authors compare patent counts, publication counts, and proportion of cross-institutional publications between funded and unfunded firms. The effects appear to differ depending on the type of firm, for instance small and medium-sized firms, or younger firms.

Mowery and Ziedonis (2015) compare the localization of knowledge flows from university inventions through market contracts (licenses) and nonmarket contracts (spillovers) on the basis of patent citations. They find that knowledge flows through market transactions are more geographically localized than those through nonmarket spillovers. Leten, Landoni, and van Looy (2014) investigate the impact of universities on the technological performance of adjacent firms. The results show a positive effect of both university graduates and scientific publications on the technological performance of firms with, however, considerable industry differences. Positive effects for scientific research are only observed in the scienceintensive technologies such as the electrical and pharmaceutical industries. A difficulty in accounting the academic engagement and commercialization activities of researchers is the accurate quantification of these activities. Perkmann et al. (2015) combine university administrative records with data retrieved from external sources and surveys to quantify academic consulting, patenting, and academic entrepreneurship. They illustrate this approach with data for 10,000 scientists at the Imperial College London and find, with the exception of consulting, no significant differences between individuals involved in supported (university-recorded) and independent activity.

The time it takes for scientific papers to gain impact in related fields of technology is studied by Fukuzawa and Ida (2016). They analyze the citation linkages between articles and patents and find that the articles of leading Japanese scientists in the life and medical sciences reach on average in the fourth year after publication a peak in citations by subsequent papers; for citations given in patents it takes on average six years to reach a peak. Walter, Schmidt, & Walter (2016) investigate why academic entrepreneurs seek patents for spin-off technology in weak organizational regimes (the employee owns the inventions) and strong organizational regimes (the employer, i.e. the university or research organization, owns the inventions). They find that characteristics of the founding scientists (expert knowledge and entrepreneurial orientation) are important in weak but not in strong regimes. In contrast, organizational patenting norms are the main driver of patenting in strong but not in weak organizational regimes.

Inventor-author Relations

An important bridge between science and technology is built on the direct connections between scientists as inventors and as authors of publications. Packer and Webster (1996) described the emergence of a patenting culture in university science. The number of studies on inventor-author relations is, however, quite limited. One of the few early studies is the CWTS-Fraunhofer work on inventor-author relations in the application of lasers in medicine (Noyons et al., 1994). These authors found that inventors of patents with many SNPRs did not publish significantly more in science than inventors of patents with few SNPRs. The former did, however, use more basic scientific journals to publish their research work than the latter. It was also found that during the preparation of a patent application, co-inventors increase their co-activity in science, and companies and universities level up their co-operation. Related work are the studies of inventor-author self-citations in Dutch (mainly Philips) patents (Tijssen et al., 2000; Tijssen, 2001), inventor networks and universities (Balconi, Breschi, & Lissoni, 2004), and the study of the role of academic inventors in companies (Murray, 2004). Meyer (2005) compared the performance of inventor-authors in nanoscience with their “non-inventing” peers.

The central problem in inventor-author studies is the accurate identification of both the inventor as well as the author. An interesting method to identify inventor-author relations is a text-based approach as developed by Cassiman, Glenisson, & van Looy (2007). Here patents and publications were first matched by content-similarity, and then, for the highest ranked matches, a name matching was applied. However, for large scale studies the lack of unification of names and precise person identification in publication-as well as patent databases severely hampers the study of inventor-author relations. Therefore, most studies are still on a smaller scale, for instance a specific country, see Maraut and Martinez (2014) for inventor-author relations in Spain.

To our knowledge there are only two large-scale studies. The CWTS–Fraunhofer study on the development of nanoscience and nanotechnology in the EU countries (Noyons et al., 2003) is a very comprehensive inventor–author study. In this study over 15,000 inventor–authors combinations were identified with help of several text analysis techniques. Boyack and Klavans (2008) studied science–technology interaction on a large scale by identifying and validating a set of nearly 20,000 inventor–authors through matching of rare names obtained from paper and patent data. With rare names the probability to identify a specific person is considerably higher than in the case of common names. Magerman, van Looy, and Debackere (2015) investigate whether involvement in patenting hampers the dissemination of a scientist’s published research. The authors conducted a citation analysis of patentpaper pairs in biotechnology by using text-mining algorithms. In a dataset of 948,432 scientific publications and 88,248 EPO and USPTO patent documents, they identify 584 patent-paper pairs. Publications linked to a patent receive more citations than publications without a patent link. These findings show that researchers with patent-publication pairs develop a larger “scientific footprint” than colleagues without patent activity.

We conclude that inventor-author relations are an important indicator of science and technology (S&T) interaction, particularly between academia and industry. As discussed above, the precise identification of inventors and authors is still a major challenge. The big advantage of an as good as possible matching of inventors and authors is that more publications than only SNPRs relevant for a specific technological innovation can be found. This may also reveal the often more than just one developmental path in the course of time that has led to the innovation.

Analytical Requirements

The foregoing sections makes clear that the analysis of patent citations to scientific literature requires an advanced merging of a patent database (e.g. PATSTAT) and a scientific literature database (WoS or Scopus). This means that, technically, we can work from two perspectives: (1) taking patents as a starting point and identifying their SNPRs in the WoS or Scopus, or (2) taking the publications covered by WoS or Scopus as a starting point and find out whether they are cited in patents or not. Recent CWTS work shows examples of this first approach

As mentioned earlier, SNPR information is often incomplete and not unified, and may contain multiple distinct references pointing to the same scientific publication. The first approach is therefore considered to be more efficient as the SNPRs are classified into those that most probably might occur in a scientific database and those for which such an occurrence is highly unlikely or even impossible.

. First, the study on the discovery of introns

An intron (intragenic region) is a part of a DNA molecule within a gene but it is not used for decoding proteins. It is still not clear what the precise function of introns is. The more developed organisms are, the more introns they have in their DNA. Richard Robert and Philip Sharp received the Nobel Prize for Medicine in 1993 for the discovery of introns, see http://www.nobelprize.org/nobel_prizes/medicine/laureates/1993/press.html.

(Winnink, Tijssen, & van Raan, 2013) reveals that of the approximately 15,000 intron-related WoS publications in the period 1986–2001, only 175 are identified as an SNPR in 1,284 (1984–2012) intron-related patents covered by 677 patent families. Thus, around 1%, which means that 99% of the relevant publications does not “show up” in the patents relevant to the same topic. Looking from the other perspective, we find that 84% of the intron-related patents have no identified SNPR.

Second, in the Leiden Ranking

For more information see http://www.leidenranking.com/.

(the version used in the UMultiRank

For more information see http://www.umultirank.org/.

) there are for instance 24,156 Leiden WoS-covered publications in the period 2005–2012. Of these 24,156 publications, 641 are cited in patents (from the period 2005–2008) which means 2.7% of the total number of Leiden publications is an SNPR. Of these 641 SNPRs, 42, thus 6.6%, are cited in the top 10% patents (i.e. the patents that are in the top 10% of the patent-to-patent citation distribution function). In other words: only 0.2% of all Leiden publications is cited as an SNPR in top-patents whereas Leiden is a university with a relatively high number of SNPRs as most research universities with a large medical school. Biomedical fields generally have higher numbers of SNPRs as compared to other fields of technology, even compared to the engineering-oriented fields. However, of the Leiden publications based on university-industry collaboration a much larger number, about 15%, is an SNPR, and for the top 10% patents it is even 22%.

Winnink and Tijssen (2015) recently identified about 1.2 million WoS publications (1980–2014) on the basis of all patents included (starting from the beginning of the 20th century) in the database PATSTAT Spring 2014 version. This means that about 3.7% of the WoS publications are identified as SNPRs with the preliminary search algorithms. Remind that these figures concern long periods of time, the Leiden data relate to much shorter periods of publication and patent years and hence they are related to more recent research and the numbers are lower. A recent, surprising finding is that 15%–30% of Sleeping Beauties (SBs)

A “Sleeping Beauty in Science” is a publication that goes unnoticed (“sleeps”) for a long time and then, almost suddenly, attracts a lot of attention (“is awakened by a prince”).

are SNPRs (van Raan, 2016).

Review of the Science and Technology Interface Mapping
Monitoring Technological Progress and the Interaction with Science

A challenging method to visualize the interface between science and technology is the use of bibliometric mapping methods such as co-citation, co-word, and co-classification techniques. A discussion of these bibliometric mapping methods can be found in van Raan (2015) and also in the next section. CWTS played a pioneering role in this mapping methodology by creating a time-series of co-word and co-classification based maps for the entire technological domain (Engelsman, van Raan, 1991, 1994; Noyons et al., 1991). A study of the interdisciplinary field of opto-mechatronics in which maps were created based on patents and on scientific publications showed the existence of similar subfields at both the science as well as the technology side (Noyons & van Raan, 1994). In this way, the field is mapped from a technological and from a research point of view.

Recent work shows an ongoing trend in mapping of the science and technology interface to detect and monitor emerging topics. However, these studies often focus on an early stage of the development of emerging fields, before these fields become a major source for patenting. An example is the study of the knowledge diffusion in two emerging fields, the therapeutic use of RNA interference and the application of nanocrystals in solar cells (Leydesdorff and Rafols, 2011). In this work knowledge diffusion between publications relevant for the emerging fields is mapped with network techniques. Given the early stage of these emerging fields, these networks, however, do not contain patents. Upham and Small (2010) and Small, Boyack, and Klavans (2014) use co-citation based mapping techniques to identify emerging topics in science with technological relevance. But also here no patent analysis is involved. Another approach to map technological development was developed by Lee and Jeong (2008). These authors identified trends in the development of robot technology by applying co-word analysis to the metadata of Korean national R&D projects. However, also here no patents were used for the mapping.

Only in a few studies combined patent-publication mapping is used. This is the case in two recent studies of the Leiden group, one on the discovery of introns (Winnink, Tijssen, & van Raan, 2013) and one on the invention of the anti-HIV medical drug Isentress (Winnink & Tijssen, 2014). In both studies network maps are created on the basis of patent-to-patent and patent-to-publication links. Such patent-publication networks are important because scientific progress is a crucial but certainly not the only basis of technological development. Also progress in other and not necessarily directly related fields of technology contribute strongly to the development of a specific field. We show in Figure 5 the patent-to-patent and patent-to-publication citation network around the patent of the anti-HIV medical drug Isentress (Winnink and Tijssen, 2014). In Figure 6 the patent-to-patent and patent-to-publication citations are separated to illustrate the S&T interface of authors and inventors around the Isentress publication of Hazuda et al. (2000).

Figure 5

Patent-to-patent and patent-to-publication citation network around the discovery of Isentress. Network of patents and publications connecting Hazuda et al. (2000) which is the “discovery paper” (green circle), and the Isentress patent (2007) (red circle). Blue circles represent patents; white circles represent publications. From: Winnink and Tijssen (2014).

Figure 6

This network configuration shows the citing-cited relations between the publications with the coauthor relationships between researchers. Blue circles represent scholarly publications; white circles represent (co-)authors. The two closely interconnected clusters are centered around the discovery paper Hazuda et al. (2000). From: Winnink and Tijssen (2014).

Further Diversification of Technology Mapping

In the last few years we notice a strong increase in the application of mapping and network methods to analyze technological developments. The major issues in these studies are monitoring technological state-of-the–art as recent as possible; identification of important, high-impact patents in new fields and in technological improvement; diffusion of technological knowledge, technological change and technological learning capacity; detection of emerging as well as converging fields of technology; and research focusing on improvement of mapping methods. We will briefly discuss recent work on the above issues.

Tackling the problem of monitoring the technological state-of-the–art as recent as possible, Ko et al. (2014) argue that patent-citation networks are insufficient to capture the most recent technological information, particularly the direct and hidden impacts among technologies. To improve technology-impact networks they integrate patent co-classification, decision making algorithms, and social network analysis. The method is illustrated using all Korean patents in the United States patent database from 2008 to 2012.

Identification of important, high-impact patents in new fields and in technological improvement is a hot topic that attracts a lot of attention. Luan et al. (2014) use technology co-classification analysis to show that significant inventions are more technologically diversified and that specific core-technology domains are probably better for creating significant inventions when R&D activities are considered as a whole. Yang et al. (2015) combine four types of patent-citation networks (direct citation, indirect citation, bibliographic coupling, and co-citation networks) and discuss why their approach performs better in covering valuable patents than a direct citation network. Briggs (2015) finds that multi-country jointly-owned patents receive more forward patent citations than patents co-owned within a single country. This indicates that multi-country joint patent co-ownership positively influences the impact of patents. Also the role of university partnerships is investigated, and the author concludes that co-ownership with a university does not result in a direct but more probably in a later impact.

In the mapping and network approaches the focus is not only on patents or on technology-related papers, but also on the performance of the patent assignees. For instance, Huang et al. (2015) use metrics based on traces of matrices composed of vectors describing the distribution of patents, the distribution of their citations, and the difference between these distributions to calculate technological performance of patent assignees. By comparing the results of this traces-based metrics with patent citation counts, with the Current Impact Index and with the patent h-index, they conclude that traces-based metrics provide a valuable complement to patent citation analysis. Guan and Yan (2015) study the impact of multi-level networks on innovation. By using the patent classification system they construct with subclass co-occurrence analysis inventor collaboration networks at city and as well as country level. They find that inter-country collaboration moderates the relationships between inter-city collaboration and innovation performance. Bakker et al. (2016) discuss the pitfalls of using patent-to-patent citations to assess the quality and impact of the patent. Depending on procedures of the patent office and whether the presence of patent families is taken into account, the calculated citation indicators may differ substantially. It is found that corrections for patent families based on a broader definition reveals the most uniform results.

The process of the diffusion of technological knowledge is studied by Ho, Lin, and Liu (2014) using patent-citation network analysis in the field of fuel cells. With help of path analysis, the authors investigate knowledge diffusion across different locations and the role played by specific technological knowledge in the diffusion process. They also find that the technological diversification of a patent had no substantial influence on its network position. Geographical locations also play an important role in the work of Morescalchi et al. (2015). These authors investigate the evolution of networks of innovators within and across borders of institutes and countries. They analyze the impact of physical distance and country borders on inter-regional links in four different networks based on co-inventorship, patent citations, inventor mobility, and the location of R&D laboratories. One of their conclusions is that they cannot detect substantial progress in European research integration other than the common global trend. Wang, Zhang, and Xu (2011) take the entire domain of technology as a starting point and analyze how the developments in the different fields of technology are related at the firm level. They use patent co-citation networks to identify the technological links between 500 important companies. Park and Yoon (2014) use IPC patent co-classification network analysis to study technological knowledge diffusion to measure the long-term role and the intermediating potential of technology sectors. The method is demonstrated with Korean national R&D patents from 2008 to 2011. Hung and Tu (2014) use forward patent citations in the analysis of complexity and chaos in the process of technological change. Learning is a specific topic within our understanding of the diffusion of technological knowledge. Wang, Roijakkers, and Vanhaverbeke (2014) use patent citation analysis to assess how fast Chinese firms learn and catch up.

In technological development the emergence as well as convergence of fields play a crucial role. Kim, Cho, and Kim (2014) use patent-citation network analysis in the field of printed electronics in order to identify key technologies in the convergence process. Kim et al. (2014) study the timely identification of potential technology opportunities by measuring connectivity between clusters of patents using both patent textual data and patent-citation networks. After identifying technology groups with high convergence potential, pairs of core patents based on their technological relatedness are selected. The method is illustrated with a set of US patents in the field of digital information and security. Another approach to study technological convergence is developed by Cho and Kim (2014) who apply the physical concepts entropy and gravity to patent-citation networks. The aim is to discover patterns of the international patent classification codes in printed electronics, and to analyze the role of each technology. The authors discuss how their findings on the evolutionary patterns of technological convergence provide implications for technology foresight. Breitzman and Thomas (2015) develop a tool for locating emerging technologies close to real time across multiple patent systems by using patent-citation techniques. They find that patents in emerging clusters consistently have a significantly higher impact on subsequent technological developments than patents outside these clusters.

Finally, at the more methodological side we find work on the effect of patent-family information on patent-citation network analysis. Nakamura et al. (2015) find in the case of automobile-drivetrain technology that technological trends cannot be understood only with the analysis of patent data issued by a single authority and they discuss the effect of bundling patent-family information. Rodriguez et al. (2015) criticize the use of text mining and keyword analysis for patent relatedness because word choice and writing style of authors may influence the patent-similarity calculations. Therefore, they focus on citations and propose to base patent-similarity measures on normalized direct and indirect co-citation links between patents. Aharonson and Schilling (2016) use network algorithms to calculate the distance between patents with path-length analysis in order to assess technological overlap, similarity, and proximity of firms and to identify outlier patents. Appio, Cesaroni, and Di Minin (2014) use co-citation analysis to map the structure of papers in the intellectual property management and strategy literature to identify its main research areas. Five clusters were found: economics of patent system, technological and institutional capabilities, university patenting, intellectual property exploitation, and division of labor.

Summary of the Findings

On the basis of our review in this section we draw the following conclusions:

Bibliometric mapping enables the visualization of technology fields and related science fields;

Mapping can be based on several methods such as co-citation analysis, bibliographic coupling analysis, co-word analysis, and co-classification analysis;

Time-series of maps enable the discovery of knowledge flows between science and technology as well as between countries or between firms;

Time-series of maps may also have a prospective potential, for instance the early detection of emerging or converging technologies.

Also for mapping and network analysis the combined patent- and publication-citation index system is of crucial importance. It enables the construction of different types of large-scale network structures of publication-patent links. These structures can be based on different bibliometric mapping procedures such as co-citation, bibliographic coupling, co-word, and co-classification analysis. This offers us a reliable, effective, and much less time-consuming way to discover important knowledge flows between science and technology, to identify the publications and patents that play a pivotal role in these flows, and to find the first signs of emerging technological themes. It will also be interesting to study more thoroughly the statistical properties of these networks (e.g. in- and out-degrees of the linkages, characteristics of the emerging clusters, and power-law scaling behavior). A time series of such maps may enable to make extrapolations in time and thus predicting developments in the near future of, say, the next five years.

We conclude the review part of this paper with a co-word analysis of papers published in 2014–2016 (up till November 7, 2016; total number of papers is 327) in the journals Scientometrics, Research Policy, Journal of the American Association for Information Science and Technology, Research Evaluation, and PLoS ONE with the author- and/or database-given keyword “patent*.” The results are shown in Figure 7. We notice that the main clusters (indicated with colors and topic-connecting links) correspond well with the themes discussed in the review part, for instance bibliometric methods in patent analysis to find the links between science and technology (blue); patent citations of the innovation (light blue), the role of universities, academic research, non-patent references, and university-industry collaboration and technology transfer (red); patent citations, R&D and firm performance (purple); and collaboration networks (green). In addition, we see at the right hand side a somewhat isolated cluster (light yellow) on topics for which patents are granted, particularly medical issues.

Figure 7

Concept (co-word) map of the patent-related papers as discussed in the text. (mapping parameter: co-occurrence threshold = 3, full counting).

A Novel Mapping Approach: Second Order SNPRs
Direct Visualization of Cited and Citing Relations of SNPRs

In the foregoing sections we presented a review of the state-of-the-art in patent analysis literature. This last section represents the second part of this paper in which we propose a new way to assess the technology-relevance of publications by identifying publications of a research group (or program, institute, university, country, etc.) that have citation-relations with SNPRs. As discussed earlier, only a small minority of publications covered by the WoS or Scopus “act” as an SNPR, about 3%–4%. This means that an SNPR-based indicator cannot play a crucial role in the evaluation and monitoring of research group or research programs. On the other hand, statistically this probability is comparable to, for instance, the top 1% highly cited publications indicator. Thus, the SNPR-based indicators can be used in an experimental way in evaluation and monitoring provided that the analyses and calculations are performed in an advanced combined patent- and publication-citation index system. Also we discussed that for publications based on university-industry collaboration the number of SNPRs is considerably higher, around 15%. A new method to assess the technological relevance of research is a “second order SNPR” approach: are publications of a research group directly related to an SNPR, more specifically, are they—within a certain time window—cited by or citing to a specific SNPR? In other words, we need an analysis of the citation network of SNPRs. As an example we take the Zarrin SNPR in patent WO2014/009721A1 discussed in Section 2.

For the analysis of the publications cited by an SNPR we use the CWTS bibliometric instrument CitNetExplorer

The CitNetExplorer is a software tool specifically designed for analyzing and visualizing citation networks of scientific literature; it can be uploaded with sets of publication records directly from the Web of Science (WoS) or Scopus. Citation networks can then be explored interactively, for instance by drilling down into a network and by identifying clusters of closely related publications. More about CitNetExplorer see http://www.citnetexplorer.nl/Home.

. By applying this CitNetExplorer we map the target SNPR with its references (cited papers) on a time scale. This enables us to find the scientific roots of the SNPR, and possibly an older but important breakthrough-paper. With the CitNetExplorer also the target SNPR can be mapped with its citing publications, as we will see further on. For a more extensive analysis of the publications citing an SNPR we use the CWTS bibliometric instrument VOS-viewer

The VOS-viewer is a software tool for constructing and visualizing (mapping) a broad range of bibliometric networks. These networks may for instance include journals, researchers, or individual publications, and they can be constructed with co-citation, bibliographic coupling, keyword co-occurrence, or co-authorship relations. In particular, the VOSviewer also offers a text mining functionality that can be used to construct and visualize conceptual (co-word based) networks of terms extracted from a body of scientific literature, particularly titles and abstracts of publications. The VOS viewer can be uploaded with any type of relational information and particularly with publications records of the WoS as well as of Scopus. More about VOSviewer see http://www.vosviewer.com/Home.

. These citing publications with their references (for all citing publications the SNPR is one of the references) enable us to create two different networks. First, the citing publications will have references in common; the more references they have in common, the stronger their relation. This is the bibliographic coupling network in which the citing publications are mapped on the basis of the co-occurrences of references. With advanced clustering techniques, a bibliographic coupling network transforms into map which visualizes a structured landscape of all publications citing an SNPR. It thus provides us with information on the research building on the SNPR. Next, a network of the references of the citing publications can be created. Two references are co-cited if they have a citing paper in common. The more citing papers they have in common, the stronger their co-citation strength. Thus, in the co-citation network the references of the citing papers are mapped. Again with advanced clustering techniques, a co-citation network transforms into map which visualizes a structured landscape of the references of the citing publications. The SNPR is the reference which is by definition a reference of all citing papers. So it will take a central position in the co-citation map. This provides us with information on which publications are often cited together with the SNPR, and therefore publications that are probably just as important for the patented invention as the SNPR.

Figure 8 presents the results of the CitNetExplorer application. The upper part of the figure shows the 63 cited papers (references) of the Zarrin SNPR and the lower part its citing papers (114 up till October 12, 2016; because of space limitations not all citing papers are represented). In both cases we marked the SNPR with a square in the figure. Connecting lines indicate citation relations, these lines always go in an upward direction, which is backward in time. In the upper part of Figure 8 we observe that the Hummers paper is prominently visible on the map as the oldest “building block”

The authors of the Zarrin paper refer erroneously to a paper by F.Kim, L.Cote, and J.Huang in the journal Advanced Materials by indicating year of publication 1954. This is, however, the first page number, the correct citation must be Kim, F., Cote, L.J., & Huang, J. (2010) Adv. Mat, 22(17): 1954–1958.

. The Hummers paper (Hummers & Offeman, 1958) is a very important one: it is a breakthrough paper in the preparation of graphitic oxide, which paved the way to the development of graphene. This work is known as the “Hummers Method”

See https://en.wikipedia.org/wiki/Hummers%27_Method.

and it is cited (up till October 12, 2016) 11,872 times (in the WoS Core Collection). Furthermore, about half of the Zarrin references (34) is cited more than 100 times, 10 of them are cited more than 1,000 times. This clearly shows that within the Zarrin references influential papers are present that are also important for the invention described in the patent WO2014/009721A1, but not cited in the patent.

Figure 8

Maps of the citation links of the Zarrin SNPR. Upper part: the cited papers (references) of the SNPR; lower part: the citing papers of the SNPR. Connecting lines indicate citation relations, and these go always in an upward direction. Colors indicate clusters on the basis of mutual citation relations.

For evaluation and monitoring purposes it seems reasonable to focus on the most recent references of an SNPR with, for instance, publication year up till five years before the publication year of the SNPR. In the Zarrin case this procedure would render 29 publications, and of these 29 again about half (15) are cited more than 100 times, and 4 more than 1,000 times. The most cited paper (6,107 times) is by Stankovich et al. (2006) on graphene-based composite materials published in Nature. This paper is also marked in the upper part of Figure 8. Further on we will discuss how papers related to an SNPR can be included in an assessment of technological relevance. But we first continue with an analysis of the papers citing an SNPR and again we take the Zarrin SNPR as an example.

The lower part of Figure 8, which is like the upper part also created with the CitNet Explorer (by uploading the set of all papers citing the Zarrin SNPR into the CitNet Explorer), presents the citing papers up till 2015. These citing papers do, of course, cite more papers, including mutual citations within the set of citing papers. These mutual connections are also visible in the lower part of Figure 8. With the uploaded set of citing papers a much more comprehensive analysis of all citation links can be carried out but this must be done in an interactive way with the CitNet Explorer.

Mapping the Landscape of the Papers Citing SNPRs

The extent to which citing papers have cited papers (references) in common, is a measure of similarity of these citing papers. As we discussed above, the method to map these similarities is called bibliographic coupling. Thus this method provides a landscape of the relations between the papers citing an SNPR on the basis of reference similarity. In Figure 9 we show the bibliographic coupling map of the papers citing the Zarrin SNPR. It reveals a visualization, including clustering, of the recent research based on the SNPR as one of the building stones. We find in Figure 9 the same papers as in the lower part of Figure 8, but now they are clustered and mapped on the basis of all their mutual citation relations (using a lower threshold than in Figure 8) in a landscape structured by bibliographic coupling.

Figure 9

Bibliographic coupling (minimum citations = 1) map of the papers citing the Zarrin SNPR. This is a detailed visualization of the links between these citing papers, thus providing a map of the recent research based on the Zarrin SNPR as one of the building stones. The size of the circles is proportional to its impact, i.e. the extent to which a paper is cited in the entire Web of Science (mapping parameter: minimum citations = 1).

The counterpart of bibliographic coupling is co-citation analysis. The extent to which cited papers have citing papers in common, is a measure of similarity of these cited papers. Thus, with the co-citation method a map is created with the relations between the references (building stones) of the papers citing an SNPR. In Figure 10 we show the co-citation map of the papers citing the Zarrin SNPR. It is a visualization of the papers cited by these citing papers, and thus providing a map of nearly all building stones of the recent research, with the SNPRs Zarrin in a central position. We also observe at the left-hand side of the map the work on graphene of the two Nobel Laureates Geim and Novosolov (Geim & Novoselov 2007, cited (as of October 30, 2016) 16,506 times; Novoselov et al. 2004, cited (as of October 30, 2016) 22,602 times), as well as the earlier discussed highly cited work of Stankovich et al. (2006) on graphene-based composite materials. We refer to Winnink and Tijssen (2015) for a detailed discussion of graphene research based on an early stage identification of breakthrough work in this field at the interface of science and technology.

Figure 10

Co-citation map of papers citing the Zarrin SNPR (co-citation threshold = 3). The size of the circles is proportional to the number of times a paper is cited in the uploaded set. By definition, the target paper (here Zarrin) is the most cited paper, as all papers in the set cite the Zarrin paper.

If we look at the Zarrin paper in Figure 10 we see another paper indicated with a large circle very close at the left-hand side. By zooming into this part of the map, Figure 11, it is clear that this is the pioneering Hummers paper we discussed earlier.

Figure 11

Zoom into the co-citation map of the papers citing the Zarrin SNPR as shown in Figure 10. The next-to-Zarrin most central paper is the Hummers paper, see text.

The Conceptual Environment of SNPRs

The above discussed results are all based on citations links. Publications can also be characterized with a list of concepts and mathematically similar mapping procedures as with citations can be carried out. In order to do so, we use natural language processing (text mining) to extract the important, publication-specific concepts (terms such as keywords or noun phrases) from the titles and abstracts of a set of publications. Alternatively, keywords given by the authors and by the database can be used. By measuring all co-occurrences of any possible pair of concepts, co-word maps can be created in which the conceptual structure of the research represented by the set of publications is visualized. For a recent discussion of the concept mapping methodology we refer to Waltman, van Raan, and Smart (2014).

In Figure 12 we present the co-word map based on both author-as well as database-given keywords of all papers citing the Zarrin SNPR. We clearly observe many concepts directly related to the Zarrin SNPR and the patent in which the Zarrin paper is cited. Examples are fuel cells, graphene, graphene oxide, water, humidity, nanocomposite membrane, proton-exchange membrane, and nafion. Colors indicate clusters of concepts and can be seen as research themes. For instance, the red cluster is about membranes particularly in relation to methanol fuel-cells, the dark blue relates to graphite oxide research, and the purple cluster to graphene oxide. Figure 13 shows the same map, but now the colors indicate the average publication year of the papers belonging to a specific concept. We see that most of the more recent work is concentrated in the middle of the map, especially around methanol fuel-cells.

Figure 12

Concept (co-word) map of the papers citing the Zarrin SNPR (mapping parameter: co-occurrence threshold = 3).

Figure 13

Same map as in Figure 12, but now the colors indicate the average publication year of the papers belonging to a specific concept.

Suggestions for Evaluation Practices

In order to assess the technological relevance of scientific publications a focus on SNPRs is not sufficient. We suggest that in the assessment of technological relevance also the highly cited references of SNPRs up till five years before the publication of the SNPR are taken into account.

A procedure to achieve this could be as follows. First, by combining the patent database with the WoS or Scopus, all SNPRs from, for instance, 2005 can be identified. Second, collect all references of these SNPRs and select (1) the “young” references with publication year up till five years before the publication year of the SNPR, and (2) within this set of “young” references those that are highly cited (in the Zarrin SNPR the citation threshold was 100 citations). Third, collect all papers citing the SNPRs and again select those citing papers with a high impact, for instance the top 10% or 20%. These steps generate about an order of magnitude more technology-relevant papers than only the SNPRs, depending, of course, on the citation threshold used to define high impact.

In the above described way, a database of technology-relevant papers (TRPs) is created. In a bibliometric assessment the publications of research groups, research programs or institutes can be matched with the TRPs and thus the extent to which the work of groups, programs, or institutes are relevant for technological development can be measured.

Figure 1

Front page of patent WO2014/009721A1 as an example.
Front page of patent WO2014/009721A1 as an example.

Figure 2

Claims of patent WO2014/009721A1; the fi rst page of the claims section is shown as an example.
Claims of patent WO2014/009721A1; the fi rst page of the claims section is shown as an example.

Figure 3

International search report of patent WO2014/009721A1, page 1. The SNPRs are indicated with a light blue box, and the references to earlier patents with a red box.
International search report of patent WO2014/009721A1, page 1. The SNPRs are indicated with a light blue box, and the references to earlier patents with a red box.

Figure 4

International search report of patent WO2014/009721A1, page 2. The SNPRs are indicated with a light blue box, and the references to earlier patents with a red box.
International search report of patent WO2014/009721A1, page 2. The SNPRs are indicated with a light blue box, and the references to earlier patents with a red box.

Figure 5

Patent-to-patent and patent-to-publication citation network around the discovery of Isentress. Network of patents and publications connecting Hazuda et al. (2000) which is the “discovery paper” (green circle), and the Isentress patent (2007) (red circle). Blue circles represent patents; white circles represent publications. From: Winnink and Tijssen (2014).
Patent-to-patent and patent-to-publication citation network around the discovery of Isentress. Network of patents and publications connecting Hazuda et al. (2000) which is the “discovery paper” (green circle), and the Isentress patent (2007) (red circle). Blue circles represent patents; white circles represent publications. From: Winnink and Tijssen (2014).

Figure 6

This network configuration shows the citing-cited relations between the publications with the coauthor relationships between researchers. Blue circles represent scholarly publications; white circles represent (co-)authors. The two closely interconnected clusters are centered around the discovery paper Hazuda et al. (2000). From: Winnink and Tijssen (2014).
This network configuration shows the citing-cited relations between the publications with the coauthor relationships between researchers. Blue circles represent scholarly publications; white circles represent (co-)authors. The two closely interconnected clusters are centered around the discovery paper Hazuda et al. (2000). From: Winnink and Tijssen (2014).

Figure 7

Concept (co-word) map of the patent-related papers as discussed in the text. (mapping parameter: co-occurrence threshold = 3, full counting).
Concept (co-word) map of the patent-related papers as discussed in the text. (mapping parameter: co-occurrence threshold = 3, full counting).

Figure 8

Maps of the citation links of the Zarrin SNPR. Upper part: the cited papers (references) of the SNPR; lower part: the citing papers of the SNPR. Connecting lines indicate citation relations, and these go always in an upward direction. Colors indicate clusters on the basis of mutual citation relations.
Maps of the citation links of the Zarrin SNPR. Upper part: the cited papers (references) of the SNPR; lower part: the citing papers of the SNPR. Connecting lines indicate citation relations, and these go always in an upward direction. Colors indicate clusters on the basis of mutual citation relations.

Figure 9

Bibliographic coupling (minimum citations = 1) map of the papers citing the Zarrin SNPR. This is a detailed visualization of the links between these citing papers, thus providing a map of the recent research based on the Zarrin SNPR as one of the building stones. The size of the circles is proportional to its impact, i.e. the extent to which a paper is cited in the entire Web of Science (mapping parameter: minimum citations = 1).
Bibliographic coupling (minimum citations = 1) map of the papers citing the Zarrin SNPR. This is a detailed visualization of the links between these citing papers, thus providing a map of the recent research based on the Zarrin SNPR as one of the building stones. The size of the circles is proportional to its impact, i.e. the extent to which a paper is cited in the entire Web of Science (mapping parameter: minimum citations = 1).

Figure 10

Co-citation map of papers citing the Zarrin SNPR (co-citation threshold = 3). The size of the circles is proportional to the number of times a paper is cited in the uploaded set. By definition, the target paper (here Zarrin) is the most cited paper, as all papers in the set cite the Zarrin paper.
Co-citation map of papers citing the Zarrin SNPR (co-citation threshold = 3). The size of the circles is proportional to the number of times a paper is cited in the uploaded set. By definition, the target paper (here Zarrin) is the most cited paper, as all papers in the set cite the Zarrin paper.

Figure 11

Zoom into the co-citation map of the papers citing the Zarrin SNPR as shown in Figure 10. The next-to-Zarrin most central paper is the Hummers paper, see text.
Zoom into the co-citation map of the papers citing the Zarrin SNPR as shown in Figure 10. The next-to-Zarrin most central paper is the Hummers paper, see text.

Figure 12

Concept (co-word) map of the papers citing the Zarrin SNPR (mapping parameter: co-occurrence threshold = 3).
Concept (co-word) map of the papers citing the Zarrin SNPR (mapping parameter: co-occurrence threshold = 3).

Figure 13

Same map as in Figure 12, but now the colors indicate the average publication year of the papers belonging to a specific concept.
Same map as in Figure 12, but now the colors indicate the average publication year of the papers belonging to a specific concept.

Aharonson, B.S., & Schilling, M.A. (2016). Mapping the technological landscape: Measuring technology distance, technological footprints, and technology evolution. Research Policy, 45(12), 81–96.AharonsonB.S.SchillingM.A.2016Mapping the technological landscape: Measuring technology distance, technological footprints, and technology evolutionResearch Policy45128196Search in Google Scholar

Albert, M.B., Avery, D., Narin, F., & McAllister, P. (1991). Direct validation of citation counts as indicators of industrially important patents. Research Policy, 20(3), 251–259.AlbertM.B.AveryD.NarinF.McAllisterP.1991Direct validation of citation counts as indicators of industrially important patentsResearch Policy203251259Search in Google Scholar

Alcácer, J., Gittelman, M., & Sampat, B. (2009). Applicant and examiner citations in U.S. patents: An overview and analysis. Research Policy, 38(2), 415–427.AlcácerJ.GittelmanM.SampatB.2009Applicant and examiner citations in U.S. patents: An overview and analysisResearch Policy382415427Search in Google Scholar

Appio, F.P., Cesaroni, F., & Di Minin, A. (2014). Visualizing the structure and bridges of the intellectual property management and strategy literature: A document co-citation analysis. Scientometrics, 101(1), 623–661.AppioF.P.CesaroniF.Di MininA.2014Visualizing the structure and bridges of the intellectual property management and strategy literature: A document co-citation analysisScientometrics1011623661Search in Google Scholar

Arts, S., Appio, F., & van Looy, B. (2012). Validating patent indicators that assess technological radicalness: The case of biotechnology. In E. Archambault, Y. Gingras, & V. Larivière (Eds.), Proceedings of 17th International Conference on Science and Technology Indicators (Vol. 1, pp. 82–97). Montréal: Science-Metrix and OST.ArtsS.AppioF.van LooyB.2012Validating patent indicators that assess technological radicalness: The case of biotechnologyArchambaultE.GingrasY.LarivièreV.Proceedings of 17th International Conference on Science and Technology Indicators18297Montréal: Science-Metrix and OSTSearch in Google Scholar

Bakker, J., Verhoeven, D., Zhang, L., & van Looy, B. (2016). Patent citation indicators: One size fits all? Scientometrics, 106(1), 187–211.BakkerJ.VerhoevenD.ZhangL.van LooyB.2016Patent citation indicators: One size fits all?Scientometrics1061187211Search in Google Scholar

Balconi, M., Breschi, S., & Lissoni, F. (2004). Networks of inventors and the role of academia: An exploration of Italian patent data. Research Policy, 33(1), 127–145.BalconiM.BreschiS.LissoniF.2004Networks of inventors and the role of academia: An exploration of Italian patent dataResearch Policy331127145Search in Google Scholar

Benson, C.L., & Magee, C.L. (2015). Quantitative determination of technological improvement from patent data. PLoS ONE, 10(4), e0121635.BensonC.L.MageeC.L.2015Quantitative determination of technological improvement from patent dataPLoS ONE104e0121635Search in Google Scholar

Boyack, K.W., & Klavans, R. (2008). Measuring science-technology interaction using rare inventor-author names. Journal of Informetrics, 2, 173–182.BoyackK.W.KlavansR.2008Measuring science-technology interaction using rare inventor-author namesJournal of Informetrics2173182Search in Google Scholar

Briggs, K. (2015). Co-owner relationships conducive to high quality joint patents. Research Policy, 44(8), 1566–1573.BriggsK.2015Co-owner relationships conducive to high quality joint patentsResearch Policy44815661573Search in Google Scholar

Breitzman, A., & Thomas, P. (2015). The emerging clusters model: A tool for identifying emerging technologies across multiple patent systems. Research Policy, 44(1), 195–205.BreitzmanA.ThomasP.2015The emerging clusters model: A tool for identifying emerging technologies across multiple patent systemsResearch Policy441195205Search in Google Scholar

Bruck, P., Rethy, I., Szente, J., Tobochnik, J., & Erdi, P. (2016). Recognition of emerging technology trends: Class-selective study of citations in the US Patent Citation Network. Scientometrics, 107(3), 1465–1475.BruckP.RethyI.SzenteJ.TobochnikJ.ErdiP.2016Recognition of emerging technology trends: Class-selective study of citations in the US Patent Citation NetworkScientometrics107314651475Search in Google Scholar

Callaert, J., van Looy, B., Verbeek, A., Debackere, K., & Thijs, B. (2006). Traces of prior art: An analysis of non-patent references found in patent documents. Scientometrics, 69(1), 3–20.CallaertJ.van LooyB.VerbeekA.DebackereK.ThijsB.2006Traces of prior art: An analysis of non-patent references found in patent documentsScientometrics691320Search in Google Scholar

Callaert, J., Grouwels, J., & van Looy, B. (2012). Delineating the scientific footprint in technology: Identifying science within non-patent references. Scientometrics, 91(2), 383–398.CallaertJ.GrouwelsJ.van LooyB.2012Delineating the scientific footprint in technology: Identifying science within non-patent referencesScientometrics912383398Search in Google Scholar

Callaert, J., Pellens, M., & van Looy, B. (2014). Sources of inspiration? Making sense of scientific references in patents. Scientometrics, 98(3), 1617–1629.CallaertJ.PellensM.van LooyB.2014Sources of inspiration? Making sense of scientific references in patentsScientometrics98316171629Search in Google Scholar

Callaert, J., Vervenne, J.B., van Looy, B., Magerman, T., Song, X., & Jeuris, W. (2014). Patterns of science-technology linkage. European Commission. Retrieved on November 29, 2016, from http://ec.europa.eu/research/innovation-union/pdf/patterns_of_science-technology_linkage.pdf.CallaertJ.VervenneJ.B.van LooyB.MagermanT.SongX.JeurisW.2014Patterns of science-technology linkageEuropean CommissionRetrieved on November 29, 2016, fromhttp://ec.europa.eu/research/innovation-union/pdf/patterns_of_science-technology_linkage.pdfSearch in Google Scholar

Carpenter, M.P., Cooper, M., & Narin, F. (1980). Linkage between basic research literature and patents. Research Management, 13(2), 30–35.CarpenterM.P.CooperM.NarinF.1980Linkage between basic research literature and patentsResearch Management1323035Search in Google Scholar

Carpenter, M.P., Narin, F., & Woolf, P. (1981). Citation rates to technologically important patents. World Patent Information, 3(4), 160–163.CarpenterM.P.NarinF.WoolfP.1981Citation rates to technologically important patentsWorld Patent Information34160163Search in Google Scholar

Carpenter, M.P., & Narin, F. (1983). Validation study: Patent citations as indicators of science and foreign dependence. World Patent Information, 5(3), 180–185.CarpenterM.P.NarinF.1983Validation study: Patent citations as indicators of science and foreign dependenceWorld Patent Information53180185Search in Google Scholar

Cassiman, B., Glenisson, P., & van Looy, B. (2007). Measuring industry-science links through inventor-author relations: A profiling methodology. Scientometrics, 70(2), 379–391.CassimanB.GlenissonP.van LooyB.2007Measuring industry-science links through inventor-author relations: A profiling methodologyScientometrics702379391Search in Google Scholar

Chai, S., & Shih, W. (2016). Bridging science and technology through academic-industry partnerships. Research Policy, 45(1), 148–158.ChaiS.ShihW.2016Bridging science and technology through academic-industry partnershipsResearch Policy451148158Search in Google Scholar

Cho, Y., & Kim, M. (2014). Entropy and gravity concepts as new methodological indexes to investigate technological convergence: Patent network-based approach. PLoS ONE, 9(6), e98009.ChoY.KimM.2014Entropy and gravity concepts as new methodological indexes to investigate technological convergence: Patent network-based approachPLoS ONE96e98009Search in Google Scholar

Chowdhury, G., Koya, K., & Philipson, P. (2016). Measuring the impact of research: Lessons from the UK’s research excellence framework 2014. PLoS ONE, 11(6), e0156978.ChowdhuryG.KoyaK.PhilipsonP.2016Measuring the impact of research: Lessons from the UK’s research excellence framework 2014PLoS ONE116e0156978Search in Google Scholar

Coward, H.R., & Franklin, J.J. (1989). Identifying the science-technology interface: Matching patent data to a bibliometric model. Science, Technology and Human Values, 14(1), 50–77.CowardH.R.FranklinJ.J.1989Identifying the science-technology interface: Matching patent data to a bibliometric modelScience, Technology and Human Values1415077Search in Google Scholar

Criscuolo, P, & Verspagen, B. (2008). Does it matter where patent citations come from? Inventor vs. examiner citations in European patents. Research Policy, 37(10), 1892–1908.CriscuoloP.VerspagenB.2008Does it matter where patent citations come from? Inventor vs. examiner citations in European patentsResearch Policy371018921908Search in Google Scholar

Engelsman, E.C., & van Raan, A.F.J. (1991). Mapping of technology. A first exploration of knowledge diffusion amongst fields of technology. Policy Studies on Technology and Economy (BTE) Series. The Hague: Netherlands Ministry of Economic Affairs.EngelsmanE.C.van RaanA.F.J.1991Mapping of technology. A first exploration of knowledge diffusion amongst fields of technologyPolicy Studies on Technology and Economy (BTE) SeriesThe HagueNetherlands Ministry of Economic AffairsSearch in Google Scholar

Engelsman, E.C., & van Raan, A.F.J. (1994). A patent-based cartography of technology. Research Policy, 23(94), 1–26.EngelsmanE.C.van RaanA.F.J.1994A patent-based cartography of technologyResearch Policy2394126Search in Google Scholar

European Commission. (2005). Study on evaluating the knowledge economy – What are patents actually worth? The value of patents for today’s economy and society (PATVAL study). Retrieved on November 29, 2016, from http://ec.europa.eu/internal_market/indprop/docs/patent/studies/patentstudy-report_en.pdf.European Commission2005Study on evaluating the knowledge economy – What are patents actually worth?The value of patents for today’s economy and society (PATVAL study)Retrieved on November 29, 2016, fromhttp://ec.europa.eu/internal_market/indprop/docs/patent/studies/patentstudy-report_en.pdfSearch in Google Scholar

Finardi, U. (2011). Time relations between scientific production and patenting of knowledge: The case of nanotechnologies. Scientometrics, 89(1), 37–50.FinardiU.2011Time relations between scientific production and patenting of knowledge: The case of nanotechnologiesScientometrics8913750Search in Google Scholar

Fukuzawa, N., & Ida, T. (2016). Science linkages between scientific articles and patents for leading scientists in the life and medical sciences field: The case of Japan. Scientometrics, 106(2), 629–644.FukuzawaN.IdaT.2016Science linkages between scientific articles and patents for leading scientists in the life and medical sciences field: The case of JapanScientometrics1062629644Search in Google Scholar

Geim, A.K., & Novoselov, K.S. (2007). The rise of graphene. Nature Materials, 6(3), 183–191.GeimA.K.NovoselovK.S.2007The rise of grapheneNature Materials63183191Search in Google Scholar

Grant, J., Green, L., & Mason, B. (2003). Basic research and health: A reassessment of the scientific basis for the support of biomedical science. Research Evaluation, 12(3), 217–224.GrantJ.GreenL.MasonB.2003Basic research and health: A reassessment of the scientific basis for the support of biomedical scienceResearch Evaluation123217224Search in Google Scholar

Grupp, H. (Ed.). (1992). Dynamics of science-based innovation. Heidelberg: Springer-Verlag.GruppH.1992Dynamics of science-based innovationHeidelbergSpringer-VerlagSearch in Google Scholar

Guan, J.C., & Yan, Y. (2015). Technological proximity and recombinative innovation in the alternative energy field. Research Policy, 44(3), 545–559.GuanJ.C.YanY.2015Technological proximity and recombinative innovation in the alternative energy fieldResearch Policy443545559Search in Google Scholar

Guerzoni, M., Aldridge, T.T., Audretsch, D.B., & Desai, S. (2014). A new industry creation and originality: Insight from the funding sources of university patents. Research Policy, 43(10), 1697–1706.GuerzoniM.AldridgeT.T.AudretschD.B.DesaiS.2014A new industry creation and originality: Insight from the funding sources of university patentsResearch Policy431016971706Search in Google Scholar

Halevi, G., & Moed, H.F. (2012). The technological impact of library science research: A patent analysis. In E. Archambault, Y. Gingras, & V. Larivière (Eds.), Proceedings of 17th International Conference on Science and Technology Indicators (Vol.1, pp. 371–380), Montréal: Science-Metrix and OST.HaleviG.MoedH.F.2012The technological impact of library science research: A patent analysisArchambaultE.GingrasY.LarivièreV.Proceedings of 17th International Conference on Science and Technology Indicators1371380MontréalScience-Metrix and OSTSearch in Google Scholar

Hall, B.H., Jaffe, A., & Trajtenberg, M. (2005). Market value and patent citations. RAND Journal of Economics, 36(1), 16–38.HallB.H.JaffeA.TrajtenbergM.2005Market value and patent citationsRAND Journal of Economics3611638Search in Google Scholar

Harhoff, D., Narin, F., Scherer, M., & Vopel, K. (1999). Citation frequency and the value of patented inventions. Review of Economics and Statistics, 81(3), 511–515.HarhoffD.NarinF.SchererM.VopelK.1999Citation frequency and the value of patented inventionsReview of Economics and Statistics813511515Search in Google Scholar

Hazuda, D.J., Felock, P., Witmer, M., Wolfe, A., Stillmock, K., Grobler, J.A., Espeseth, A., Gabryelski, L., Schleif, W., Blau, C., & Miller, M.D. (2000). Inhibitors of strand transfer that prevent integration and inhibit HIV-1 replication in cells. Science, 287(5453), 646–650.HazudaD.J.FelockP.WitmerM.WolfeA.StillmockK.GroblerJ.A.EspesethA.GabryelskiL.SchleifW.BlauC.MillerM.D.2000Inhibitors of strand transfer that prevent integration and inhibit HIV-1 replication in cellsScience2875453646650Search in Google Scholar

Heilbron, J.L. (1972). Illinois Institute of Technology Research Institute - Technology in retrospect and critical events in science. Isis, 63(1), 115.HeilbronJ.L.1972Illinois Institute of Technology Research Institute - Technology in retrospect and critical events in scienceIsis631115Search in Google Scholar

Ho, M.H.C., Lin, V.H., & Liu, J.S. (2014). Exploring knowledge diffusion among nations: A study of core technologies in fuel cells. Scientometrics, 100(1), 149–171.HoM.H.C.LinV.H.LiuJ.S.2014Exploring knowledge diffusion among nations: A study of core technologies in fuel cellsScientometrics1001149171Search in Google Scholar

Hu, D., Chen, H., Huang, Z., & Roco, M.C. (2007). Longitudinal study on patent citation to academic research articles in nanotechnology (1976–2004). Journal of Nanoparticle Research, 9(9), 529–542.HuD.ChenH.HuangZ.RocoM.C.2007Longitudinal study on patent citation to academic research articles in nanotechnology (1976–2004)Journal of Nanoparticle Research99529542Search in Google Scholar

Huang, M.H., Chen, D.Z., Shen, D.Q., Wang, M.S., & Ye, F.Y. (2015). Measuring technological performance of assignees using trace metrics in three fields. Scientometrics, 104(1), 61–86.HuangM.H.ChenD.Z.ShenD.Q.WangM.S.YeF.Y.2015Measuring technological performance of assignees using trace metrics in three fieldsScientometrics10416186Search in Google Scholar

Hummers, W.S., & Offeman, R.E. (1958). Preparation of graphitic oxide. Journal of the American Chemical Society, 80(6), 1339–1339.HummersW.S.OffemanR.E.1958Preparation of graphitic oxideJournal of the American Chemical Society80613391339Search in Google Scholar

Hung, S.C., & Tu, M.F. (2014). Is small actually big? The chaos of technological change. Research Policy, 43(7), 1227–1238.HungS.C.TuM.F.2014Is small actually big? The chaos of technological changeResearch Policy43712271238Search in Google Scholar

Hung, W.C., Ding, C.G., Wang, H.J., Lee, M.C., & Lin, C.P. (2015). Evaluating and comparing the university performance in knowledge. Scientometrics, 102(2), 1269–1286.HungW.C.DingC.G.WangH.J.LeeM.C.LinC.P.2015Evaluating and comparing the university performance in knowledgeScientometrics102212691286Search in Google Scholar

Illinois Institute of Technology (IIT). (1968). Technology in retrospect and critical events in science. Vol. 1. Chicago, Illinois: Illinois Institute of Technology Research Institute.Illinois Institute of Technology (IIT)1968Technology in retrospect and critical events in science1Chicago, IllinoisIllinois Institute of Technology Research InstituteSearch in Google Scholar

Illinois Institute of Technology (IIT). (1969). Technology in retrospect and critical events in science. Vol. 2. Chicago, Illinois: Illinois Institute of Technology Research Institute.Illinois Institute of Technology (IIT)1969Technology in retrospect and critical events in science2Chicago, IllinoisIllinois Institute of Technology Research InstituteSearch in Google Scholar

Isenson, R.S. (1969). Project Hindsight (final report). Washington, DC, 20301: Office of the Director of Defense Research Engineering, AD495905.IsensonR.S.1969Project Hindsight (final report)Washington, DC, 20301Office of the Director of Defense Research EngineeringAD495905Search in Google Scholar

Kim, B., Gazzola, G., Lee, J.M., Kim, D., Kim, K., & Jeong, M.K. (2014). Inter-cluster connectivity analysis for technology opportunity discovery. Scientometrics, 98(3), 1811–1825.KimB.GazzolaG.LeeJ.M.KimD.KimK.JeongM.K.2014Inter-cluster connectivity analysis for technology opportunity discoveryScientometrics98318111825Search in Google Scholar

Kim, E., Cho, Y., & Kim, W. (2014). Dynamic patterns of technological convergence in printed electronics technologies: Patent citation network. Scientometrics, 98(2), 975–998.KimE.ChoY.KimW.2014Dynamic patterns of technological convergence in printed electronics technologies: Patent citation networkScientometrics982975998Search in Google Scholar

Ko, S.S., Ko, N, Kim, D., Park, H., & Yoon, J. (2014). Analyzing technology impact networks for R&D planning using patents: Combined application of network approaches. Scientometrics, 101(1), 917–936.KoS.S.KoNKimD.ParkH.YoonJ.2014Analyzing technology impact networks for R&D planning using patents: Combined application of network approachesScientometrics1011917936Search in Google Scholar

Lee, B., & Jeong, Y. (2008). Mapping Korea’s national R&D domain of robot technology by using the co-word analysis. Scientometrics, 77(1), 3–19.LeeB.JeongY.2008Mapping Korea’s national R&D domain of robot technology by using the co-word analysisScientometrics771319Search in Google Scholar

Leten, B., Landoni, P., & van Looy, B. (2014). Science or graduates: How do firms benefit from the proximity of universities? Research Policy, 43(8), 1398–1412.LetenB.LandoniP.van LooyB.2014Science or graduates: How do firms benefit from the proximity of universities?Research Policy43813981412Search in Google Scholar

Leydesdorff, L., & Rafols, I. (2011). Local emergence and global diffusion of research technologies: An exploration of patterns of network formation. Journal of the American Society for Information Science and Technology, 62(5), 846–860.LeydesdorffL.RafolsI.2011Local emergence and global diffusion of research technologies: An exploration of patterns of network formationJournal of the American Society for Information Science and Technology625846860Search in Google Scholar

Lo, S.S. (2010). Scientific linkage of science research and technology development: A case of genetic engineering research. Scientometrics, 82(1), 109–120.LoS.S.2010Scientific linkage of science research and technology development: A case of genetic engineering researchScientometrics821109120Search in Google Scholar

Luan, C.J., Hou, H.Y., Wang, Y.T., & Wang, X.W. (2014). Are significant inventions more diversified? Scientometrics, 100(2), 459–470.LuanC.J.HouH.Y.WangY.T.WangX.W.2014Are significant inventions more diversified?Scientometrics1002459470Search in Google Scholar

Magerman, T., van Looy, B., & Song, X. (2010). Exploring the feasibility and accuracy of Latent Semantic Analysis based text mining techniques to detect similarity between patent documents and scientific publications. Scientometrics, 82(2), 289–306.MagermanT.van LooyB.SongX.2010Exploring the feasibility and accuracy of Latent Semantic Analysis based text mining techniques to detect similarity between patent documents and scientific publicationsScientometrics822289306Search in Google Scholar

Magerman, T., van Looy, B., & Debackere, K. (2015). Does involvement in patenting jeopardize one’s academic footprint? An analysis of patent-paper pairs in biotechnology. Research Policy, 44(9), 1702–1713.MagermanT.van LooyB.DebackereK.2015Does involvement in patenting jeopardize one’s academic footprint? An analysis of patent-paper pairs in biotechnologyResearch Policy44917021713Search in Google Scholar

Maraut, S., & Martinez, C (2014). Identifying author–inventors from Spain: Methods and a first insight into results. Scientometrics, 101(1), 445–476.MarautS.MartinezC2014Identifying author–inventors from Spain: Methods and a first insight into resultsScientometrics1011445476Search in Google Scholar

Mehta, A., Rysman, M., & Simcoe, T. (2010). Identifying the age profile of patent citations: New estimates of knowledge diffusion. Journal of Applied Econometrics, 25(7), 1179–1204.MehtaA.RysmanM.SimcoeT.2010Identifying the age profile of patent citations: New estimates of knowledge diffusionJournal of Applied Econometrics25711791204Search in Google Scholar

Meyer, M. (2000). Patent citations in a novel field of technology: What can they tell about interactions between emerging communities of science and technology. Scientometrics, 48(2), 151–178.MeyerM.2000Patent citations in a novel field of technology: What can they tell about interactions between emerging communities of science and technologyScientometrics482151178Search in Google Scholar

Meyer, M. (2001). Patent citation analysis in a novel field of technology: An exploration of nano-science and nano-technology. Scientometrics, 51(1), 163–183.MeyerM.2001Patent citation analysis in a novel field of technology: An exploration of nano-science and nano-technologyScientometrics511163183Search in Google Scholar

Meyer, M. (2005). Inventor-authors: Knowledge integrators or weak links? An exploratory comparison of co-active researchers with their non-inventing peers in nano-science and technology. Working Paper No 2005/1, Helsinki University of Technology.MeyerM.2005Inventor-authors: Knowledge integrators or weak links?An exploratory comparison of co-active researchers with their non-inventing peers in nano-science and technologyWorking Paper No 2005/1, Helsinki University of TechnologySearch in Google Scholar

Morescalchi, A., Pammolli, F., Penner, O., Petersen, A.M., & Riccaboni, M. (2015). The evolution of networks of innovators within and across borders: Evidence from patent data. Research Policy, 44(3), 651–668.MorescalchiA.PammolliF.PennerO.PetersenA.M.RiccaboniM.2015The evolution of networks of innovators within and across borders: Evidence from patent dataResearch Policy443651668Search in Google Scholar

Mowery, D.C., & Ziedonis, A.A. (2015). Markets versus spillovers in outflows of university research. Research Policy, 44(1), 50–66.MoweryD.C.ZiedonisA.A.2015Markets versus spillovers in outflows of university researchResearch Policy4415066Search in Google Scholar

Murray, F. (2004). The role of academic inventors in entrepreneurial firms: Sharing the laboratory life. Research Policy, 33(4), 643–659.MurrayF.2004The role of academic inventors in entrepreneurial firms: Sharing the laboratory lifeResearch Policy334643659Search in Google Scholar

Nakamura, H., Suzuki, S., Kajikawa, Y., & Osawa, M. (2015). The effect of patent family information in patent citation network analysis: A comparative case study in the drivetrain domain. Scientometrics, 104(2), 437–452.NakamuraH.SuzukiS.KajikawaY.OsawaM.2015The effect of patent family information in patent citation network analysis: A comparative case study in the drivetrain domainScientometrics1042437452Search in Google Scholar

Narin, F., & Noma, E. (1985). Is technology becoming science? Scientometrics, 7(3–6), 369–381.NarinF.NomaE.1985Is technology becoming science?Scientometrics73–6369381Search in Google Scholar

Narin, F., Rosen, M., & Olivastro, D. (1989). Patent citation analysis: New validation studies and linkage statistics. In A.F.J. van Raan, A.J. Nederhoff, & H.F. Moed (Eds.), Science and Technology Indicators: Their Use in Science Policy and their Role in Science Studies. Leiden: DSWO Press.NarinF.RosenM.OlivastroD.1989Patent citation analysis: New validation studies and linkage statisticsvan RaanA.F.J.NederhoffA.J.MoedH.F.Science and Technology Indicators: Their Use in Science Policy and their Role in Science StudiesLeidenDSWO PressSearch in Google Scholar

Narin, F., Hamilton, K., & Olivastro, D. (1997). The increasing linkage between U.S. technology and public science. Research Policy, 26(3), 317–330.NarinF.HamiltonK.OlivastroD.1997The increasing linkage between U.S. technology and public scienceResearch Policy263317330Search in Google Scholar

Novoselov, K.S., Geim, A.K., Morozov, S.V., Jiang, D., Zhang, Y., Dubonos, S.V., Grigorieva, I.V., & Firsov, A.A. (2004). Electric field effect in atomically thin carbon films. Science, 306(5696), 666–669.NovoselovK.S.GeimA.K.MorozovS.V.JiangD.ZhangY.DubonosS.V.GrigorievaI.V.FirsovA.A.2004Electric field effect in atomically thin carbon filmsScience3065696666669Search in Google Scholar

Noyons, E.C.M., Engelsman, E.C., & van Raan, A.F.J. (1991). Tracing technological developments. Policy Studies on Technology and Economy (BTE) Series. The Hague: Netherlands Ministry of Economic Affairs.NoyonsE.C.M.EngelsmanE.C.van RaanA.F.J.1991Tracing technological developmentsPolicy Studies on Technology and Economy (BTE) SeriesThe HagueNetherlands Ministry of Economic AffairsSearch in Google Scholar

Noyons, E.C.M., & van Raan, A.F.J. (1994). Bibliometric cartography of scientific and technological developments of an R&D field. The case of optomechatronics. Scientometrics, 30(1), 157–173.NoyonsE.C.M.van RaanA.F.J.1994Bibliometric cartography of scientific and technological developments of an R&D field. The case of optomechatronicsScientometrics301157173Search in Google Scholar

Noyons, E.C.M., van Raan, A.F.J., Grupp, H., & Schmoch, U. (1994). Exploring the science and technology interface: Inventor-author relations in laser medicine research. Research Policy, 23(4), 443–457.NoyonsE.C.M.van RaanA.F.J.GruppH.SchmochU.1994Exploring the science and technology interface: Inventor-author relations in laser medicine researchResearch Policy234443457Search in Google Scholar

Noyons, E.C.M., Buter, R.K., van Raan, A.F.J., Schmoch, U., Heinze, T., Hinze, S., & Rangnow, R. (2003). Mapping excellence in science and technology across Europe: Nanoscience and nanotechnology. Report of project EC-PPN CT-2002-0001 to the European Commission. Leiden: Centre for Science and Technology Studies (CWTS), Leiden University.NoyonsE.C.M.ButerR.K.van RaanA.F.J.SchmochU.HeinzeT.HinzeS.RangnowR.2003Mapping excellence in science and technology across Europe: Nanoscience and nanotechnologyReport of project EC-PPN CT-2002-0001 to the European CommissionLeidenCentre for Science and Technology Studies (CWTS), Leiden UniversitySearch in Google Scholar

Packer, A., & Webster, K. (1996). Patenting culture in science: Reinventing the scientific wheel of credibility. Science, Technology and Human Values, 21(4), 427–453.PackerA.WebsterK.1996Patenting culture in science: Reinventing the scientific wheel of credibilityScience, Technology and Human Values214427453Search in Google Scholar

Park, H., & Yoon, J. (2014). Assessing coreness and intermediarity of technology sectors using patent co-classification analysis: The case of Korean national R&D. Scientometrics, 98(2), 853–890.ParkH.YoonJ.2014Assessing coreness and intermediarity of technology sectors using patent co-classification analysis: The case of Korean national R&DScientometrics982853890Search in Google Scholar

Perkmann, M., Fini, R., Ross, J.M., Salter, A., Silvestri, C., & Tartari, V. (2015). Accounting for universities’ impact: Using augmented data to measure academic engagement and commercialization by academic scientists. Research Evaluation, 24(4), 380–391.PerkmannM.FiniR.RossJ.M.SalterA.SilvestriC.TartariV.2015Accounting for universities’ impact: Using augmented data to measure academic engagement and commercialization by academic scientistsResearch Evaluation244380391Search in Google Scholar

Ribeiro, L.C., Kruss, G., Britto, G., Bernardes, A.T., & Albuquerque, E.D.E. (2014). A methodology for unveiling global innovation networks: Patent citations as clues to cross border knowledge flows. Scientometrics, 101(1), 61–83.RibeiroL.C.KrussG.BrittoG.BernardesA.T.AlbuquerqueE.D.E.2014A methodology for unveiling global innovation networks: Patent citations as clues to cross border knowledge flowsScientometrics10116183Search in Google Scholar

Rodriguez, A., Kim, B., Turkoz, M., Lee, J.M., Coh, B.Y., & Jeong, M.K. (2015). New multi-stage similarity measure for calculation of pairwise patent similarity in a patent citation network. Scientometrics, 103(2), 565–581.RodriguezA.KimB.TurkozM.LeeJ.M.CohB.Y.JeongM.K.2015New multi-stage similarity measure for calculation of pairwise patent similarity in a patent citation networkScientometrics1032565581Search in Google Scholar

Schmoch, U. (1993). Tracing the knowledge transfer from science to technology as reflected in patent indicators. Scientometrics, 26(1), 193–211.SchmochU.1993Tracing the knowledge transfer from science to technology as reflected in patent indicatorsScientometrics261193211Search in Google Scholar

Sherwin, C.W., & Isenson, R.S. (1967). Project Hindsight - A defense department study of the utility of research. Science, 156(3782), 1571–1577.SherwinC.W.IsensonR.S.1967Project Hindsight - A defense department study of the utility of researchScience156378215711577Search in Google Scholar

Squicciarini, M., Dernis, H., & Crisculo, C. (2013). Measuring patent quality: Indicators of technological and economic value. OECD Science, Technology and Industry Working Papers, 2013/03, OECD Publishing. Retrieved on November 29, 2016, from http://dx.doi.org/10.1787/5k4522wkw1r8-en.SquicciariniM.DernisH.CrisculoC.2013Measuring patent quality: Indicators of technological and economic valueOECD Science, Technology and Industry Working Papers2013/03OECD Publishing. Retrieved on November 29, 2016fromhttp://dx.doi.org/10.1787/5k4522wkw1r8-enSearch in Google Scholar

Small, H.G., Boyack, K.W., & Klavans, R. (2014). Identifying emerging topics in science and technology. Research Policy, 43(8),1450–1467.SmallH.G.BoyackK.W.KlavansR.2014Identifying emerging topics in science and technologyResearch Policy43814501467Search in Google Scholar

Stankovich, S., Dikin, D.A., Dommett, G.H.B., Kohlhaas, K.M., Zimney, E.J., Stach, E.A., Piner, R.D., Nguyen, S.T., & Ruoff, R.S. (2006). Graphene-based composite materials. Nature, 442(7100), 282–286.StankovichS.DikinD.A.DommettG.H.B.KohlhaasK.M.ZimneyE.J.StachE.A.PinerR.D.NguyenS.T.RuoffR.S.2006Graphene-based composite materialsNature4427100282286Search in Google Scholar

Sternitzke, C. (2010). Knowledge sources, patent protection, and commercialization of pharmaceutical innovations. Research Policy, 39(6), 810–821.SternitzkeC.2010Knowledge sources, patent protection, and commercialization of pharmaceutical innovationsResearch Policy396810821Search in Google Scholar

Tijssen, R.J.W., Buter, R.K., & van Leeuwen, T.N. (2000). Technological relevance of science: Validation and analysis of citation linkages between patents and research papers. Scientometrics, 47(2), 389–412.TijssenR.J.W.ButerR.K.van LeeuwenT.N.2000Technological relevance of science: Validation and analysis of citation linkages between patents and research papersScientometrics472389412Search in Google Scholar

Tijssen, R.J.W. (2001). Global and domestic utilization of industrial relevant science: Patent citation analysis of science-technology interactions and knowledge flows. Research Policy, 30(1), 35–54.TijssenR.J.W.2001Global and domestic utilization of industrial relevant science: Patent citation analysis of science-technology interactions and knowledge flowsResearch Policy3013554Search in Google Scholar

Trajtenberg, M. (1990). A penny for your quotes: Patent citations and the value of innovations. RAND Journal of Economics, 21(1), 172–187.TrajtenbergM.1990A penny for your quotes: Patent citations and the value of innovationsRAND Journal of Economics211172187Search in Google Scholar

Upham, S.P., & Small, H. (2010). Emerging research fronts in science and technology: Patterns of new knowledge development. Scientometrics, 83(1), 15–38.UphamS.P.SmallH.2010Emerging research fronts in science and technology: Patterns of new knowledge developmentScientometrics8311538Search in Google Scholar

van Looy, B., Debackere, K., Callaert, J., Tijssen, R., & van Leeuwen, T. (2006). Scientific capabilities and technological performance: An exploration of emerging industrial relevant research domains. Scientometrics, 66(2), 295–310.van LooyB.DebackereK.CallaertJ.TijssenR.van LeeuwenT.2006Scientific capabilities and technological performance: An exploration of emerging industrial relevant research domainsScientometrics662295310Search in Google Scholar

van Looy, B., Magerman, T., & Debackere, K. (2007). Developing technology in the vicinity of science: An examination of the relationship between science intensity (of patents) and technological productivity within the field of biotechnology. Scientometrics, 70(2), 441–458.van LooyB.MagermanT.DebackereK.2007Developing technology in the vicinity of science: An examination of the relationship between science intensity (of patents) and technological productivity within the field of biotechnologyScientometrics702441458Search in Google Scholar

van Raan, A.F.J. (2015). Dormitory of physical and engineering sciences: Sleeping beauties may be sleeping innovations. PLoS ONE, 10(10), e0139786.van RaanA.F.J.2015Dormitory of physical and engineering sciences: Sleeping beauties may be sleeping innovationsPLoS ONE1010e0139786Search in Google Scholar

van Raan, A.F.J. (2016). Sleeping beauties cited in patents: Is there also a dormitory of inventions? To be published, preprint retrieved on November 29, 2016, from https://arxiv.org/abs/1604.05750.van RaanA.F.J.2016Sleeping beauties cited in patents: Is there also a dormitory of inventions?To be published, preprint retrieved on November 29, 2016, fromhttps://arxiv.org/abs/1604.05750Search in Google Scholar

van Vianen, B.G., Moed, H.F., & van Raan, A.F.J. (1990). An exploration of the science base of recent technology. Research Policy, 19(1), 61–81.van VianenB.G.MoedH.F.van RaanA.F.J.1990An exploration of the science base of recent technologyResearch Policy1916181Search in Google Scholar

Verbeek, A., Debackere, K., Luwel, M., Andries, P., Zimmermann, E., & Deleus, F. (2002). Linking science to technology: Using bibliographic references in patents to build linkage schemes. Scientometrics, 54(3), 399–420.VerbeekA.DebackereK.LuwelM.AndriesP.ZimmermannE.DeleusF.2002Linking science to technology: Using bibliographic references in patents to build linkage schemesScientometrics543399420Search in Google Scholar

Verhoeven, D., Bakker, J., & Veugelers, R. (2016). Measuring technological novelty with patentbased indicators. Research Policy, 45(3), 707–723.VerhoevenD.BakkerJ.VeugelersR.2016Measuring technological novelty with patentbased indicatorsResearch Policy453707723Search in Google Scholar

Wada, T. (2016). Obstacles to prior art searching by the trilateral patent offices: Empirical evidence from International Search Reports. Scientometrics, 107(2), 701–722.WadaT.2016Obstacles to prior art searching by the trilateral patent offices: Empirical evidence from International Search ReportsScientometrics1072701722Search in Google Scholar

Walter, S.G., Schmidt, A., & Walter, A. (2016). Patenting rationales of academic entrepreneurs in weak and strong organizational regimes. Research Policy, 45(2), 533–545.WalterS.G.SchmidtA.WalterA.2016Patenting rationales of academic entrepreneurs in weak and strong organizational regimesResearch Policy452533545Search in Google Scholar

Waltman, L., van Raan, A.F.J., & Smart, S. (2014). Exploring the relationship between the engineering and physical sciences and the health and life sciences by advanced bibliometric methods. PLoS ONE, 9(10), e111530.WaltmanL.van RaanA.F.J.SmartS.2014Exploring the relationship between the engineering and physical sciences and the health and life sciences by advanced bibliometric methodsPLoS ONE910e111530Search in Google Scholar

Wang, X., Zhang, X., & Xu, S. (2011). Patent co-citation networks of Fortune 500 companies. Scientometrics, 88(3), 761–770.WangX.ZhangX.XuS.2011Patent co-citation networks of Fortune 500 companiesScientometrics883761770Search in Google Scholar

Wang, Y., Roijakkers, N., & Vanhaverbeke, W. (2014). How fast do Chinese firms learn and catch up? Evidence from patent citations. Scientometrics, 98(1), 743–761.WangY.RoijakkersN.VanhaverbekeW.2014How fast do Chinese firms learn and catch up? Evidence from patent citationsScientometrics981743761Search in Google Scholar

Winnink, J.J., Tijssen, R.J.W., & van Raan, A.F.J. (2013). The discovery of introns: Analysis of the science-technology interface. In S. Hinze, & A. Lottmann (Eds.), Translational Twists and Turns: Science as a Socio-economic Endeavor. Proceedings of the 18th International Conference on Science and Technology Indicators (pp. 427–438). Berlin, Institute for Research Information and Quality Assurance (iFQ). Retrieved on November 29, 2016, from http://www.forschungsinfo.de/sti2013/download/sti_2013_proceedings.pdf.WinninkJ.J.TijssenR.J.W.van RaanA.F.J.2013The discovery of introns: Analysis of the science-technology interfaceHinzeS.LottmannA.Translational Twists and Turns: Science as a Socio-economic Endeavor. Proceedings of the 18th International Conference on Science and Technology Indicators427438BerlinInstitute for Research Information and Quality Assurance (iFQ). Retrieved on November 29, 2016fromhttp://www.forschungsinfo.de/sti2013/download/sti_2013_proceedings.pdfSearch in Google Scholar

Winnink, J.J., & Tijssen, R.J.W. (2014). R&D dynamics and scientific breakthroughs in HIV/AIDS drugs development: The case of integrase inhibitors. Scientometrics, 101(1), 1–16.WinninkJ.J.TijssenR.J.W.2014R&D dynamics and scientific breakthroughs in HIV/AIDS drugs development: The case of integrase inhibitorsScientometrics1011116Search in Google Scholar

Winnink, J.J., & Tijssen, R.J.W. (2015). Early stage identification of breakthroughs at the interface of science and technology: Lessons drawn from a landmark publication. Scientometrics, 102(1), 113–114.WinninkJ.J.TijssenR.J.W.2015Early stage identification of breakthroughs at the interface of science and technology: Lessons drawn from a landmark publicationScientometrics1021113114Search in Google Scholar

Yang, G.C., Li, G., Li, C.Y., Zhao, Y.H., Zhang, J., Liu, T., Chen, D.Z., & Huang, M.H. (2015). Using the comprehensive patent citation network (CPC) to evaluate patent value. Scientometrics, 105(3), 1319–1346.YangG.C.LiG.LiC.Y.ZhaoY.H.ZhangJ.LiuT.ChenD.Z.HuangM.H.2015Using the comprehensive patent citation network (CPC) to evaluate patent valueScientometrics105313191346Search in Google Scholar

Zarrin, H., Higgins, D., Jun, Y., Chen, Z.W., & Fowler, M. (2011). Functionalized graphene oxide nanocomposite membrane for low humidity and high temperature proton exchange membrane fuel cells. Journal of Physical Chemistry C, 115(42), 20774–20781.ZarrinH.HigginsD.JunY.ChenZ.W.FowlerM.2011Functionalized graphene oxide nanocomposite membrane for low humidity and high temperature proton exchange membrane fuel cellsJournal of Physical Chemistry C115422077420781Search in Google Scholar

Recommended articles from Trend MD

Plan your remote conference with Sciendo