1. bookAHEAD OF PRINT
Journal Details
License
Format
Journal
First Published
30 Mar 2017
Publication timeframe
4 times per year
Languages
English
access type Open Access

A New Citation Recommendation Strategy Based on Term Functions in Related Studies Section

Published Online: 09 May 2021
Page range: -
Received: 03 Nov 2020
Accepted: 07 Apr 2021
Journal Details
License
Format
Journal
First Published
30 Mar 2017
Publication timeframe
4 times per year
Languages
English
AbstractPurpose

Researchers frequently encounter the following problems when writing scientific articles: (1) Selecting appropriate citations to support the research idea is challenging. (2) The literature review is not conducted extensively, which leads to working on a research problem that others have well addressed. This study focuses on citation recommendation in the related studies section by applying the term function of a citation context, potentially improving the efficiency of writing a literature review.

Design/methodology/approach

We present nine term functions with three newly created and six identified from existing literature. Using these term functions as labels, we annotate 531 research papers in three topics to evaluate our proposed recommendation strategy. BM25 and Word2vec with VSM are implemented as the baseline models for the recommendation. Then the term function information is applied to enhance the performance.

Findings

The experiments show that the term function-based methods outperform the baseline methods regarding the recall, precision, and F1-score measurement, demonstrating that term functions are useful in identifying valuable citations.

Research limitations

The dataset is insufficient due to the complexity of annotating citation functions for paragraphs in the related studies section. More recent deep learning models should be performed to future validate the proposed approach.

Practical implications

The citation recommendation strategy can be helpful for valuable citation discovery, semantic scientific retrieval, and automatic literature review generation.

Originality/value

The proposed citation function-based citation recommendation can generate intuitive explanations of the results for users, improving the transparency, persuasiveness, and effectiveness of recommender systems.

Keywords

Introduction

The amount of scientific literature has been increasing exponentially in recent years. For example, publications in Computer Science in Web of Science have been growing from 396 in 1995 to 50,644 in 2021

https://apps.webofknowledge.com/UA_GeneralSearch_input.do?product=UA&SID=7F4FLVerSQulGPpm4AT&search_mode=GeneralSearch

. Due to the explosion of scientific literature, it has become more and more time-consuming for readers to review related literature and decide which article to cite. Fortunately, citation recommendation (CR) has been proved to be useful in helping users to decide which papers should be cited from their reading list (Beel & Dinesh, 2017; He et al., 2010; Liu, Yan, & Yan, 2013; McNee et al., 2002; Raamkumar, Foo, & Pang, 2016; Strohman, Croft, & Jensen, 2007). A CR system suggests previous studies be reviewed and cited for new research articles. It helps researchers to cite appropriate previous studies and to avoid missing important literature. Usually, an automatic CR system accepts a research topic and provides a list of publications that can be cited. Unlike traditional search approaches offered by search engines and digital libraries, CR systems focus on finding the relevant publications rather than any texts or pages similar to the topics (Strohman, Croft, & Jensen, 2007).

Research has applied semantic information and non-semantic information for CR. Semantic information such as citation function, citation sentiment, citation importance, term function, is usually included in the citation context, which is defined as a sequence of words appearing around a citation placeholder. For example, He et al. (2010) designed a non-parametric probabilistic CR model which measured the context-based relevance between a citation context and a document to be recommended and automatically identified citation contexts in a manuscript where citations were needed by applying contextual information (He et al., 2011). Tang and Zhang (2009) discovered topical aspects of the citation contexts of each paper and recommended papers based on the discovered topic distribution. Zarrinkalam and Kahani (2013) used citation context as a textual feature to enhance the performance of CR tasks. Duma et al. (2016a) and Duma (2019) integrated core scientific concepts classification and discourse annotation into context-based CR.

Non-semantic information mainly focuses on the relationship between articles and authors, such as citation network, author collaboration, co-occurrence. For example, McNee et al. (2002) created a rating matrix using the citation network between papers. Chen et al. (2011) also leveraged such citation-network-based methodology but named it citation authority diffusion (CAD). Livne et al. (2014) developed a CR system that took the author similarity, venue relevancy, and co-citation into consideration for augmenting sparse citation networks. Son and Kim (2018) proposed a multilevel citation network-based scientific paper recommender system by comparing all the indirectly linked papers to the paper of interest. It can recommend both the research topic and the academic theory related papers. Some scholars also combined semantic information with non-semantic information to enhance the CR performance (Bethard & Jurafsky, 2010; Ebesu & Fang, 2017; Jeong et al. 2019; Strohman, Croft, & Jensen, 2007).

Current CR systems could be further improved by considering new factors. One promising approach to enhance CR service is to recommend citations in the related studies sections. Citation content analysis results showed that more than 60% of the references and the most highly cited articles appeared in the introduction and literature review sections (collectively referred to as related studies sections in this paper) of the citing papers (Ding et al., 2013). Recommending citations in related work sections has been discussed as meaningful to fascinating literature review writing (Huang et al., 2014; Livne et al., 2014). However, little research has been conducted on this valuable task, except for (Sesagiri Raamkumar, Foo, & Pang, 2015; 2016), who constructed a recommender system for providing a shortlist of papers based on article type preference, coverage, citation count, and user-specified prospective keywords to assist researchers’ literature review writing. The drawback is that users still need to spend much energy on how to organize these articles. We believe that a CR algorithm which can recommend papers categorized by their term functions in related studies sections will save users a lot of time.

In this paper, the term function refers to the semantic role of a segment or a paragraph in the related studies section (Cheng, 2015). Term function has been proved useful for academic retrieval and recommendation (Li, Cheng, & Lu, 2017). For example, when people conduct research on citation context recognition (CCR), the related work may involve problem statement on CCR, the CCR methods, datasets used in CCR, CCR related tools, CCR evaluation method, and the applications of CCR. Therefore, citation recommendation services can provide recommendation lists according to these “term functions” based on users’ requirements, this will be more likely to meet their information needs. This paper focuses on this innovative and challenging task, aiming to explore a more efficient CR strategy thus improving scholars’ reading experience. Compared with previous studies, the contributions of this paper mainly include three aspects:

We investigate the citation organization patterns in the related work sections. Scientific articles tend to follow a particular style of organizing sections and paragraphs (Luong, Nguyen, & Kan, 2012). To understand how researchers usually organize related studies, we develop a term function annotation scheme at the paragraph-level. An annotation experiment showed that there were four common patterns of organizing literature in the related work sections.

We propose a term function-based citation recommendation framework to recommend articles for users based on their expected term function of a specific paragraph in the related work sections. This is the first framework to introduce term function into citation recommendation task to the best of our knowledge.

We conduct several experiments on “real-world” datasets obtained from ACL Anthology to evaluate the impacts of the term function factors and the performance of the proposed methods. The experimental results show that our proposed Word2vec-VSM model with term functions achieve the best recommendation performance, indicating the effectiveness of term functions on citation recommendation.

The remainder of this paper is structured as follows: Section 2 reviews the related work. Section 3 introduces the proposed framework for term function-based citation recommendation, including the term function annotation experiment and the recommendation algorithms. Section 4 describes our recommendation experiments based on the ACL Anthology dataset and reports the results. Section 5 concludes the paper and discusses the future study.

Related work

In this study, we assume that paragraphs in related work sections could be organized by the term function of the citation sentences in them. Therefore, our work in this paper first annotates each paragraph in related work sections with a particular term function, then combines with paragraph content and term function weight to recommend citations for this paragraph. Given this focus, we review scientific literature related to term function and citation recommendation.

Term function

Initially, term function refers to the semantic role that a term plays in the scientific literature (Cheng, 2015). It also represents the sentence's function where the term belongs if all the citation sentences in a paragraph in the related work section share a specific “term function” (for example, the research method related to the citing article). We believe this paragraph will be easier to understand because all the citations focus on a specific term function (of a topic).

Except for the term function “research method” we mentioned above, there are many other term functions, such as research topic, technology dataset, application, evaluation. However, according to previous definitions, term functions could be classified into different schemes, as shown in table 1.

Classification of term function (Li, Cheng, & Lu, 2017).

Classification of term functionAuthors
Head, goal, method, otherKondo (2009)
Technology, effectNanba, Kondo, & Takezawa (2010)
Focus, technique, domainGupta & Manning (2012)
Technique, applicationTsai, Kundu, & Roth (2013)
Method, task, otherHuang & Wan (2013)
Domain-independent: Research topic, Research methodCheng (2015)
Domain-related: Case, tool, dataset, etc.

However, the classification of the term function is still obscure and not uniform. Considering the specific research problem in this article, we took some categories from existing literature and proposed three new categories of term function, which we will introduce in the next section. As a novel task, although several attempts have been made on term function analysis, the topic still remains to be developed, especially the gap between term function analysis and its applications, which forms the exact initiation of our study.

Citation recommendation

Citation recommendation for research papers was first introduced by McNee et al. (2002). Since then, there has been a rich line of research on this topic. For example, Strohman, Croft, and Jensen (2007) combined the content of previous literature and its citation network to recommend relevant material that a given article should cite. They found that this mixed method performed much better than the traditional text-similarity approach. Tang and Zhang (2009) proposed a topic distribution discovery-based CR model that performed well on sentence-level CR on the NIPS dataset. They argued that CR could be integrated into academic search systems to improve service. He et al. (2011) presented a CR system that automatically identified contexts where citations were needed in a manuscript and recommended appropriate citations. In 2013, Kates-harbeck and Haggblade (2013) used a machine learning method that utilized context-based features and text-based features to rank references for a given short text. Recently, with the growth of scholarly data and the development of deep learning, neural networks were transplanted to CR problems, aiming to train more robust models and enhance CR performance (Ebesu & Fang, 2017; Huang et al., 2015).

In terms of the context scope based on which a citation recommendation list was generated, the CR task can be divided into two aspects: local citation recommendation (LCR) (Tang, Wan, & Zhang, 2014; Yang et al., 2019) and global citation recommendation (GCR) (Tang, Wan, & Zhang, 2014; Ayala-Gomez et al., 2018).

Local citation recommendation

Local citation recommendation aims to recommend citations for a specific context where citations are needed, which is also called context-aware citation recommendation (Tang, Wan, & Zhang, 2014). Here, the specific context can be one sentence or several sentences; in this paper, it represents all the sentences in a paragraph in the related work sections. Since contexts contain rich semantic information, content-based approaches were usually used in LCR.

He et al. (2010) proposed an effective context-aware citation recommendation approach, which designed a non-parametric probabilistic model to measure the relevance between a citation context and a document. In a similar task, the translation model was used to translate the query terms in the citation context, which bridged the vocabulary gap between the citation context and recommended document (Lu et al., 2011; He et al., 2012). Rokach et al. (2013) presented a supervised learning method utilizing three types of features (general features, author-aware features, and context-aware features) to recommend citations for a given citation context. This approach had been applied for citation recommendation service in CiteSeerX digital library. Except for these features, time or publication date has also been proven to be important in citation recommendation (Gao, 2016; Jiang, 2015; Jiang, Liu, & Gao, 2014; 2015). To provide a personalized citation recommendation service, Liu, Yan, and Yan (2013) combined user profiles with context-aware citation recommendation. Experimental results showed that the proposed strategy outperformed language model-based and translation model-based algorithms. Other research explored the effectiveness of citation networks (Jiang et al., 2018; Livne et al., 2014; Son and Kim, 2018), core scientific concepts (Duma et al., 2016) in local citation recommendation. (Duma & Klein, 2014) introduced a citation resolution method to evaluate context-based citation recommendation systems. Citation context, as the primary evidence to recommend citations, has become a theorem in this field (Bhagavatula et al., 2018).

Recently, with the popularity of deep learning in both academia and industry, different deep learning algorithms have been applied to citation recommendation. (Huang et al., 2014) represented words and documents by learning simultaneously from citation context and cited document pairs; the probabilistic neural model was used to estimate the probability of words appeared in the citation context under a candidate reference paper. This approach achieved 5% improvement on Recall than the translation model. Ebesu and Fang (2017) proposed a neural citation network (NCN) that can model the semantic composition of citation contexts and corresponding cited documents titles by exploiting author relations. This method improved several state-of-the-art baselines on all metrics (Recall, MAP, MRR, and NDCG) by 13–16%.

Global citation recommendation

Unlike LCR, global citation recommendation targets on recommending a reference list for a given paper (Tang, Wan, & Zhang, 2014). However, only focus on finding the relevant papers rather than the essential papers can be a drawback of GCR because users need to evaluate the quality of the recommended papers (Chen et al. 2011). This is challenge for new researchers.

To bridge this gap, Küçüktunç et al. (2012; 2013; 2015) recommended a diverse reference list which allowed the users to reach either old, well-cited, well-known research papers or recent, less known ones. To improve users’ experience, Wu et al. (2012) proposed to recommend references based on their information needs, such as publication time preference, self-citation preference, co-citation preference, and publication reputation preference. However, most existing research used a single data source for the recommendation, which failed to meet users’ information needs about different aspects in writing (Zarrinkalam & Kahani, 2012). Therefore, Zarrinkalam and Kahani (2012) introduced a strategy to recommend citations based on multiple linked data sources, which enriched the background data of recommender systems thus enhancing the recommendations.

Kates-harbeck and Haggblade (2013) recommended references for a given abstract with a set of key technical words, an author list, and publication data using a two-stage methodology. In the first stage, they trained a classifier to rank a candidate reference list based on the paper score; in the second stage, they re-ranked the scored candidate papers using connectivity information. Their experiment results showed that text-based features were most effective in rankings. Caragea et al. (2013) compared the performance of the collaborative filtering-based (CF) approach and singular value decomposition-based (SVD) approach on GCR using CiteSeer dataset, finding that SVD performed better than CF since SVD can easily incorporate additional information.

As a benefit of different citation recommendation tasks and algorithms mentioned above, many citation recommendation systems have been developed, for example, ActiveCite (Zhou, 2010), CiteSight (Livne et al., 2014), RefSeer (Huang et al., 2014), ClusCite (Ren et al., 2014), DiSCern (Chakraborty et al., 2015), and Rec4LRW (Sesagiri Raamkumar, Foo, & Pang, 2015). Due to the usefulness but challenge of this task, citation recommendation is attracting more and more attention from researchers.

Proposed approach

The main objective of our work is to recommend citations in the related work sections and verify the effectiveness of term function in this recommendation task. In terms of the first objective, we notice that the literature review sections are usually organized in specific patterns to address different aspects related to an article, thus recommending citations by following these patterns might customize users’ personal information needs. As for the second objective, we observe that a specific term in a segment or a paragraph might indicate its role in related work sections; in this paper, we define it as term function, which is valuable information for CR.

The main problems addressed in our work are: (1) identify the paragraph organization patterns in related work sections, and (2) based on these patterns, recommend citations by involving term functions as the weighting parameter. Fig. 1 presents the framework of our proposed approach to solve the problems. Given a user topic, the CR system first retrieves a list of publications as candidates for recommendation. It then assigns weights to the candidates based on their term functions for reranking. At last, the top-ranked candidates are provided to the user.

Figure 1

Framework of term function-based citation recommendation

Paragraph organization patterns analysis in related work sections

Typically, researchers organize related literature by following specific patterns (e.g. based on the similarity, based on different topics, based on the published time, or based on the roles) rather than roughly putting a list of relevant literature together. Meanwhile, the literature is organized to demonstrate previous research problems, methods, datasets, and applications, defined as term functions in our study, that are related to a research. We propose a term function classification scheme used to annotate a real-world dataset acquired from ACL Anthology. After that, we analyze the annotation results to identify the paragraph organization patterns in related work sections.

Term function classification scheme in paragraph level

Term function means the semantic role that a term plays in the scientific literature (Cheng, 2015). The term's function also reflects the function of the citation context it locates in the scientific literature. For example, support vector machine (SVM) is the core research problem in (Li, Wang, & Wang, 2010) but the method to solve image classification problems in (Melgani & Bruzzone, 2004). Therefore, the paper “A Tutorial on Support Vector Machines for Pattern Recognition” (Burges, 1998) is cited as a related research problem in (Li, Wang, & Wang, 2010) but cited as a related method to solve image classification problems in (Melgani & Bruzzone, 2004). In other words, citations could be recommended according to the “term function” needed in the citing article, supposing the related literature in the citing is organized by these term functions. To verify this hypothesis, we develop a term function classification scheme by combining categories from existing literature and three new categories, which we draw from a pilot study, as shown in Table 2. The reason that we add three new functions is existing categories of term functions are proposed for sentence-level. Paragraphs have more complex structures than sentences, and they are usually organized by different roles in the related studies section. For example, a paragraph discusses how the method used in the original paper has been applied to solve other problems; we define this pattern as “Method+ Problem.” On the other hand, a paragraph may also focus on discussing how other relevant methods have solved the research problem mentioned in the original paper except for the method proposed in the original paper; we define this pattern as “Problem+ Method.” We also define another function, “Topic-irrelevant,” since some paragraphs that served as background knowledge are not closely relevant to the original paper's topics. Fig. 3 demonstrates that the three new term functions frequently appear in the related studies section.

Term functions in the related studies section.

CategorySourceDescription
ApplicationTsai et al. (2013)Describes existing application of the core problem and method in this article
DatasetCheng (2015)Describes related datasets to this article
EvaluationCheng (2015)Describes related evaluation methods to this article
MethodHuang & Wan (2013)Describes previous work related to the core method of the article
Method+ ProblemNewDescribes the core method of the article and introduces what problems it can be used to solve
ProblemKondo et al. (2009)Describes previous work related to the core research problem of the article
Problem+ MethodNewDescribes the core research problem of the article and introduces the existing method to the problem
ToolCheng (2015)Describes related tools used in this article
Topic-irrelevantNewDescribes previous work not very relevant
Annotation

We collected a dataset consisting of 238, 109, and 184 research papers in sentiment analysis, information extraction, and recommender systems, respectively. We manually downloaded the pdf files from ACL Anthology and placed them in three separate folders. We converted the pdf files into text format using Apache PDFBox

https://pdfbox.apache.org/

. We then extracted the title, abstract, and related work section information for annotation.

Paragraph organization pattern annotation is difficult since it requires extensive domain knowledge. To ensure the data quality, we invited two Ph.D. students who are familiar with sentiment analysis, information extraction, and recommender systems to annotate the data. Before annotation, they were required to pre-annotate ten articles under each folder to understand the annotation guideline, shown as follows:

Firstly, read the title and abstract to get the research problems and research methods of the article.

Secondly, find the paragraph that needs to be annotated (ignore paragraphs without citations), read the paragraph's contents to get the features that indicate its term function, label the paragraph with a category of term function mentioned above.

Avoid subjective judgment during annotation.

Fig. 2 describes an annotation example, which was saved into the text format. The pairwise Kappa coefficients (Viera & Garrett, 2005) were applied to calculate the inter-annotator agreement. Cohen's kappa scores were 0.778, 0.871, and 0.748 on sentiment analysis, information extraction, and recommender systems, respectively. These are satisfactory according to the existing evaluation standard (Viera & Garrett, 2005).

Figure 2

An annotation example of citation function in paragraph level.

Paragraph organization patterns analysis

To investigate how authors organized literature in related work sections, we conducted a statistical analysis of the annotation results. Fig. 3 represents the distribution of term functions under each category over different topics. Based on the results, most paragraphs belong to problem+ method, problem, method+ problem, and method, with a percentage of 38.2%, 29.5%, 15.6%, and 6.0% on average, respectively, but with a slight difference between different topics. The result supports the hypothesis that authors usually pay more attention to the research problems and methods relevant to their topics while writing a literature review. In other words, they tend to focus on investigating who was involved in studying the problems and methods, how they described the problems and methods, what methods have been used to solve the problem, and how the method was applied to other problems. Existing datasets, evaluation methods, tools, and applications related to their research were also occasionally involved in two or more independent paragraphs. In addition, some topic-irrelevant materials were referred to in a few paragraphs, suggesting that the research topic is novel, and existing literature in this area was lacking.

Figure 3

Statistical analysis of the term function distribution in three fields.

The findings enlighten us to conduct citation recommendation in the related work sections by considering these information needs. In this paper, we explored this innovative citation recommendation task based on four major term functions: problem+ method, problem, method+ problem, and method.

Term function-based citation recommendation

As discussed previously, the term function-based citation recommendation is a kind of global citation recommendation at the paragraph level. Users are required to input the topic (e.g. citation recommendation, translation model) and the term function (e.g. problem+ method, method+ problem). Our approach will recommend a list of citations that are both topic and term function relevant.

Problem definition

Definition 1 (Original document collection). Given a document d with t and a as its title and abstract, the document has a set of paragraphs P = {p1, p2, …, pi} in the related works section, the term function of each paragraph pi is defined as li. Therefore, the paragraph set P has a corresponding term function set L = {l1, l2, …, li}. All the original documents together with their paragraph sets and term function sets form the original document collection D.

Definition 2 (Candidate document collection). Given a document c with t and a as its title and abstract, we define its term function as F = {f1, f2, f3, f4}, where f1, f2, f3, f4 refer to problem+ method, problem, method+ problem, and method respectively, which are determined by the term function of its citing paragraphs. All the candidate documents were extracted from the references of the original documents; they form the candidate document collection C.

Definition 3 (Term function-based citation recommendation). Given a paragraph p whose topic is t and expected term function is f, our goal is to estimate the probability of each document in the candidate document collection C to be cited in this paragraph.

Citation recommendation algorithms
BM25 model

BM25 (Robertson & Zaragoza, 2009) has been widely employed in many information retrieval systems and proved to be effective. Recently, it has also been frequently selected as a baseline model in many citation recommendation tasks (Bhagavatula et al., 2018; Ebesu & Fang 2017; Gao, 2016; Jiang, Liu, & Gao, 2015; Ren et al., 2014; Sesagiri Raamkumar, Foo, & Pang, 2015). In this paper, we use BM25 as the baseline method. With the BM25 ranking function, the relevance score of a candidate document d with respect to a paragraph as a query q is calculated as follows: Score(q,d)=inlogNn(qi)+0.5n(qi)+0.5fi(k1+1)fi+k1(1b+bdlavgdl)Score\left( {q,d} \right) = \sum\nolimits_i^n {\log {{N - n\left( {{q_i}} \right) + 0.5} \over {n\left( {{q_i}} \right) + 0.5}}} \cdot {{{f_i} \cdot \left( {{k_1} + 1} \right)} \over {{f_i} + {k_1} \cdot \left( {1 - b + b \cdot {{dl} \over {avgdl}}} \right)}}

Where k1 and b are free hyper-parameters. avgdl and N respectively donate the average document length and the total number of documents in the collection. fi and n(qi) represent the frequencies of a query term qi in a candidate document and the total number of documents containing the query term qi.

Word2vec based vector space model

Word2vec is one of the most widely used word embedding models used to represent texts with numerical vectors (Mikolov et al., 2013). Compared to the term frequency used in the BM25 algorithm, Word2vec can preserve the semantic and syntactic relationship between words. Therefore, we use Word2vec to create vectors for both queries and documents in this paper. We train our Word2vec model with Gensim

https://radimrehurek.com/gensim/models/word2vec.html

, a python package used for topic modeling, text processing, and working with word vector models such as Word2vec and FastText. The Word2vec model is trained on the ACL Anthology Reference Corpus

https://web.eecs.umich.edu/~lahiri/acl_arc.html

, which contains 10,920 academic papers from the ACL Anthology. Finally, we generate vectors of size 300 for the queries and documents. The similarity between a query and the document is calculated by using the vector space model (VSM) with the following equation: Score(q,d)=q*dqdScore\left( {q,d} \right) = {{\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over q} *\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over d} } \over {\left\| {\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over q} } \right\|\left\| {\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over d} } \right\|}}

Where q and d are the vector representations of the query and the document, respectively.

Term function weighting-based recommendation models

In this paper, we explore the influence of term functions in citation recommendation and propose the term function weighting-based recommendation model. Compared with the original BM25 model and Word2vec-based VSM, an essential step of term function weighting-based recommendation models is to compute the weight of each document in the candidate document collection on each of the four term functions.

We define F = (f1, f2, f3, f4) as the weighted term function of each document d in the candidate document collection, where f1, f2, f3, f4 respectively represents the frequency of problem, method, problem+ method, method+ problem. After that, the term function should be added to the three recommendation models to improve performance.

For a document d and a paragraph as a query q, qf donates the expected term function of this query/paragraph. Therefore, we calculate the relevance score on the term function dimension as follow: p(qf,d)=ff1+f2+f3+f4,f{f1,f2,f3,f4}p\left( {{q_f},d} \right) = {f \over {{f_1} + {f_2} + {f_3} + {f_4}}},f \in \left\{ {{f_1},{f_2},{f_3},{f_4}} \right\}

For example, if a user inputs a query q with the term function as problem, one of a document d in the candidate document collection whose term function is F = (4, 2, 8, 1), we compute its term function weight on problem, method, problem+ method, method+problem respectively as 415{4 \over {15}} , 215{2 \over {15}} , 815{8 \over {15}} , 115{1 \over {15}} , where the sum of the weights should adhere to the following constraint: j=14weightj=1(weightj[0,1])\sum\nolimits_{j = 1}^4 {weigh{t_j} = 1\left( {weigh{t_j} \in \left[ {0,1} \right]} \right)}

Finally, the relevance score of a candidate document d with respect to a paragraph as a query q which combines text similarity and term function matching is computed with the following two equations respectively: Score(q,d)=(1+p(qf,d))inlogNn(qi)+0.5n(qi)+0.5fi(k1+1)fi+k1(1b+bdlavgdl)Score\left( {q,d} \right) = \left( {1 + p\left( {{q_f},d} \right)} \right) \cdot \sum\nolimits_i^n {log {{N - n\left( {{q_i}} \right) + 0.5} \over {n\left( {{q_i}} \right) + 0.5}}} \cdot {{{f_i} \cdot \left( {{k_1} + 1} \right)} \over {{f_i} + {k_1} \cdot \left( {1 - b + b \cdot {{dl} \over {avgdl}}} \right)}}Score(q,d)=(1+p(qf,d))q*dqdScore\left( {q,d} \right) = \left( {1 + p\left( {{q_f},d} \right)} \right) \cdot {{\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over q} *\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over d} } \over {\left\| {\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over q} } \right\|\left\| {\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\rightharpoonup$}}\over d} } \right\|}}Equation (5) is for term function weighting-based BM25 model, while equation (6) is for Word2vec-based VSM models with term function weighting.

Pseudo code for the baseline recommendation models or term function weighting-based recommendation models is described in Algorithm 1.

Recommendation with baseline models or term function weighting-based recommendation models

Input:
Candidate paper list
Rank papers by relevance scores
If Year of candidate paper < year of original paper
    Add candidate paper into new recommendation list
Else
    Continue
End if Length of new recommendation list is 30
Output:
New recommendation list
Experiments

In this section, we first introduce the dataset for experiments and the experimental setup to evaluate the recommendation performance of our proposed approach. After that, we report the results of standard measures for information retrieval and recommendation (Recall, Precision, and F1 score). Finally, analysis and discussion are presented based on the experimental results.

Dataset

Since there is no existing standard benchmark dataset with annotated term function at the paragraph level for term function-based citation recommendation, we collect and build our data set from ACL Anthology

http://www.aclweb.org/anthology/

in the domains of information extraction, sentiment analysis, and recommender systems. In total, 531 research articles with full text and their 2,875 reference articles with title, published year, and abstract are included. As mentioned in section 3.1, all the paragraphs in the related work sections of the 531 research articles have been labeled with one of the four term functions. Since many of the reference articles have been co-cited by the original articles, it is easy for us to figure out their term function distributions. For example, the term function distribution of paper “On the recommending of citations for research papers (McNee et al., 2002)” is (3/8, 4/8, 0, 1/8) in terms of (problem+ method, problem, method+ problem, and method), which can be used as weighting parameters. We use the Python NLTK tool

https://www.nltk.org/

to perform text pre-processing like removing numbers, stop words, and stemming. Finally, the experiment dataset is stored in a MySQL database.

In our experiment, we randomly partition the dataset into five subsamples and then perform 5-fold cross-validation on the same partition for our approaches and the baseline methods. At each time, four sets were used as training sets for term function weighting, and the remaining one set was used for testing. Also, a small portion of examples split from the training set was used for validation. We performed the process five times and averaged their performance for evaluation.

Experimental setup

Baseline models. We compute the similarity scores between a paragraph and the candidate recommendation articles with the BM25 model (the default values of b = 0.75 and k1 = 1.2 are applied) and Word2vec based VSM model, respectively. Meanwhile, we restrict the published date of the candidate articles to a year before the original article.

Term function weighting-based recommendation models. In this method, we modify the baseline models by involving term function information and rank the candidate recommendation articles based on the scores computed by Eq. (5).

Moreover, we conduct a comparison study on three fields, including information extraction (300 paragraphs), sentiment analysis (383 paragraphs), and recommender system (510 paragraphs), to analyze the effect of term function on citation recommendation performance in different fields. Therefore, twelve runs in total, including six baseline methods and six term function weighting-based methods. They are BM25 (BM25), word2vec-based VSM (W2V-VSM), BM25 with term function weighting (TFW-BM25), word2vec-based VSM with term function weighting (TFW-W2V-VSM) in information extraction, sentiment analysis, and recommender system, respectively.

Evaluation metrics

Citation recommendation is essentially an information retrieval task. The top-ranked documents are the most important to get the correct recommendation (Wu et al., 2012). Therefore, we employ IR evaluation measures, including Precision, Recall, and F-measure, in our experiments. For a given query (paragraph with its labeled term function) in the test set, we use the original set of references, which were not present while training, as the ground truth (Rokach et al., 2013). In experiments, the number of recommended citations is set as 5, 10, 20, and 30, respectively.

Performance comparison

Table 3, table 4, and table 5 show the result for each compared method on our dataset regarding the three fields, respectively. In the results, the average precision score, recall score, and F1 score in terms of a different number of citations in the recommendation list.

Recommendation performance in information extraction.

MetricsRunsTop 5Top 10Top 20Top 30
PrecisionBM2510.6%12.6%7.1%6.0%
W2V-VSM12.8%12.0%10.8%8.9%
TFW-BM2513.6%16.6%9.6%7.4%
TFW-W2V-VSM16.4%14.9%13.7%13.0%
RecallBM2514.9%35.1%40.0%50.6%
W2V-VSM17.5%38.7%48.9%58.6%
TFW-BM2519.0%46.4%53.6%61.9%
TFW-W2V-VSM23.3%48.7%57.0%66.8%
F1-scoreBM2512.4%18.5%12.1%10.7%
W2V-VSM14.8%18.3%17.7%17.2%
TFW-BM2517.5%24.5%16.3%13.2%
TFW-W2V-VSM19.3%22.4%22.1%21.8%

Recommendation performance in sentiment analysis.

MetricsRunsTop 5Top 10Top 20Top 30
PrecisionBM2511.1%10.5%8.2%7.0%
W2V-VSM13.6%12.9%10.5%9.5%
TFW-BM2512.9%13.0%9.1%8.2%
TFW-W2V-VSM16.7%15.5%13.8%11.0%
RecallBM2515.6%29.6%46.3%59.3%
W2V-VSM19.5%34.3%52.6%67.9%
TFW-BM2518.1%36.7%51.5%69.6%
TFW-W2V-VSM27.7%41.0%59.2%73.8%
F1-scoreBM2513.0%15.5%13.9%12.5%
W2V-VSM16.0%18.7%17.5%16.7%
TFW-BM2515.1%19.2%15.4%17.2%
TFW-W2V-VSM20.8%22.5%22.4%19.0%

Recommendation performance in recommender system.

MetricsRunsTop 5Top 10Top 20Top 30
PrecisionBM2514.3%12.1%7.5%6.4%
W2V-VSM16.4%14.8%12.3%11.7%
TFW-BM2517.1%15.7%10.7%8.3%
TFW-W2V-VSM21.7%19.9%16.2%15.2%
RecallBM2521.3%36.2%44.7%57.4%
W2V-VSM24.7%44.4%54.2%73.1%
TFW-BM2525.5%46.8%63.8%74.5%
TFW-W2V-VSM26.8%50.1%68.9%77.8%
F1-scoreBM2517.1%18.1%12.8%11.5%
W2V-VSM19.7%21.9%20.0%20.2%
TFW-BM2520.5%23.5%18.3%14.9%
TFW-W2V-VSM24.0%28.5%26.2%25.2%

There are some interesting findings from the results:

First, the term function-based methods outperform the baseline methods. For example, the term function-based BM25 methods improve the baseline BM25 methods by 5.0% (average in the three fields, the same in the following description) on F1 score when the number of recommended citations is set as 20 and improve the baseline by 12.9% on recall when the number of recommended citations is set as 30, while the term function-based Word2vec-VSM methods improve the baseline Word2vec-VSM methods by 5.4% on F1 score when the number of recommended citations is set as 20 and improve the baseline by 9.8% on recall when the number of recommended citations is set as 20. The improvement demonstrates that term function is a useful feature in citation recommendation, particularly in the related work sections.

Second, there are no significant differences between different fields on both the baseline methods and the term function-based methods. Intuitively, the styles of organizing related studies in the three fields are similar, as proved in Fig. 3. Therefore, it would be interesting to investigate the literature review from different disciplines such as LIS and economy, or different type of publications such as quantitative and qualitative research.

Third, the Word2vec-VSM model outperforms the BM25 model before and after term function-weighting, indicates the potential of applying term functions to more complex information retrieval such as BERT-based deep learning models to improve both performance and interpretability.

Results analysis and discussion

Our experiment results show that compared to the traditional retrieval models such as BM25 and Word2vec-VSM, which have been proved effective in information retrieval and recommendation, our term function weighting-based strategies recommend a more accurate and structured literature list, demonstrating that term function is an effective feature in citation recommendation, especially when recommending papers for structured literature review. Compared with deep learning-based-recommendation approaches (Zhang et al., 2019), our strategy can generate intuitive explanations of the results for users or system designers, improving system transparency and persuasiveness, trustworthiness, and effectiveness. For example, users can quickly figure out both the content and term function relevance of a recommended item. Not only will this new citation recommendation strategy benefit for semantic scientific information retrieval, but it also benefits automatically structured summarization and literature review generation.

However, there are some limitations to our proposed strategy. In our measurement, we assume that any paper other than the actual citation is not relevant. In fact, there may be multiple papers that provide the same support/evidence and can equally serve as valid citations. Therefore, the above measures probably underestimate the actual performance. It might be interesting to look at which papers other than the one they are citing are relevant. However, that would require subjective and manual relevance judgments. Another is the limited amount of data. Since term function annotation is still a challenging task, it requires field experts and manual work. Although Cheng (2015) is trying to develop automatic term function identification techniques, there is a considerable gap between experiment and application. Therefore, using machine learning models for automatic classification of the term functions will be explored to get more training data. Absolutely, our citation recommendation strategy initiated a research direction of how term function could be used and how to construct a structured literature review system.

Conclusion and future work

This paper proposed a new citation recommendation strategy based on term functions in the related studies section. Based on the hypothesis that researchers usually organize citations in the related work sections with some patterns, we develop a term function annotation scheme at the paragraph-level. An annotation experiment showed that there were four common patterns of organizing literature in the related work sections. Following this theory, we develop a term function-based citation recommendation framework to recommend users’ articles based on their assigned term function of a specific paragraph in the related work sections. Using the “real-world” dataset collected from ACL Anthology, we designed a recommendation experiment in three filed: information extraction, recommender system, and sentiment analysis, with the BM25 model and Word2vec-VSM model as the baseline method. The experiment results show that our proposed citation recommendation strategies outperform all the baseline methods in terms of different evaluation metrics, demonstrating that term function is an effective feature in citation recommendation, especially when recommending papers for structured literature review.

In the future, we will try to develop algorithms to automatically build large-scale data collections with term function for each paragraph. In this way, we can combine the state-of-the-art contextual word embeddings such as SCIBERT (Beltagy, Lo, & Cohan, 2019) with term functions to improve the recommendation performance and provide explainable recommendation results. We will also explore the usefulness of some other features, such as citing time and citation location. More practically, we will implement a recommender system that helps researchers write structured literature reviews more efficiently based on our citation recommendation strategy.

Figure 1

Framework of term function-based citation recommendation
Framework of term function-based citation recommendation

Figure 2

An annotation example of citation function in paragraph level.
An annotation example of citation function in paragraph level.

Figure 3

Statistical analysis of the term function distribution in three fields.
Statistical analysis of the term function distribution in three fields.

Term functions in the related studies section.

CategorySourceDescription
ApplicationTsai et al. (2013)Describes existing application of the core problem and method in this article
DatasetCheng (2015)Describes related datasets to this article
EvaluationCheng (2015)Describes related evaluation methods to this article
MethodHuang & Wan (2013)Describes previous work related to the core method of the article
Method+ ProblemNewDescribes the core method of the article and introduces what problems it can be used to solve
ProblemKondo et al. (2009)Describes previous work related to the core research problem of the article
Problem+ MethodNewDescribes the core research problem of the article and introduces the existing method to the problem
ToolCheng (2015)Describes related tools used in this article
Topic-irrelevantNewDescribes previous work not very relevant

Recommendation with baseline models or term function weighting-based recommendation models

Input:
Candidate paper list
Rank papers by relevance scores
If Year of candidate paper < year of original paper
    Add candidate paper into new recommendation list
Else
    Continue
End if Length of new recommendation list is 30
Output:
New recommendation list

Recommendation performance in information extraction.

MetricsRunsTop 5Top 10Top 20Top 30
PrecisionBM2510.6%12.6%7.1%6.0%
W2V-VSM12.8%12.0%10.8%8.9%
TFW-BM2513.6%16.6%9.6%7.4%
TFW-W2V-VSM16.4%14.9%13.7%13.0%
RecallBM2514.9%35.1%40.0%50.6%
W2V-VSM17.5%38.7%48.9%58.6%
TFW-BM2519.0%46.4%53.6%61.9%
TFW-W2V-VSM23.3%48.7%57.0%66.8%
F1-scoreBM2512.4%18.5%12.1%10.7%
W2V-VSM14.8%18.3%17.7%17.2%
TFW-BM2517.5%24.5%16.3%13.2%
TFW-W2V-VSM19.3%22.4%22.1%21.8%

Classification of term function (Li, Cheng, & Lu, 2017).

Classification of term functionAuthors
Head, goal, method, otherKondo (2009)
Technology, effectNanba, Kondo, & Takezawa (2010)
Focus, technique, domainGupta & Manning (2012)
Technique, applicationTsai, Kundu, & Roth (2013)
Method, task, otherHuang & Wan (2013)
Domain-independent: Research topic, Research methodCheng (2015)
Domain-related: Case, tool, dataset, etc.

Recommendation performance in sentiment analysis.

MetricsRunsTop 5Top 10Top 20Top 30
PrecisionBM2511.1%10.5%8.2%7.0%
W2V-VSM13.6%12.9%10.5%9.5%
TFW-BM2512.9%13.0%9.1%8.2%
TFW-W2V-VSM16.7%15.5%13.8%11.0%
RecallBM2515.6%29.6%46.3%59.3%
W2V-VSM19.5%34.3%52.6%67.9%
TFW-BM2518.1%36.7%51.5%69.6%
TFW-W2V-VSM27.7%41.0%59.2%73.8%
F1-scoreBM2513.0%15.5%13.9%12.5%
W2V-VSM16.0%18.7%17.5%16.7%
TFW-BM2515.1%19.2%15.4%17.2%
TFW-W2V-VSM20.8%22.5%22.4%19.0%

Recommendation performance in recommender system.

MetricsRunsTop 5Top 10Top 20Top 30
PrecisionBM2514.3%12.1%7.5%6.4%
W2V-VSM16.4%14.8%12.3%11.7%
TFW-BM2517.1%15.7%10.7%8.3%
TFW-W2V-VSM21.7%19.9%16.2%15.2%
RecallBM2521.3%36.2%44.7%57.4%
W2V-VSM24.7%44.4%54.2%73.1%
TFW-BM2525.5%46.8%63.8%74.5%
TFW-W2V-VSM26.8%50.1%68.9%77.8%
F1-scoreBM2517.1%18.1%12.8%11.5%
W2V-VSM19.7%21.9%20.0%20.2%
TFW-BM2520.5%23.5%18.3%14.9%
TFW-W2V-VSM24.0%28.5%26.2%25.2%

Ayala-Gomez, F., Daroczy, B., Benczur, A., Mathioudakis, M., & Gionis, A. (2018). Global citation recommendation using knowledge graphs. Journal of Intelligent and Fuzzy Systems, 34(5), 3089–3100. https://doi.org/10.3233/JIFS-169493.Ayala-GomezF.DaroczyB.BenczurA.MathioudakisM.GionisA.2018Global citation recommendation using knowledge graphsJournal of Intelligent and Fuzzy Systems34530893100https://doi.org/10.3233/JIFS-169493Search in Google Scholar

Beel, J., & Dinesh, S. (2017). Real-World Recommender Systems for Academia: The Pain and Gain in Building, Operating, and Researching them. In Proceedings of the ECIR’17 workshop on bibliometric-enhanced information retrieval (pp. 6–17).BeelJ.DineshS.2017Real-World Recommender Systems for Academia: The Pain and Gain in Building, Operating, and Researching themInProceedings of the ECIR’17 workshop on bibliometric-enhanced information retrieval617Search in Google Scholar

Beltagy, I., Lo, K., & Cohan, A. (2019). SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615–3620.BeltagyI.LoK.CohanA.2019SciBERT: A pretrained language model for scientific textInProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)36153620Search in Google Scholar

Bethard, S., & Jurafsky, D. (2010). Who should I cite: Learning literature search models from citation behavior. In Proceedings of the 19th ACM international conference on information and knowledge management (CIKM’10) (pp. 609–618). ACM.BethardS.JurafskyD.2010Who should I cite: Learning literature search models from citation behaviorInProceedings of the 19th ACM international conference on information and knowledge management (CIKM’10)609618ACMSearch in Google Scholar

Bhagavatula, C., Feldman, S., Power, R., & Ammar, W. (2018). Content-based citation recommendation. arXiv preprint arXiv:1802.08301.BhagavatulaC.FeldmanS.PowerR.AmmarW.2018Content-based citation recommendationarXiv preprint arXiv:1802.08301.Search in Google Scholar

Burges, C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2), 121–167.BurgesC.1998A tutorial on support vector machines for pattern recognitionData Mining and Knowledge Discovery22121167Search in Google Scholar

Caragea, C., Silvescu, A., Mitra, P., & Giles, C.L. (2013). Can’t see the forest for the trees?: A citation recommendation system. In Proceedings of the 13th ACM/IEEE-CS joint conference on digital libraries (JCDL’13) (pp. 111–114). ACM.CarageaC.SilvescuA.MitraP.GilesC.L.2013Can’t see the forest for the trees?: A citation recommendation systemInProceedings of the 13th ACM/IEEE-CS joint conference on digital libraries (JCDL’13)111114ACMSearch in Google Scholar

Chakraborty, T., Modani, N., Narayanam, R., & Nagar, S. (2015). Discern: a diversified citation recommendation system for scientific queries. In Proceedings of the 31th international conference on data engineering (ICDE’15) (pp. 555–566). IEEE.ChakrabortyT.ModaniN.NarayanamR.NagarS.2015Discern: a diversified citation recommendation system for scientific queriesInProceedings of the 31th international conference on data engineering (ICDE’15)555566IEEESearch in Google Scholar

Chen, C., Mayanglambam, S., Hsu, F., Lu, C., Lee, H., & Ho, J. (2012). Novelty paper recommendation using citation authority diffusion. In Proceedings of the international conference on technologies and applications of artificial intelligence (TAAI’11) (pp. 126–131). IEEE.ChenC.MayanglambamS.HsuF.LuC.LeeH.HoJ.2012Novelty paper recommendation using citation authority diffusionInProceedings of the international conference on technologies and applications of artificial intelligence (TAAI’11)126131IEEESearch in Google Scholar

Cheng, Q. (2015). Term Function Recognition of Academic Text. (Unpublished doctoral dissertation). Wuhan University, Wuhan, China.ChengQ.2015Term Function Recognition of Academic Text(Unpublished doctoral dissertation).Wuhan UniversityWuhan, ChinaSearch in Google Scholar

Ding, Y., Liu, X.Z., Guo, C., & Cronin, B. (2013). The distribution of references across texts: Some implications for citation analysis. Journal of Informetrics, 7(3), 583–592.DingY.LiuX.Z.GuoC.CroninB.2013The distribution of references across texts: Some implications for citation analysisJournal of Informetrics73583592Search in Google Scholar

Duma, D., & Klein, E. (2014). Citation resolution: A method for evaluating context-based citation recommendation systems. In Proceedings of the 52th annual meeting of the association for computational linguistics (ACL’14) (Vol. 2, pp. 358–363).DumaD.KleinE.2014Citation resolution: A method for evaluating context-based citation recommendation systemsInProceedings of the 52th annual meeting of the association for computational linguistics (ACL’14)2358363Search in Google Scholar

Duma, D., Liakata, M., Clare, A., Ravenscroft, J., & Klein, E. (2016). Applying Core Scientific Concepts to Context-Based Citation Recommendation. In Proceedings of the 10th edition of the language resources and evaluation conference (LREC’16).DumaD.LiakataM.ClareA.RavenscroftJ.KleinE.2016Applying Core Scientific Concepts to Context-Based Citation RecommendationInProceedings of the 10th edition of the language resources and evaluation conference (LREC’16)Search in Google Scholar

Duma, D., Liakata, M., Clare, A., Ravenscroft, J., & Klein, E. (2016). Rhetorical Classification of Anchor Text for Citation Recommendation. D-Lib Magazine, 22(9/10).DumaD.LiakataM.ClareA.RavenscroftJ.KleinE.2016Rhetorical Classification of Anchor Text for Citation RecommendationD-Lib Magazine229/10Search in Google Scholar

Duma, D. (2019). Contextual citation recommendation using scientific discourse annotation schemes (Doctoral dissertation). United Kingdom, Edinburgh: The University of Edinburgh.DumaD.2019Contextual citation recommendation using scientific discourse annotation schemes(Doctoral dissertation)United Kingdom, EdinburghThe University of EdinburghSearch in Google Scholar

Ebesu, T., & Fang, Y. (2017). Neural citation network for context-aware citation recommendation. In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval (SIGIR’17) (pp. 1093–1096). ACM.EbesuT.FangY.2017Neural citation network for context-aware citation recommendationInProceedings of the 40th International ACM SIGIR conference on research and development in information retrieval (SIGIR’17)10931096ACMSearch in Google Scholar

Gao, Z. (2016). Examining influences of publication dates on citation recommendation systems. In Proceedings of the 12th international conference on fuzzy systems and knowledge discovery (FSKD’15) (pp. 1400–1405). IEEE.GaoZ.2016Examining influences of publication dates on citation recommendation systemsInProceedings of the 12th international conference on fuzzy systems and knowledge discovery (FSKD’15)14001405IEEESearch in Google Scholar

Huang, W.Y., Wu, Z.H., Mitra, P., & Giles, C. (2014). RefSeer: Citation Recommendation System. In Proceedings of the 14th ACM/IEEE-CS joint conference on digital libraries (JCDL’14) (pp. 371–374). ACMHuangW.Y.WuZ.H.MitraP.GilesC.2014RefSeer: Citation Recommendation SystemInProceedings of the 14th ACM/IEEE-CS joint conference on digital libraries (JCDL’14)371374ACMSearch in Google Scholar

He, J., Nie, J.Y., Lu, Y., & Zhao, W. (2012). Position-aligned translation model for citation recommendation. In Proceedings of the international symposium on string processing and information retrieval (pp. 251–263). Springer, Berlin, Heidelberg.HeJ.NieJ.Y.LuY.ZhaoW.2012Position-aligned translation model for citation recommendationInProceedings of the international symposium on string processing and information retrieval251263Springer, Berlin, HeidelbergSearch in Google Scholar

He, Q., Kifer, D., Pei, J., Mitra, P., & Giles, C. (2011). Citation recommendation without author supervision. In Proceedings of the fourth ACM international conference on web search and data mining (WSDM’11) (pp. 755–764). ACM.HeQ.KiferD.PeiJ.MitraP.GilesC.2011Citation recommendation without author supervisionInProceedings of the fourth ACM international conference on web search and data mining (WSDM’11)755764ACMSearch in Google Scholar

He, Q., Pei, J., Kifer, D., Mitra, P., & Giles, C. (2010). Context-aware citation recommendation. In Proceedings of the 19th international conference on world wide web (WWW’10) (pp. 421–430). ACM.HeQ.PeiJ.KiferD.MitraP.GilesC.2010Context-aware citation recommendationInProceedings of the 19th international conference on world wide web (WWW’10)421430ACMSearch in Google Scholar

Huang, W.Y., Wu, Z.H., Chen, L., Mitra, P., & Giles, C. (2015). A Neural Probabilistic Model for Context Based Citation Recommendation. In Proceedings of the 29th association for the advancement of artificial intelligence (AAAI’2015) (pp. 2404–2410).HuangW.Y.WuZ.H.ChenL.MitraP.GilesC.2015A Neural Probabilistic Model for Context Based Citation RecommendationInProceedings of the 29th association for the advancement of artificial intelligence (AAAI’2015)24042410Search in Google Scholar

Jeong, C., Jang, S., Shin, H., Park, E., & Choi, S. (2019). A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional Networks. arXiv preprint arXiv:1903.06464.JeongC.JangS.ShinH.ParkE.ChoiS.2019A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional NetworksarXiv preprint arXiv:1903.06464.Search in Google Scholar

Jiang, Z.R., Liu, X.Z., & Gao, L.C. (2014). Dynamic topic/citation influence modeling for chronological citation recommendation. In Proceedings of the 5th international workshop on web-scale knowledge representation retrieval & reasoning (pp. 15–18). ACM.JiangZ.R.LiuX.Z.GaoL.C.2014Dynamic topic/citation influence modeling for chronological citation recommendationInProceedings of the 5th international workshop on web-scale knowledge representation retrieval & reasoning1518ACMSearch in Google Scholar

Jiang, Z. (2015). Chronological scientific information recommendation via supervised dynamic topic modeling. In Proceedings of the 8th ACM international conference on web search and data mining (WSDM’15) (pp. 453–458). ACM.JiangZ.2015Chronological scientific information recommendation via supervised dynamic topic modelingInProceedings of the 8th ACM international conference on web search and data mining (WSDM’15)453458ACMSearch in Google Scholar

Jiang, Z.R., Liu, X.Z., & Gao, L.C. (2015). Chronological citation recommendation with information-need shifting. In Proceedings of the 24th ACM international on conference on information and knowledge management (CIKM’15) (pp. 1291–1300). ACM.JiangZ.R.LiuX.Z.GaoL.C.2015Chronological citation recommendation with information-need shiftingInProceedings of the 24th ACM international on conference on information and knowledge management (CIKM’15)12911300ACMSearch in Google Scholar

Jiang, Z.R., Yin, Y., Gao, L.C., Lu, Y., & Liu, X.Z. (2018). Cross-Language Citation Recommendation via Hierarchical Representation Learning on Heterogeneous Graph. In Proceedings of the 41th international ACM SIGIR conference on research and development in information retrieval(SIGIR’18). ACM.JiangZ.R.YinY.GaoL.C.LuY.LiuX.Z.2018Cross-Language Citation Recommendation via Hierarchical Representation Learning on Heterogeneous GraphInProceedings of the 41th international ACM SIGIR conference on research and development in information retrieval (SIGIR’18)ACMSearch in Google Scholar

Jia, H.F., & Saule, E. (2018). Towards Finding Non-obvious Papers: An Analysis of Citation Recommender Systems. arXiv preprint arXiv:1812.11252.JiaH.F.SauleE.2018Towards Finding Non-obvious Papers: An Analysis of Citation Recommender SystemsarXiv preprint arXiv:1812.11252.Search in Google Scholar

Kates-Harbeck, J., & Haggblade, M. (2013). A two-stage citation recommendation system. Stanford University.Kates-HarbeckJ.HaggbladeM.2013A two-stage citation recommendation systemStanford UniversitySearch in Google Scholar

Küçüktunç, O., Saule, E., Kaya, K., & Çatalyürek, U. (2012). Direction awareness in citation recommendation. In Proceedings of the international workshop on ranking in databases (DBRank’12) in conjunction with VLDB’12.KüçüktunçO.SauleE.KayaK.ÇatalyürekU.2012Direction awareness in citation recommendationInProceedings of the international workshop on ranking in databases (DBRank’12) in conjunction with VLDB’12Search in Google Scholar

Küçüktunç, O., Saule, E., Kaya, K., & Çatalyürek, Ü. (2013). Result diversification in automatic citation recommendation. In Proceedings of the iConference workshop on computational scientometrics: theory and applications (pp. 1–4).KüçüktunçO.SauleE.KayaK.ÇatalyürekÜ.2013Result diversification in automatic citation recommendationInProceedings of the iConference workshop on computational scientometrics: theory and applications14Search in Google Scholar

Küçüktunç, O., Saule, E., Kaya, K., & Çatalyürek, Ü. (2015). Diversifying citation recommendations. ACM Transactions on Intelligent Systems and Technology, 5(4), 55.KüçüktunçO.SauleE.KayaK.ÇatalyürekÜ.2015Diversifying citation recommendationsACM Transactions on Intelligent Systems and Technology5455Search in Google Scholar

Liu, Y.N., Yan, R., & Yan, H.F. (2013). Guess what you will cite: Personalized citation recommendation based on users’ preference. In Asia information retrieval symposium (pp. 428–439). Springer, Berlin, Heidelberg.LiuY.N.YanR.YanH.F.2013Guess what you will cite: Personalized citation recommendation based on users’ preferenceInAsia information retrieval symposium428439SpringerBerlin, HeidelbergSearch in Google Scholar

Li, M., Wang, M., & Wang, C.G. (2010). Research on SVM classification performance in rolling bearing diagnosis. In Proceeding of the international conference on intelligent computation technology and automation (ICICTA’10) (pp. 132–135). IEEE.LiM.WangM.WangC.G.2010Research on SVM classification performance in rolling bearing diagnosisInProceeding of the international conference on intelligent computation technology and automation (ICICTA’10)132135IEEESearch in Google Scholar

Livne, A., Gokuladas, V., Teevan, J., Dumais, S., & Adar, E. (2014). CiteSight: supporting contextual citation recommendation using differential search. In Proceedings of the 37th international ACM SIGIR conference on research & development in information retrieval (SIGIR’14) (pp. 807–816). ACM.LivneA.GokuladasV.TeevanJ.DumaisS.AdarE.2014CiteSight: supporting contextual citation recommendation using differential searchInProceedings of the 37th international ACM SIGIR conference on research & development in information retrieval (SIGIR’14)807816ACMSearch in Google Scholar

Li, X., Cheng, Q., & Lu, W. (2017). “CS-LAS: A Scientific Literature Retrieval and Analysis System Based on Term Function Recognition (TFR).” In Proceedings of the 16th international conference of the international society for scientometrics and informetrics (ISSI’17).LiX.ChengQ.LuW.2017“CS-LAS: A Scientific Literature Retrieval and Analysis System Based on Term Function Recognition (TFR).”InProceedings of the 16th international conference of the international society for scientometrics and informetrics (ISSI’17)Search in Google Scholar

Lu, Y., He, J., Shan, D.D., & Yan, H.F. (2011). Recommending citations with translation model. In Proceedings of the 20th ACM international conference on information and knowledge management (CIKM’11) (pp. 2017–2020). ACM.LuY.HeJ.ShanD.D.YanH.F.2011Recommending citations with translation modelInProceedings of the 20th ACM international conference on information and knowledge management (CIKM’11)20172020ACMSearch in Google Scholar

Luong, M., Nguyen, T., & Kan, M.(2012). Logical structure recovery in scholarly articles with rich document features. In multimedia storage and retrieval innovations for digital library systems (pp. 270–292). IGI Global.LuongM.NguyenT.KanM.2012Logical structure recovery in scholarly articles with rich document featuresIn multimedia storage and retrieval innovations for digital library systems270292IGI GlobalSearch in Google Scholar

McNee, S., Albert, I., Cosley, D., Gopalkrishnan, P., Lam, S., Rashid, A., Konstan, J., & Riedl, J. (2002). On the recommending of citations for research papers. In Proceedings of the 2002 ACM conference on computer supported cooperative work (pp. 116–125). ACM.McNeeS.AlbertI.CosleyD.GopalkrishnanP.LamS.RashidA.KonstanJ.RiedlJ.2002On the recommending of citations for research papersInProceedings of the 2002 ACM conference on computer supported cooperative work116125ACMSearch in Google Scholar

Melgani, F., & Bruzzone, L. (2004). Classification of hyperspectral remote sensing images with support vector machines. IEEE Transactions on Geoscience and Remote Sensing, 42(8), 1778–1790.MelganiF.BruzzoneL.2004Classification of hyperspectral remote sensing images with support vector machinesIEEE Transactions on Geoscience and Remote Sensing42817781790Search in Google Scholar

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.MikolovT.ChenK.CorradoG.DeanJ.2013Efficient estimation of word representations in vector spacearXiv preprint arXiv:1301.3781.Search in Google Scholar

Ren, X., Liu, J.L., Yu, X., Khandelwal, U., Gu, Q.Q., Wang, L.D., & Han, J.W. (2014). Cluscite: Effective citation recommendation by information network-based clustering. In Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining (KDD’2014) (pp. 821–830). ACM.RenX.LiuJ.L.YuX.KhandelwalU.GuQ.Q.WangL.D.HanJ.W.2014Cluscite: Effective citation recommendation by information network-based clusteringInProceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining (KDD’2014)821830ACMSearch in Google Scholar

Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval, 3(4), 333–389.RobertsonS.ZaragozaH.2009The probabilistic relevance framework: BM25 and beyondFoundations and Trends® in Information Retrieval34333389Search in Google Scholar

Rokach, L., Mitra, P., Kataria, S., Huang, W., & Giles, C. (2013). A Supervised Learning Method for ContextAware Citation Recommendation in a Large Corpus. Proceedings of the Large-Scale and Distributed Systems for Information Retrieval Workshop (LSDS-IR) (pp. 17–22).RokachL.MitraP.KatariaS.HuangW.GilesC.2013A Supervised Learning Method for ContextAware Citation Recommendation in a Large CorpusProceedings of the Large-Scale and Distributed Systems for Information Retrieval Workshop (LSDS-IR)1722Search in Google Scholar

Sesagiri Raamkumar, A., Foo, S., & Pang, N. (2015). Rec4LRW–Scientific Paper Recommender System for Literature Review and Writing. In Proceedings of the 6th international conference on applications of digital information and web technologies (pp. 106–119).Sesagiri RaamkumarA.FooS.PangN.2015Rec4LRW–Scientific Paper Recommender System for Literature Review and WritingInProceedings of the 6th international conference on applications of digital information and web technologies106119Search in Google Scholar

Sesagiri Raamkumar, A., Foo, S., & Pang, N. (2016). What papers should I cite from my reading list? User evaluation of a manuscript preparatory assistive task. In Proceedings of the joint workshop on bibliometric-enhanced information retrieval and natural language processing for digital libraries (BIRNDL’16) (pp. 51–62).Sesagiri RaamkumarA.FooS.PangN.2016What papers should I cite from my reading list? User evaluation of a manuscript preparatory assistive taskInProceedings of the joint workshop on bibliometric-enhanced information retrieval and natural language processing for digital libraries (BIRNDL’16)5162Search in Google Scholar

Son, J., & Kim, S. (2018). Academic paper recommender system using multilevel simultaneous citation networks. Decision Support Systems, 105, 24–33.SonJ.KimS.2018Academic paper recommender system using multilevel simultaneous citation networksDecision Support Systems1052433Search in Google Scholar

Strohman, T., Croft, W., & Jensen, D. (2007). Recommending citations for academic papers. In Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR’07) (pp. 705–706). ACM.StrohmanT.CroftW.JensenD.2007Recommending citations for academic papersInProceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR’07)705706ACMSearch in Google Scholar

Tang, J., & Zhang, J. (2009). A discriminative approach to topic-based citation recommendation. In Pacific-Asia Conference on knowledge discovery and data mining (PAKDD’09) (pp. 572–579). Springer, Berlin, Heidelberg.TangJ.ZhangJ.2009A discriminative approach to topic-based citation recommendationInPacific-Asia Conference on knowledge discovery and data mining (PAKDD’09)572579Springer, Berlin, HeidelbergSearch in Google Scholar

Tang, X.W., Wan, X.J., & Zhang, X. (2014). Cross-language context-aware citation recommendation in scientific articles. In Proceedings of the 37th international ACM SIGIR conference on research & development in information retrieval (SIGIR’14) (pp. 817–826). ACM.TangX.W.WanX.J.ZhangX.2014Cross-language context-aware citation recommendation in scientific articlesInProceedings of the 37th international ACM SIGIR conference on research & development in information retrieval (SIGIR’14)817826ACMSearch in Google Scholar

Viera, A., & Garrett, J. (2005). Understanding interobserver agreement: the kappa statistic. Fam Med, 37(5), 360–363.VieraA.GarrettJ.2005Understanding interobserver agreement: the kappa statisticFam Med375360363Search in Google Scholar

Wu, H., Hua, Y., Li, B., & Pei, Y.J. (2012). Enhancing citation recommendation with various evidences. In Proceedings of the 9th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD’12) (pp. 1160–1165). IEEE.WuH.HuaY.LiB.PeiY.J.2012Enhancing citation recommendation with various evidencesInProceedings of the 9th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD’12)11601165IEEESearch in Google Scholar

Yang, L.B., Zhang, Z.Q., Cai, X.Y., & Dai, T. (2019). Attention-Based Personalized Encoder-Decoder Model for Local Citation Recommendation. Computational Intelligence and Neuroscience, 2019.YangL.B.ZhangZ.Q.CaiX.Y.DaiT.2019Attention-Based Personalized Encoder-Decoder Model for Local Citation RecommendationComputational Intelligence and Neuroscience2019Search in Google Scholar

Yang, L.B., Zhang, Z.Q., Cai, X.Y., & Guo, L.T. (2019). Citation recommendation as edge prediction in heterogeneous bibliographic network: A network representation approach. IEEE Access, 7, 23232–23239.YangL.B.ZhangZ.Q.CaiX.Y.GuoL.T.2019Citation recommendation as edge prediction in heterogeneous bibliographic network: A network representation approachIEEE Access72323223239Search in Google Scholar

Zarrinkalam, F., & Kahani, M. (2012). A multi-criteria hybrid citation recommendation system based on linked data. In Proceedings of the 2nd International eConference on Computer and Knowledge Engineering (ICCKE’12) (pp. 283–288). IEEE.ZarrinkalamF.KahaniM.2012A multi-criteria hybrid citation recommendation system based on linked dataInProceedings of the 2nd International eConference on Computer and Knowledge Engineering (ICCKE’12)283288IEEESearch in Google Scholar

Zarrinkalam, F., & Kahani, M. (2013). SemCiR: A citation recommendation system based on a novel semantic distance measure. Program, 47(1), 92–112.ZarrinkalamF.KahaniM.2013SemCiR: A citation recommendation system based on a novel semantic distance measureProgram47192112Search in Google Scholar

Zhang, S., Yao, L., Sun, A., & Tay, Y. (2019). Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys (CSUR), 52(1), 5.ZhangS.YaoL.SunA.TayY.2019Deep learning based recommender system: A survey and new perspectivesACM Computing Surveys (CSUR)5215Search in Google Scholar

Zhou, S.P. (2010). ActiveCite: An Interactive System for Automatic Citation Suggestion. National University of Singapore.ZhouS.P.2010ActiveCite: An Interactive System for Automatic Citation SuggestionNational University of SingaporeSearch in Google Scholar

Recommended articles from Trend MD

Plan your remote conference with Sciendo