Assistência bibliográfica durante a escrita de textos científicos : uma abordagem com modelos de linguagem pré-treinados

Abstract

Scientific output includes the production of textual artifacts, as scientific papers and research projects. Scientific writing, in turn, presents its own challenges. One of them is to face information overload in literature, which hinders contextualization of a new document with the most relevant information of its research area. In face of this difficulty, in this work we will investigate ways of embedding sentences and paragraphs of scientific texts, with the purpose of content recomendation along scientific writing. Through automated scanning of databases of publishers Springer and Elsevier, a local mirror was built for access of research papers, from which it was possible to obtain a corpus with 40 thousand papers from Computer Science journals. Pre-trained language models were used to obtain embedding for words or wordpieces for a sample of 1605 papers. Using these representations, aggregation methods for generating embeddings for sentences and paragraphs were investigated. Embedding strategies for these textual elements were evaluated in two aspects. Firstly, in their capacity of reflecting the aggregation of paragraphs within sections of already published papers. Secondly, they were used frozen to train a bidirectional long short-term memory (BiLSTM) neural network for the task of classifying wether a text fragment and an abstract belonged to the same paper. In both situations, a better performance was obtained with embeddings from bidirectional encoder representations from Transformers (BERT), specifically with a version pre-trained on a scientific corpus (SciBERT). Also in both circumstances, embeddings of sentences obtained by taking the mean of pre-trained word embeddings, weighted considering the word frequency, showed worse performance when compared to the mean of vectors after removing stopwords. Paragraph embeddings using the encoding of a sequence of sentences with a BiLSTM presented superior performance when compared to simple mean of sentence vectors and, when applied to the introduction of a scientific paper in test set, was able to return its own original paper within the 5% most likely ones, on average. A qualitative demonstration of content recommendation which integrates the result of both analyses is presented. Considering the studied strategies, automated bibligraphical assistance during production of scientific texts was feasible, with potential for improvement with more optimized hierarchical BiLSTMs.