Nina Smirnova

Nina Smirnova

Auto-generated profile, based on publicly available information

Nina Smirnova

GESIS -Leibniz Institute for the Social Sciences Unter Sachsenhausen 6-8 50667 Cologne

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
14 November 2022 in Scientometrics
Verified
Journal Article
Open access
PDF file
NSNina Smirnova
Nina Smirnova
PMPhilipp Mayr
Philipp Mayr
Analysis of acknowledgments is particularly interesting as acknowledgments may give information not only about funding, but they are also able to reveal hidden contributions to authorship and the researcher’s collaboration patterns, context in which research was conducted, and specific aspects of the academic work. The focus of the present research is the analysis of a large sample of acknowledgement texts indexed in the Web of Science (WoS) Core Collection. Record types “article” and “review” from four different scientific domains, namely social sciences, economics, oceanography and computer science, published from 2014 to 2019 in a scientific journal in English were considered. Six types of acknowledged entities, i.e., funding agency, grant number, individuals, university, corporation and miscellaneous, were extracted from the acknowledgement texts using a named entity recognition tagger and subsequently examined. A general analysis of the acknowledgement texts showed that indexing of funding information in WoS is incomplete. The analysis of the automatically extracted entities revealed differences and distinct patterns in the distribution of acknowledged entities of different types between different scientific domains. A strong association was found between acknowledged entity and scientific domain, and acknowledged entity and entity type. Only negligible correlation was found between the number of citations and the number of acknowledged entities. Generally, the number of words in the acknowledgement texts positively correlates with the number of acknowledged funding organizations, universities, individuals and miscellaneous entities. At the same time, acknowledgement texts with the larger number of sentences have more acknowledged individuals and miscellaneous categories.

23 August 2023 in Scientometrics
Verified
Journal Article
Open access
PDF file
NSNina Smirnova
Nina Smirnova
PMPhilipp Mayr
Philipp Mayr
Acknowledgments in scientific papers may give an insight into aspects of the scientific community, such as reward systems, collaboration patterns, and hidden research trends. The aim of the paper is to evaluate the performance of different embedding models for the task of automatic extraction and classification of acknowledged entities from the acknowledgment text in scientific papers. We trained and implemented a named entity recognition (NER) task using the flair NLP framework. The training was conducted using three default Flair NER models with four differently-sized corpora and different versions of the flair NLP framework. The Flair Embeddings model trained on the medium corpus with the latest FLAIR version showed the best accuracy of 0.79. Expanding the size of a training corpus from very small to medium size massively increased the accuracy of all training algorithms, but further expansion of the training corpus did not bring further improvement. Moreover, the performance of the model slightly deteriorated. Our model is able to recognize six entity types: funding agency, grant number, individuals, university, corporation, and miscellaneous. The model works more precisely for some entity types than for others; thus, individuals and grant numbers showed a very good F1-Score over 0.9. Most of the previous works on acknowledgment analysis were limited by the manual evaluation of data and therefore by the amount of processed data. This model can be applied for the comprehensive analysis of acknowledgment texts and may potentially make a great contribution to the field of automated acknowledgment analysis.

7 November 2023 in JMIR Preprints
Verified
Preprint
Open access
XTXu Tong
Xu Tong
NSNina Smirnova
Nina Smirnova
SUSharmila Upadhyaya
Sharmila Upadhyaya
RYRan Yu
Ran Yu
JCJack H. Culbert
Jack H. Culbert
PMPhilipp Mayr
Philipp Mayr
BACKGROUND Recent advances in large language models (LLMs) have shown remarkable performance on various downstream tasks in zero- and few-shot scenarios, shedding light on named entity recognition (NER) in low-resource domains. Traditional Chinese medicine (TCM) against COVID-19 has been a new research topic and led to niche research literature. NER techniques are crucial for extracting and utilizing the rich knowledge in such literature. OBJECTIVE To explore and compare the performance of ChatGPT and other state-of-the-art LLMs on domain-specific NER tasks covering different entity types and domains in TCM against COVID-19 literature. METHODS We established a dataset of 389 articles on TCM against COVID-19, and manually annotated 48 of them with 6 types of entities belonging to 3 domains as the ground truth, against which the NER performance of LLMs can be assessed. We then performed NER tasks for the 6 entity types using ChatGPT (GPT-3.5 and GPT-4) and 4 state-of-the-art BERT-based question-answering (QA) models (RoBERTa, MiniLM, PubMedBERT and SciBERT) without prior training on the specific task. A domain fine-tuned model (GSAP-NER) was also applied for a comprehensive comparison on one of the entity types. In the task setup, we considered two different matching methods, namely exact match and fuzzy match, to have better coverage of potential application scenarios. The evaluation metrics we used are Precision, Recall, and F-1 in both exact match and fuzzy match. RESULTS The overall performance of LLMs varied significantly in exact match and fuzzy match. In the fuzzy match, ChatGPT surpassed BERT-based QA models in 5 out of 6 tasks, while in exact match, BERT-based QA models outperformed ChatGPT in 5 out of 6 tasks but with a smaller F-1 difference. GPT-4 showed a significant advantage over other models in fuzzy match, especially on the entity type of TCM formula and the Chinese patent drug (TFD) and ingredient (IG), achieving a higher F-1 score of 0.814 and 0.689, respectively. Although GPT-4 outperformed BERT-based models on entity type of herb (HB, 0.324), target (TG, 0.319), and research method (RM, 0.443), none of the F-1 scores exceeded 0.5. GSAP-NER, outperformed GPT-4 in terms of F-1 by a slight margin (0.45 vs 0.433) on RM. ChatGPT achieved considerably higher recalls than precisions, particularly in the fuzzy match. CONCLUSIONS The NER performance of LLMs is highly dependent on the entity type, and their performance varies across application scenarios. ChatGPT could be a good choice for scenarios where high recall is favored, e.g., for domain novices obtaining an extensive overview of the field. However, for knowledge acquisition in rigorous scenarios, neither ChatGPT nor BERT-based QA models are off-the-shelf tools for professional practitioners. Besides, these BERT-based models are open-source, and thus more accessible for scientific inquiry and hence worth further exploration.

5 April 2025 in Journal of Informetrics
Verified
Journal Article
Open access
PNPanggih Kusuma Ningrum
Panggih Kusuma Ningrum
PMPhilipp Mayr
Philipp Mayr
NSNina Smirnova
Nina Smirnova
IAIana Atanassova
Iana Atanassova

26 April 2024 in arXiv
Verified
Preprint
Open access
PDF file
AMAasif Ahmad Mir
Aasif Ahmad Mir
NSNina Smirnova
Nina Smirnova
JRJeyshankar Ramalingam
Jeyshankar Ramalingam
PMPhilipp Mayr
Philipp Mayr
The study aims to highlight the growth and development of Indo-German collaborative research over the past three decades. Moreover, this study encompasses an in-depth examination of funding acknowledgements to gain valuable insights into the financial support that underpins these collaborative endeavors. Together with this paper, we provide an openly accessible dataset of Indo-German research articles for further and reproducible research activities (the "Indo-German Literature Dataset"). The data were retrieved from the Web of Science (WoS) database from the year 1990 till the 30th of November 2022. A total of 36,999 records were retrieved against the employed query. Acknowledged entities were extracted using a NER model specifically trained for this task. Interrelations between the extracted entities and scientific domains, lengths of acknowledgement texts, number of authors and affiliations, number of citations, and gender of the first author, as well as collaboration patterns between Indian and German funders were examined. The study brings to light that Physics, Chemistry, Materials Science, Astronomy and Astrophysics, and Engineering prominently dominate the Indo-German collaborative research. The United States, followed by England and France, are the most active collaborators in Indian and German research. Additionally, relations between entity, entity type, and scientific domain, were discovered. The study highlights a deeper understanding of the composition of the Indo-German collaborative research landscape of the last 30 years and its significance in advancing scientific knowledge and fostering international partnerships. Furthermore, we provide an open version of the original WoS dataset. The Indo-German Literature Dataset consists of 22,844 articles from OpenAlex and is available for related studies like literature studies and Scientometrics.

26 March 2024 in arXiv
Verified
Preprint
Open access
PDF file
THTobias Holtdirk
Tobias Holtdirk
NSNina Smirnova
Nina Smirnova
This technical report describes the incorporation of political blogs into Pollux, the Specialised Information Service (FID) for Political Science in Germany. Considering the widespread use of political blogs in political science research, we decided to include them in the Pollux search system to enhance the available information infrastructure. We describe the crawling and analyzing of the blogs and the pipeline that integrates them into the Pollux system. To demonstrate the content of the incorporated blogs, we also provide a visualization of the topics covered by the blog posts during the first three months following integration.