SIG-MET Distinguished Speaker Series: Lecture by Professor Mike Thelwall

Description

Title: Can Large Language Models replace citation data for some types of bibliometric analysis? 

Recent evidence has shown that some Large Language Models (LLMs) can score published journal articles for research quality in most fields in a way that associates more strongly than field normalized citation counts with expert judgements. This suggests that LLMs may be more useful than citation data for some contexts in which citation data is currently used. Also, since LLM scores are available earlier than mature citation counts, they may be useful for some applications where citations are unavailable. Nevertheless, the reason why LLMs have this capability is not clear, their biases have not been extensively explored, and there may be unwanted systemic effects from using LLMs in formative evaluations. This talk will discuss these issues and make some recommendations for future research. 

Mike Thelwall is a Professor of Data Science in the School of Information, Journalism and Communication at the University of Sheffield in the UK, and a visiting professor at Peking University, University of Malaya, and Wolverhampton University. He investigates quantitative methods to support research evaluation, with a current focus on large language models for research quality evaluation, and a previous emphasis on alternative indicators for research impact. He leads the AI Peer project investigating whether and how ChatGPT and similar models can support research assessment as well as the LLMs Supporting Grant Peer Review project. His recent free book is: Quantitative Methods in Research Evaluation: Citation Indicators, Altmetrics, and Artificial Intelligence (https://doi.org/10.48550/arXiv.2407.00135). He sits on eight editorial boards, including Journal of Data and Information Science and Journal of the Association for Information Science and Technology.