SIG-MET Distinguished Speaker Series: Research Performance Evaluation: What to Measure, How to Measure It, and What to Avoid

Description

Research performance evaluation increasingly shapes funding allocation, institutional strategy, academic incentives, and science policy. Yet many widely adopted assessment methods and indicators rest on questionable assumptions and can produce substantial distortions in scores and rankings. This talk examines what should be measured in research evaluation, how it can be measured, and which common practices should be avoided. It begins by clarifying the entities and objectives of research assessment and by comparing peer review with bibliometric approaches. Although peer review remains indispensable where bibliographic databases provide inadequate coverage, its subjectivity, cost, limited scalability, and reliance on small subsets of research outputs raise serious concerns when it is used in large-scale national assessment exercises. Bibliometric evaluation also has important limitations, but, under appropriate conditions, it can offer a more comprehensive, timely, and cost-effective basis for assessing scholarly impact. The presentation then introduces Fractional Scientific Strength (FSS), an indicator designed to measure research productivity as scholarly impact generated per unit of research input. It explains why researchers must be classified by field and why performance at the aggregate level, whether for disciplines, departments, institutions, or countries, should be derived from measurements conducted at the individual level. Finally, the talk critically examines several widely used indicators and assessment systems, including the MNCS, PP(10%), the h-index, international university rankings, and national peer-review exercises. Empirical examples illustrate how unsuitable indicators and methods can significantly alter rankings and lead to misleading conclusions. The central message is straightforward: sound research evaluation requires valid measures, suitable data, and methods aligned with the purpose and context of the assessment. We should count only what counts, while remaining fully aware of what cannot be counted.

Speaker: Professor Giovanni Abramo
Bio: Giovanni Abramo holds a Laurea in electrical engineering from the University of Rome "Sapienza" and a Master of Science in Management from the Massachusetts Institute of Technology (MIT). In November 2023, after nearly 40 years at the National Research Council of Italy (CNR) as a Research Director, Giovanni transitioned to Universitas Mercatorum in Italy, where he is now a Full Professor of Management of Innovation and Entrepreneurship. In 2004, Giovanni founded the CNR Laboratory for Studies in Research Evaluation. In 2006, he established Research Value, a CNR spinoff company dedicated to providing research evaluation services to research institutions and governments. Since 2017, he has also held the title of Honorary Professor of Management at the University of Waikato in New Zealand. Throughout his career, Giovanni has consulted for prestigious organizations, including the United Nations, the European Commission, Italian Ministries, and various public and private entities. His research focuses on how individuals, organizations, and countries produce and disseminate knowledge, with a special emphasis on research performance evaluation, gender equity, and cross-sector knowledge transfer. Since 2007, he has published over 170 indexed publications in the field of scientometrics in journals such as Quantitative Science Studies, Scientometrics, Journal of Informetrics, JASIST, Higher Education, Research Evaluation, Research Policy, Science and Public Policy, Journal of Technology Transfer, Technovation, R&D Management, and PLoS ONE. In 2023, Giovanni was elected President and Chairman of the Board of the International Society for Scientometrics and Informetrics (ISSI). Giovanni embraces a Zen Buddhist vision of life, “making little distinction between his work and his play, his labor and his leisure, his mind and his body, his education and his recreation, his love and his religion. He hardly knows which is which. He simply pursues his vision of excellence in whatever he does, leaving others to decide whether he is working or playing. To him, he is always doing both”.