How can we measure reproducibility of IR experiments? (Downlaod slides)

Maria Maistro, University of Copenhagen

Abstract

We are today facing the so-called reproducibility crisis across all areas of science, where researchers have trouble reproducing and confirming previous experimental findings. In Information Retrieval (IR), the most common attitude for reproducibility is some sort of “close enough”: researchers put any reasonable effort to understand how an approach was implemented and how an experiment was conducted. After some iterations, when they obtain performance scores which somehow resemble the original ones, they decide that an experimental result is reproduced. This talk focuses on reproducibility in lR: it presents challenges related to reproducibility, covers some initiatives that have been proposed to ease reproducibility, investigates the problem of “objectively” measuring reproducibility, with reproducibility measures as Kendall’s tau and RMSE to name some, and explores the behaviour and properties of some reproducibility measures.

Speaker Bio

Maria Maistro studied initially Mathematics (BSc, University of Padua, 2011; MSc, University of Padua, 2014) and then Computer Science (PhD, University of Padua, 2018). She is a Marie Curie Fellow and a tenure track assistant professor at the Department of Computer Science, University of Copenhagen (DIKU). She conducts research in information retrieval, and particularly on evaluation, reproducibility and replicability, click log analysis, learning to rank and applied machine learning. She has already co-organized several international scientific events and she has served as member of programme committees and reviewer for highly ranked conferences and journals in information retrieval.

Register for this talk

Mixture-Based Correction for Position and Trust Bias in Counterfactual Learning to Rank, CIKM 2021

Ali Vardasbi, University of Amsterdam

Using RobBERT and eXtreme Multi-Label Classification to Extract Implicit and Explicit Skills From Dutch Job Descriptions

Sepideh Mesbah, Randstad Groep

Conversational Entity Linking: Problem Definition and Datasets, SIGIR 2021

Hideaki Joko, Radboud University

Time-aware evidence ranking for fact-checking, Journal of Web Semantics 2021

Liesbeth Allein, KU Leuven

Understanding Multi-channel Customer Behavior in Retail, CIKM 2021

Mozhdeh Ariannezhad, University of Amsterdam

Register for this talk

Supercalifragilisticexpialidocious: Why Using the “Right” Readability Formula in Children’s Web Search Matters, ECIR 2022

Garett Allen, TU Delft

Neural Information Retrieval for Educational Resources, ECIR Industry Day 2022

Carsten Schnober, WizeNoze

Embarrassingly shallow auto-encoders for dynamic collaborative filtering, Springer User Modeling and User-Adapted Interaction, Special Issue on Dynamic Recommender Systems and User Models 2022

Olivier Jeunen, Amazon

Leave No User Behind: Towards Improving the Utility of Recommender Systems for Non-mainstream Users, WSDM 2021

Zhe Roger Li, TU Delft

Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness, SIGIR 2021

Harrie Oosterhuis, Radboud University

Register for this session

IR Evaluation - An Industry Perspective

Gabriella Kazai, Microsoft

Abstract

Measurement is key in the development of any product or service, including information retrieval (IR) systems. The challenge of developing IR metrics from which reliable conclusions can be drawn is no trivial matter. For example, issues in sampling, instrumentation, label collection or metric computation methodologies can invalidate an evaluation experiment or even lead to problems in the training of machine learning systems. In this talk I will reflect on some of the challenges when developing IR metrics and building offline evaluation pipelines at Bing.

Speaker Bio

I am a Principal Applied Scientist at Microsoft, tackling a range of IR evaluation challenges as part of the Bing Web and AI Science Group, focusing on offline evaluation, crowdsourcing, and metric development for various search scenarios, including organic web search, news, autosuggestions, and covering aspects from relevance to source credibility, trust and bias. Prior to this, I worked at two startups: As Lead Data Scientist at Mudano Ltd, I worked on an AI-driven project management system to deliver large-scale IT change projects in the financial sector; and as VP of Data Science at Semion Ltd, I led a team of data scientists implementing the AI behind the Lumi news recommender app, providing out-of-the-box personalised media content to users based on interests discovered from their Twitter streams. Before that, I was a postdoc at Microsoft Research after obtaining my PhD from the University of London under the supervision of Prof Mounia Lalmas. My research interests include information retrieval (IR), IR evaluation, human computation, gamification, recommender systems, social IR, information seeking behaviour, activity based personal information management.

Register for this talk