DIR 2021 ½ is inviting Belgium and Netherlands-based researchers to present their work.
We invite contributions of previously published articles and work-in-progress to be delivered as a 15-minutes lightning talks. Please, fill in the form if you would like to present your work at DIR this year or would like to nominate someone.
You can expect vibrant discussions and plenty of social interactions. Don't hesitate and claim your spot.
Application deadline: 6th of February, 23.59 CET
How can we measure reproducibility of IR experiments? (Downlaod slides)
Maria Maistro, University of Copenhagen
We are today facing the so-called reproducibility crisis across all areas of science, where researchers have trouble reproducing and confirming previous experimental findings. In Information Retrieval (IR), the most common attitude for reproducibility is some sort of “close enough”: researchers put any reasonable effort to understand how an approach was implemented and how an experiment was conducted. After some iterations, when they obtain performance scores which somehow resemble the original ones, they decide that an experimental result is reproduced. This talk focuses on reproducibility in lR: it presents challenges related to reproducibility, covers some initiatives that have been proposed to ease reproducibility, investigates the problem of “objectively” measuring reproducibility, with reproducibility measures as Kendall’s tau and RMSE to name some, and explores the behaviour and properties of some reproducibility measures.
Maria Maistro studied initially Mathematics (BSc, University of Padua, 2011; MSc, University of Padua, 2014) and then Computer Science (PhD, University of Padua, 2018). She is a Marie Curie Fellow and a tenure track assistant professor at the Department of Computer Science, University of Copenhagen (DIKU). She conducts research in information retrieval, and particularly on evaluation, reproducibility and replicability, click log analysis, learning to rank and applied machine learning. She has already co-organized several international scientific events and she has served as member of programme committees and reviewer for highly ranked conferences and journals in information retrieval.
Lightning Talks — TBA ⚡
Mixture-Based Correction for Position and Trust Bias in Counterfactual Learning to Rank, CIKM 2021
Ali Vardasbi, University of Amsterdam
Using RobBERT and eXtreme Multi-Label Classification to Extract Implicit and Explicit Skills From Dutch Job Descriptions
Sepideh Mesbah, Randstad Groep
Conversational Entity Linking: Problem Definition and Datasets, SIGIR 2021
Hideaki Joko, Radboud University
Time-aware evidence ranking for fact-checking, Journal of Web Semantics 2021
Liesbeth Allein, KU Leuven
Understanding Multi-channel Customer Behavior in Retail, CIKM 2021
Mozhdeh Ariannezhad, University of Amsterdam
Lightning Talks — TBA ⚡
Supercalifragilisticexpialidocious: Why Using the “Right” Readability Formula in Children’s Web Search Matters, ECIR 2022
Garett Allen, TU Delft
Neural Information Retrieval for Educational Resources, ECIR Industry Day 2022
Carsten Schnober, WizeNoze
Embarrassingly shallow auto-encoders for dynamic collaborative filtering, Springer User Modeling and User-Adapted Interaction, Special Issue on Dynamic Recommender Systems and User Models 2022
Olivier Jeunen, Amazon
Leave No User Behind: Towards Improving the Utility of Recommender Systems for Non-mainstream Users, WSDM 2021
Zhe Roger Li, TU Delft
Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness, SIGIR 2021
Harrie Oosterhuis, Radboud University
IR Evaluation - An Industry Perspective
Gabriella Kazai, Microsoft
Measurement is key in the development of any product or service, including information retrieval (IR) systems. The challenge of developing IR metrics from which reliable conclusions can be drawn is no trivial matter. For example, issues in sampling, instrumentation, label collection or metric computation methodologies can invalidate an evaluation experiment or even lead to problems in the training of machine learning systems. In this talk I will reflect on some of the challenges when developing IR metrics and building offline evaluation pipelines at Bing.
I am a Principal Applied Scientist at Microsoft, tackling a range of IR evaluation challenges as part of the Bing Web and AI Science Group, focusing on offline evaluation, crowdsourcing, and metric development for various search scenarios, including organic web search, news, autosuggestions, and covering aspects from relevance to source credibility, trust and bias. Prior to this, I worked at two startups: As Lead Data Scientist at Mudano Ltd, I worked on an AI-driven project management system to deliver large-scale IT change projects in the financial sector; and as VP of Data Science at Semion Ltd, I led a team of data scientists implementing the AI behind the Lumi news recommender app, providing out-of-the-box personalised media content to users based on interests discovered from their Twitter streams. Before that, I was a postdoc at Microsoft Research after obtaining my PhD from the University of London under the supervision of Prof Mounia Lalmas. My research interests include information retrieval (IR), IR evaluation, human computation, gamification, recommender systems, social IR, information seeking behaviour, activity based personal information management.
Social Event 🍹
Informal drinks and optional social games via Zoom
Arjen de Vries