Fernando Diaz

2022-11-20

NeurIPS.cc/2022/Workshop/TSRML (accepté)

openreview.net

Measuring Commonality in Recommendation of Cultural Content: Recommender Systems to Enhance Cultural Citizenship

Andres Ferraro

Gustavo Ferreira

Georgina Born

2022-09-13

Proceedings of the 16th ACM Conference on Recommender Systems (publié)

Joint Multisided Exposure Fairness for Recommendation

Haolun Wu

Bhaskar Mitra

Chen Ma

Xue (Steve) Liu

Prior research on exposure fairness in the context of recommender systems has focused mostly on disparities in the exposure of individual or… (voir plus) groups of items to individual users of the system. The problem of how individual or groups of items may be systemically under or over exposed to groups of users, or even all users, has received relatively less attention. However, such systemic disparities in information exposure can result in observable social harms, such as withholding economic opportunities from historically marginalized groups (allocative harm) or amplifying gendered and racialized stereotypes (representational harm). Previously, Diaz et al. developed the expected exposure metric---that incorporates existing user browsing models that have previously been developed for information retrieval---to study fairness of content exposure to individual users. We extend their proposed framework to formalize a family of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in recommendation. Furthermore, we study and discuss the relationships between the different exposure fairness dimensions proposed in this paper, as well as demonstrate how stochastic ranking policies can be optimized towards said fairness goals.

2022-07-07

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (publié)

On Natural Language User Profiles for Transparent and Scrutable Recommendation

Filip Radlinski

Krisztian Balog

Lucas Dixon

Ben Wedin

Natural interaction with recommendation and personalized search systems has received tremendous attention in recent years. We focus on the c… (voir plus)hallenge of supporting people's understanding and control of these systems and explore a fundamentally new way of thinking about representation of knowledge in recommendation and personalization systems. Specifically, we argue that it may be both desirable and possible for algorithms that use natural language representations of users' preferences to be developed. We make the case that this could provide significantly greater transparency, as well as affordances for practical actionable interrogation of, and control over, recommendations. Moreover, we argue that such an approach, if successfully applied, may enable a major step towards systems that rely less on noisy implicit observations while increasing portability of knowledge of one's interests.

2022-07-07

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (publié)

Offline Retrieval Evaluation Without Evaluation Metrics

Andres Ferraro

Offline evaluation of information retrieval and recommendation has traditionally focused on distilling the quality of a ranking into a scala… (voir plus)r metric such as average precision or normalized discounted cumulative gain. We can use this metric to compare the performance of multiple systems for the same request. Although evaluation metrics provide a convenient summary of system performance, they also collapse subtle differences across users into a single number and can carry assumptions about user behavior and utility not supported across retrieval scenarios. We propose recall-paired preference (RPP), a metric-free evaluation method based on directly computing a preference between ranked lists. RPP simulates multiple user subpopulations per query and compares systems across these pseudo-populations. Our results across multiple search and recommendation tasks demonstrate that RPP substantially improves discriminative power while correlating well with existing metrics and being equally robust to incomplete data.

2022-07-07

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (publié)

Retrieval-Enhanced Machine Learning

Hamed Zamani

Mostafa Dehghani

Donald Metzler

Michael Bendersky

Although information access systems have long supportedpeople in accomplishing a wide range of tasks, we propose broadening the scope of use… (voir plus)rs of information access systems to include task-driven machines, such as machine learning models. In this way, the core principles of indexing, representation, retrieval, and ranking can be applied and extended to substantially improve model generalization, scalability, robustness, and interpretability. We describe a generic retrieval-enhanced machine learning (REML) framework, which includes a number of existing models as special cases. REML challenges information retrieval conventions, presenting opportunities for novel advances in core areas, including optimization. The REML research agenda lays a foundation for a new style of information access research and paves a path towards advancing machine learning and artificial intelligence.

2022-07-07

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (publié)

Retrieval-Enhanced Machine Learning

Hamed Zamani

Mostafa Dehghani

Donald Metzler

Michael Bendersky

Although information access systems have long supportedpeople in accomplishing a wide range of tasks, we propose broadening the scope of use… (voir plus)rs of information access systems to include task-driven machines, such as machine learning models. In this way, the core principles of indexing, representation, retrieval, and ranking can be applied and extended to substantially improve model generalization, scalability, robustness, and interpretability. We describe a generic retrieval-enhanced machine learning (REML) framework, which includes a number of existing models as special cases. REML challenges information retrieval conventions, presenting opportunities for novel advances in core areas, including optimization. The REML research agenda lays a foundation for a new style of information access research and paves a path towards advancing machine learning and artificial intelligence.

2022-05-02

ArXiv (preprint)

Joint Multisided Exposure Fairness for Recommendation

Haolun Wu

Bhaskar Mitra

Chen Ma

Xue (Steve) Liu

Prior research on exposure fairness in the context of recommender systems has focused mostly on disparities in the exposure of individual or… (voir plus) groups of items to individual users of the system. The problem of how individual or groups of items may be systemically under or over exposed to groups of users, or even all users, has received relatively less attention. However, such systemic disparities in information exposure can result in observable social harms, such as withholding economic opportunities from historically marginalized groups (allocative harm) or amplifying gendered and racialized stereotypes (representational harm). Previously, Diaz et al. developed the expected exposure metric---that incorporates existing user browsing models that have previously been developed for information retrieval---to study fairness of content exposure to individual users. We extend their proposed framework to formalize a family of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in recommendation. Furthermore, we study and discuss the relationships between the different exposure fairness dimensions proposed in this paper, as well as demonstrate how stochastic ranking policies can be optimized towards said fairness goals.

2022-04-29

ArXiv (preprint)

Offline Retrieval Evaluation Without Evaluation Metrics

Andres Ferraro

Offline evaluation of information retrieval and recommendation has traditionally focused on distilling the quality of a ranking into a scala… (voir plus)r metric such as average precision or normalized discounted cumulative gain. We can use this metric to compare the performance of multiple systems for the same request. Although evaluation metrics provide a convenient summary of system performance, they also collapse subtle differences across users into a single number and can carry assumptions about user behavior and utility not supported across retrieval scenarios. We propose recall-paired preference (RPP), a metric-free evaluation method based on directly computing a preference between ranked lists. RPP simulates multiple user subpopulations per query and compares systems across these pseudo-populations. Our results across multiple search and recommendation tasks demonstrate that RPP substantially improves discriminative power while correlating well with existing metrics and being equally robust to incomplete data.

2022-04-25

ArXiv (preprint)

Overview of the TREC 2021 Fair Ranking Track

Asia J. Biega

Michael D. Ekstrand

Sebastian Kohlmeier

The TREC Fair Ranking Track aims to provide a platform for participants to develop and evaluate novel retrieval algorithms that can provide … (voir plus)a fair exposure to a mixture of demographics or attributes, such as ethnicity, that are represented by relevant documents in response to a search query. For example, particular demographics or attributes can be represented by the documents' topical content or authors. The 2021 Fair Ranking Track adopted a resource allocation task. The task focused on supporting Wikipedia editors who are looking to improve the encyclopedia's coverage of topics under the purview of a WikiProject. WikiProject coordinators and/or Wikipedia editors search for Wikipedia documents that are in need of editing to improve the quality of the article. The 2021 Fair Ranking track aimed to ensure that documents that are about, or somehow represent, certain protected characteristics receive a fair exposure to the Wikipedia editors, so that the documents have an fair opportunity of being improved and, therefore, be well-represented in Wikipedia. The under-representation of particular protected characteristics in Wikipedia can result in systematic biases that can have a negative human, social, and economic impact, particularly for disadvantaged or protected societal groups.

2021-01-01

TREC (publié)

Overview of the TREC 2019 Fair Ranking Track

Asia J. Biega

Michael D. Ekstrand

Sebastian Kohlmeier

The goal of the TREC Fair Ranking track was to develop a benchmark for evaluating retrieval systems in terms of fairness to different conten… (voir plus)t providers in addition to classic notions of relevance. As part of the benchmark, we defined standardized fairness metrics with evaluation protocols and released a dataset for the fair ranking problem. The 2019 task focused on reranking academic paper abstracts given a query. The objective was to fairly represent relevant authors from several groups that were unknown at the system submission time. Thus, the track emphasized the development of systems which have robust performance across a variety of group definitions. Participants were provided with querylog data (queries, documents, and relevance) from Semantic Scholar. This paper presents an overview of the track, including the task definition, descriptions of the data and the annotation process, as well as a comparison of the performance of submitted systems.

2020-03-25

ArXiv (preprint)

Overview of the TREC 2019 Fair Ranking Track

Asia J. Biega

Michael D. Ekstrand

Sebastian Kohlmeier

2020-03-25

ArXiv (prépublication)