Portrait of Fernando Diaz is unavailable

Fernando Diaz

Core Industry Member
fernando.diaz@mila.quebec
Associate Professor, Carnegie Mellon University, School of Computer Science, Language Technologies Institutes
Adjunct Professor, McGill University, School of Computer Science
Research Scientist, Google Pittsburgh

Biography

Fernando Diaz is an associate professor at Carnegie Mellon University's School of Computer Science, a research scientist at Google Pittsburgh, and an adjunct professor in McGill University’s School of Computer Science.

Diaz’s expertise lies in the formal study of the search for small fragments of information in large data sets. His interests include distributed approaches to web-based documentary research, interactive and faceted research, the exploration of temporal models from news and queries, multilingual information research, and graph-based methods.

Diaz’s primary research interest is information retrieval, i.e., the formal study of searching large collections of data for small bits of information. The most familiar form of information retrieval is the web search, where users search a collection of webpages for one or a few relevant webpages. However, information retrieval goes far beyond web searches to include processes like cross-lingual retrieval, personalization, desktop search and interactive retrieval.

Diaz’s research experience includes distributed information retrieval approaches to web searching, interactive and faceted retrieval, mining of temporal patterns from news and query logs, cross-lingual information retrieval, graph-based retrieval methods, and exploiting information from multiple corpora.

For his PhD research, Diaz studied the relationship between document clustering and document scoring for retrieval using methods from machine learning and statistics. As a result, he developed an algorithm for system self-assessment and self-tuning that significantly improves the performance of retrieval algorithms across a variety of corpora.

Publications

Measuring Commonality in Recommendation of Cultural Content: Recommender Systems to Enhance Cultural Citizenship
Andres Ferraro
Gustavo Ferreira
Fernando Diaz
Georgina Born
2022-09-13
Proceedings of the 16th ACM Conference on Recommender Systems (published)
doi.org
arxiv.org
Joint Multisided Exposure Fairness for Recommendation
Haolun Wu
Bhaskar Mitra
Chen Ma
Fernando Diaz
Xue (Steve) Liu
Prior research on exposure fairness in the context of recommender systems has focused mostly on disparities in the exposure of individual or… (see more) groups of items to individual users of the system. The problem of how individual or groups of items may be systemically under or over exposed to groups of users, or even all users, has received relatively less attention. However, such systemic disparities in information exposure can result in observable social harms, such as withholding economic opportunities from historically marginalized groups (allocative harm) or amplifying gendered and racialized stereotypes (representational harm). Previously, Diaz et al. developed the expected exposure metric---that incorporates existing user browsing models that have previously been developed for information retrieval---to study fairness of content exposure to individual users. We extend their proposed framework to formalize a family of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in recommendation. Furthermore, we study and discuss the relationships between the different exposure fairness dimensions proposed in this paper, as well as demonstrate how stochastic ranking policies can be optimized towards said fairness goals.
2022-07-07
Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (published)
doi.org
arxiv.org
On Natural Language User Profiles for Transparent and Scrutable Recommendation
Filip Radlinski
Krisztian Balog
Fernando Diaz
Lucas Dixon
Ben Wedin
Natural interaction with recommendation and personalized search systems has received tremendous attention in recent years. We focus on the c… (see more)hallenge of supporting people's understanding and control of these systems and explore a fundamentally new way of thinking about representation of knowledge in recommendation and personalization systems. Specifically, we argue that it may be both desirable and possible for algorithms that use natural language representations of users' preferences to be developed. We make the case that this could provide significantly greater transparency, as well as affordances for practical actionable interrogation of, and control over, recommendations. Moreover, we argue that such an approach, if successfully applied, may enable a major step towards systems that rely less on noisy implicit observations while increasing portability of knowledge of one's interests.
2022-07-07
Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (published)
doi.org
arxiv.org
Offline Retrieval Evaluation Without Evaluation Metrics
Fernando Diaz
Andres Ferraro
Offline evaluation of information retrieval and recommendation has traditionally focused on distilling the quality of a ranking into a scala… (see more)r metric such as average precision or normalized discounted cumulative gain. We can use this metric to compare the performance of multiple systems for the same request. Although evaluation metrics provide a convenient summary of system performance, they also collapse subtle differences across users into a single number and can carry assumptions about user behavior and utility not supported across retrieval scenarios. We propose recall-paired preference (RPP), a metric-free evaluation method based on directly computing a preference between ranked lists. RPP simulates multiple user subpopulations per query and compares systems across these pseudo-populations. Our results across multiple search and recommendation tasks demonstrate that RPP substantially improves discriminative power while correlating well with existing metrics and being equally robust to incomplete data.
2022-07-07
Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (published)
doi.org
arxiv.org
Retrieval-Enhanced Machine Learning
Hamed Zamani
Fernando Diaz
Mostafa Dehghani
Donald Metzler
Michael Bendersky
Although information access systems have long supportedpeople in accomplishing a wide range of tasks, we propose broadening the scope of use… (see more)rs of information access systems to include task-driven machines, such as machine learning models. In this way, the core principles of indexing, representation, retrieval, and ranking can be applied and extended to substantially improve model generalization, scalability, robustness, and interpretability. We describe a generic retrieval-enhanced machine learning (REML) framework, which includes a number of existing models as special cases. REML challenges information retrieval conventions, presenting opportunities for novel advances in core areas, including optimization. The REML research agenda lays a foundation for a new style of information access research and paves a path towards advancing machine learning and artificial intelligence.
2022-07-07
Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (published)
doi.org
arxiv.org
Retrieval-Enhanced Machine Learning
Hamed Zamani
Fernando Diaz
Mostafa Dehghani
Donald Metzler
Michael Bendersky
Although information access systems have long supportedpeople in accomplishing a wide range of tasks, we propose broadening the scope of use… (see more)rs of information access systems to include task-driven machines, such as machine learning models. In this way, the core principles of indexing, representation, retrieval, and ranking can be applied and extended to substantially improve model generalization, scalability, robustness, and interpretability. We describe a generic retrieval-enhanced machine learning (REML) framework, which includes a number of existing models as special cases. REML challenges information retrieval conventions, presenting opportunities for novel advances in core areas, including optimization. The REML research agenda lays a foundation for a new style of information access research and paves a path towards advancing machine learning and artificial intelligence.
2022-05-02
ArXiv (preprint)
doi.org
arxiv.org
Joint Multisided Exposure Fairness for Recommendation
Haolun Wu
Bhaskar Mitra
Chen Ma
Fernando Diaz
Xue (Steve) Liu
Prior research on exposure fairness in the context of recommender systems has focused mostly on disparities in the exposure of individual or… (see more) groups of items to individual users of the system. The problem of how individual or groups of items may be systemically under or over exposed to groups of users, or even all users, has received relatively less attention. However, such systemic disparities in information exposure can result in observable social harms, such as withholding economic opportunities from historically marginalized groups (allocative harm) or amplifying gendered and racialized stereotypes (representational harm). Previously, Diaz et al. developed the expected exposure metric---that incorporates existing user browsing models that have previously been developed for information retrieval---to study fairness of content exposure to individual users. We extend their proposed framework to formalize a family of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in recommendation. Furthermore, we study and discuss the relationships between the different exposure fairness dimensions proposed in this paper, as well as demonstrate how stochastic ranking policies can be optimized towards said fairness goals.
2022-04-29
ArXiv (preprint)
doi.org
arxiv.org
Offline Retrieval Evaluation Without Evaluation Metrics
Fernando Diaz
Andres Ferraro
Offline evaluation of information retrieval and recommendation has traditionally focused on distilling the quality of a ranking into a scala… (see more)r metric such as average precision or normalized discounted cumulative gain. We can use this metric to compare the performance of multiple systems for the same request. Although evaluation metrics provide a convenient summary of system performance, they also collapse subtle differences across users into a single number and can carry assumptions about user behavior and utility not supported across retrieval scenarios. We propose recall-paired preference (RPP), a metric-free evaluation method based on directly computing a preference between ranked lists. RPP simulates multiple user subpopulations per query and compares systems across these pseudo-populations. Our results across multiple search and recommendation tasks demonstrate that RPP substantially improves discriminative power while correlating well with existing metrics and being equally robust to incomplete data.
2022-04-25
ArXiv (preprint)
doi.org
arxiv.org
Overview of the TREC 2021 Fair Ranking Track
Asia J. Biega
Fernando Diaz
Michael D. Ekstrand
Sebastian Kohlmeier
The TREC Fair Ranking Track aims to provide a platform for participants to develop and evaluate novel retrieval algorithms that can provide … (see more)a fair exposure to a mixture of demographics or attributes, such as ethnicity, that are represented by relevant documents in response to a search query. For example, particular demographics or attributes can be represented by the documents' topical content or authors. The 2021 Fair Ranking Track adopted a resource allocation task. The task focused on supporting Wikipedia editors who are looking to improve the encyclopedia's coverage of topics under the purview of a WikiProject. WikiProject coordinators and/or Wikipedia editors search for Wikipedia documents that are in need of editing to improve the quality of the article. The 2021 Fair Ranking track aimed to ensure that documents that are about, or somehow represent, certain protected characteristics receive a fair exposure to the Wikipedia editors, so that the documents have an fair opportunity of being improved and, therefore, be well-represented in Wikipedia. The under-representation of particular protected characteristics in Wikipedia can result in systematic biases that can have a negative human, social, and economic impact, particularly for disadvantaged or protected societal groups.
2021-01-01
TREC (published)
doi.org
arxiv.org
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega
Fernando Diaz
Michael D. Ekstrand
Sebastian Kohlmeier
2020-03-25
ArXiv (preprint)
arxiv.org
arxiv.org
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega
Fernando Diaz
Michael D. Ekstrand
Sebastian Kohlmeier
The goal of the TREC Fair Ranking track was to develop a benchmark for evaluating retrieval systems in terms of fairness to different conten… (see more)t providers in addition to classic notions of relevance. As part of the benchmark, we defined standardized fairness metrics with evaluation protocols and released a dataset for the fair ranking problem. The 2019 task focused on reranking academic paper abstracts given a query. The objective was to fairly represent relevant authors from several groups that were unknown at the system submission time. Thus, the track emphasized the development of systems which have robust performance across a variety of group definitions. Participants were provided with querylog data (queries, documents, and relevance) from Semantic Scholar. This paper presents an overview of the track, including the task definition, descriptions of the data and the annotation process, as well as a comparison of the performance of submitted systems.
2020-03-25
ArXiv (preprint)
arxiv.org
arxiv.org
On the Evaluation of Common-Sense Reasoning in Natural Language Understanding
Paul Trichelair
Ali Emami
Jackie Cheung
Adam Trischler
Kaheer Suleman
Fernando Diaz
The NLP and ML communities have long been interested in developing models capable of common-sense reasoning, and recent works have significa… (see more)ntly improved the state of the art on benchmarks like the Winograd Schema Challenge (WSC). Despite these advances, the complexity of tasks designed to test common-sense reasoning remains under-analyzed. In this paper, we make a case study of the Winograd Schema Challenge and, based on two new measures of instance-level complexity, design a protocol that both clarifies and qualifies the results of previous work. Our protocol accounts for the WSC's limited size and variable instance difficulty, properties common to other common-sense benchmarks. Accounting for these properties when assessing model results may prevent unjustified conclusions.
2018-11-05
arXiv.org (preprint)
dblp.uni-trier.de