Portrait de Golnoosh Farnadi

Golnoosh Farnadi

Membre académique principal
Chaire en IA Canada-CIFAR
Professeure adjointe, McGill University
Professeure associée, Université de Montréal, Département d'informatique et de recherche opérationnelle
Chercheuse invitée, Google
Sujets de recherche
Apprentissage profond
Modèles génératifs

Biographie

Golnoosh Farnadi est professeure associée à l'École d'informatique de l'Université McGill et professeure associée à l'Université de Montréal. Elle est membre académique principal à Mila - Institut québécois d'intelligence artificielle et est titulaire d'une chaire CIFAR d'intelligence artificielle au Canada.

Mme Farnadi a fondé le laboratoire EQUAL à Mila / Université McGill, dont elle est l'une des principales chercheuses. Le laboratoire EQUAL (EQuity & EQuality Using AI and Learning algorithms) est un laboratoire de recherche de pointe dédié à l'avancement des domaines de l'équité algorithmique et de l'IA responsable.

Étudiants actuels

Doctorat - McGill
Co-superviseur⋅e :
Maîtrise recherche - UdeM
Superviseur⋅e principal⋅e :
Collaborateur·rice de recherche - UWindsor
Baccalauréat - McGill
Maîtrise recherche - McGill
Baccalauréat - McGill
Postdoctorat - McGill
Maîtrise recherche - McGill

Publications

Auditing is not Evaluating: LLM Audit Requires Dynamic, Contextual, Budget-aware and Reliable Evidence
Frank-LoRA: Federated Rank-Aware LoRA for Fine-Tuning Large Models
Kevin Kuo
Virginia Smith
From Infrastructure to Interface, the AI Value Chain Drives LLM Homogenization
Scaling Laws for Multimodal Data Mixtures
Ayush K. Tarun
Yixuan Xu
Imanol Schlag
Luke Zettlemoyer
Antoine Bosselut
Bridging Neural Tangent Kernel Analysis and Reinforcement Learning for Deep Function Approximation
Despite their widespread success across machine learning, the learning dynamics and generalization properties of neural networks remain poor… (voir plus)ly understood. The neural tangent kernel (NTK) provides an analytical characterization of the learning dynamics of infinitely wide neural networks and serves as a useful proxy for finite-width networks in supervised learning. In this work, we extend the NTK framework to reinforcement learning (RL) by introducing a temporal-difference (TD) flow formulation that accommodates time-varying mini-batches and TD targets, yielding a closed-form solution for value function evolution in an overparametrized setting. We also devise an optimization-based state-space exploration method that leverages the closed-form solution, coupled with constraints on the environment, to model the set of feasible states. On a deterministic grid environment, we validate that the proposed RL-based NTK formulation is indeed learning efficiently and adapting to the distribution shift in the training data distribution, and we demonstrate that the resulting exploration strategy improves sample efficiency. To our knowledge, this is the first work to extend NTK theory to characterize the learning dynamics of RL training with time-varying mini-batches and TD updates.
Privacy in Context Is Being Overlooked in the Age of General-Purpose LLMs
Yash More
Marco Romanelli
Ferdinando Fioretto
As large language models (LLMs) continue to grow in scope and capability, privacy concerns are on the rise. Since users increasingly interac… (voir plus)t with a single general-purpose system across many distinct contexts, there is a natural tendency to pursue a one-size-fits-all solution to privacy. Unfortunately, this overlooks a crucial insight from decades of privacy research: privacy is inherently contextual. In this paper, we argue that existing work in prompt privacy is drifting towards attempts to design a unified framework, which is highly misleading and can foster a false sense of security. Furthermore, we argue that even future work in this direction needs to wrestle with fundamental tensions across various desirable properties of prompt privacy. Thus, such continued attempts towards a unified framework need to be reconsidered. We recommend three critical paths forward: application-specific deployment of LLMs, context-specific evaluation of privacy frameworks, and better privacy literacy.
Let's Unlearn Stereotypes Before Decision-Making: Assessing the Impact of Intrinsic Bias Mitigation on Downstream Fairness in LLMs
Large Language Models (LLMs) are increasingly used in high-stakes decision-making systems, where biased predictions can reinforce social and… (voir plus) economic disparities. Although prior work has examined intrinsic representational bias and unfair downstream behavior separately, it remains unclear whether mitigating intrinsic bias leads to fairer downstream outcomes. We introduce Fairness-Aware Concept Unlearning (FACU), a model-level mitigation method that adapts concept unlearning to fairness-oriented representation balancing. Unlike suppression-based approaches, FACU explicitly regularizes probability differences between stereotypical and anti-stereotypical associations while preserving predictive performance and language modeling quality. We evaluate FACU across three open-source LLMs, multiple intrinsic bias benchmarks, and three socio-economic classification datasets using both frozen LLM embeddings and LoRA-fine-tuned classifiers. FACU produces statistically significant reductions in intrinsic gender bias that are associated with downstream fairness improvements across most evaluated settings, datasets, models, and fairness metrics, without significantly degrading predictive performance. Combining FACU with extrinsic mitigation methods, particularly counterfactual data augmentation, yields further fairness improvements. These findings suggest that fairness-aware intrinsic mitigation can support fairer LLM-based decision-making and that bias mitigation should be addressed across both model development and downstream deployment stages.
Loose Lips: Understanding Privacy Leakage in Agentic Dialogue
As LLM agents are increasingly deployed in high-stakes enterprise settings, understanding how they handle sensitive information in agentic d… (voir plus)ialogue has become critical. Privacy leakage in such settings requires reasoning about whether information flows respect the contextual norms under which they originated. We extend CIMemories to CIMEMORIES DIALOGUE, a benchmark of 4,410 two-agent dialogue instances grounded in Contextual Integrity theory. Using this benchmark, we systematically evaluate 7 open-weight models from the Qwen3 and Llama 3.1 families across conversation lengths of 1 and 6 turns under three prompting conditions. We find that task capability and privacy leakage are fundamentally coupled: more capable models are systematically leakier, with no model in our evaluation able to decouple the two. Thinking models consistently leak more than their non-thinking counterparts, a pattern we attribute to the extended reasoning surface created by chain-of-thought. Leakage compounds with conversation length, rising up to 3x from 1 to 6 turns. Our results suggest that the coupling between task capability and privacy leakage is not an artifact of misconfiguration but a structural feature of current agentic systems.
Beyond Fair Answers: Learning Fair and Unlearning Biased Reasoning in LLMs
Large Language Models (LLMs) have demonstrated strong reasoning capabilities on tasks with well-defined solutions, such as mathematical and … (voir plus)logical reasoning. However, reasoning capability does not necessarily imply unbiased reasoning. In socially ambiguous contexts, where the available evidence is insufficient to support one conclusion over another, models may rely on demographic stereotypes or unsupported social assumptions. Recent work shows that chain-of-thought (CoT) reasoning can contain social stereotypes and that biased reasoning steps are associated with stereotype expression and incorrect predictions. This motivates evaluating fairness in the reasoning process, not only in the final answer. We therefore ask: Can we train LLMs not only to produce fair answers, but to reason fairly? We study this question using a fairness-oriented adaptation of BBQ, where ambiguous cases test whether demographic alternatives receive comparable probability when evidence is insufficient, while disambiguated cases test evidence-grounded accuracy. Following reasoning-trace fine-tuning approaches such as STaR and ReGiFT, we fine-tune instruction-tuned LLMs on reasoning traces generated by stronger models and compare base and fine-tuned models under direct and CoT inference. We evaluate both final decisions and generated reasoning traces, and test generalization on SocioEconomicQA and CrowS-Pairs. Building on this analysis, we propose a reasoning-level fairness framework that jointly learns fair reasoning and unlearns biased reasoning. Evidence-grounded traces form the positive learning set, while traces containing unsupported demographic or stereotype-based inferences form the forget set. Our objective combines supervised fine-tuning on fair traces, a negative objective on biased traces, and KL regularization on unrelated reasoning data to preserve general reasoning ability. We evaluate whether this process improves selection of the fair answer and, more importantly, reduces the probability gap between gendered options when the context provides no evidence favoring either one. We separately measure accuracy in disambiguated contexts to ensure that the model still uses available evidence correctly. At the reasoning level, we evaluate stereotype-dependent inferences and the relative likelihood of fair versus biased reasoning traces. We compare against ReGiFT, standard CoT, and instruction-based reasoning baselines, and evaluate generalization across additional bias benchmarks while monitoring general reasoning performance for capability loss. This work moves fairness intervention from the final decision to the reasoning process that produces it. By combining fair-reasoning supervision, biased-reasoning unlearning, and capability preservation, we investigate whether LLMs can learn not only to give fair answers, but to reason without relying on unsupported demographic stereotypes.
LLM Persona Unlearning
Kemou Li
Qizhou Wang
Fengpeng Li
Jiantao Zhou
Pre-training equips large language models (LLMs) with a broad repertoire of behavioral patterns associated with roles, styles, values, and g… (voir plus)oals. Post-training teaches conditional enactment and makes a helpful Assistant the default, but it does not erase alternative modes from the weights; explicit prompts can therefore elicit personas that repeatedly shape judgment, language, and action. In open-weight settings, runtime controls can be removed, motivating persona unlearning: a weight-level edit that makes a designated persona difficult to elicit and enact on unseen contexts. We introduce PersonaUnlearnBench, a model-specific paired benchmark spanning six LLMs from three families and five personas, with aligned forget/retain sets, held-out instruction paraphrases, and four-axis evaluation. The benchmark shows that standard unlearning methods cannot reliably erase the target persona without sacrificing meaningful generation or general utility. We therefore propose PaCE, which compares target and desirable responses to the same questions to locate an internal behavior direction, then trains target-prompt states away from the target mode and toward the matched desirable response. Experiments show that PaCE consistently suppresses target personas with high response quality and useful counterpart behavior, at moderate utility cost. These results establish persona unlearning as a distinct behavior-level editing problem and a practical route toward persistent control of latent LLM response policies.
Preemptive LLM Unlearning against Forbidden Capability Acquisition via Gradient Sealing
Kemou Li
Qizhou Wang
Yue Wang
Fengpeng Li
Masashi Sugiyama
Jiantao Zhou
Open-weight LLMs are released not only as fixed products but also as substrates for downstream fine-tuning. This openness, however, creates … (voir plus)legal and ethical risks because users may misuse fine-tuning to instill illicit knowledge or enable hostile operations. Model providers therefore need apre-release defense against such acquisition, motivating the problem of preemptive unlearning. Unlike retrospective unlearning, which removes capabilities already present in a fixed model, preemptive unlearning seeks to prevent their acquisition under unseen attack data and future fine-tuning procedures. Despite its practical importance, this setting remains largely unexplored, presents distinct challenges, and is therefore the central focus of our work. We first verify that existing retrospective methods provide insufficient pre-release protection. Even when forbidden capabilities are suppressed in current outputs, forbidden-domain data can still induce gradients through internal pathways, enabling later acquisition. Motivated by this finding, we propose a gradient-sealing principle that blocks these pathways by pushing relevant pre-activations into the negative region, where ReLU-family activations exhibit zero or near-zero derivatives. Experiments across multiple LLM families demonstrate our stronger resistance to downstream acquisition than retrospective baselines, validating gradient sealing as an effective mechanism for pre-release protection.
LLJ Cards: Best practices for the Use of LLMs as Judges
Melina Medjdoub
Jackie Chi Kit Cheung
In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJ… (voir plus)s), these systems have been widely adopted by researchers and practitioners across a broad range of measurement tasks, driven by their strong performance, scalability, and cost-effectiveness relative to human judgment. However, a growing body of work has shown that the use of LLJs raise concerns about their validity and reliability as evaluators. Existing efforts to address these challenges have largely focused on developing bias-mitigation techniques and refining prompting strategies. While these approaches represent an important step forward, they primarily offer technical fixes and leave a more fundamental challenge unaddressed: the lack of standardized, transparent, and reproducible evaluation practices. In this paper, we introduce LLJ Cards, a framework that synthesizes best practices from measurement theory, natural language generation, and machine learning literature into practical guidelines for LLJ-based evaluations. While LLJs offer a promising path toward scalable evaluation, their effective use requires grounding in rigorous evaluation principles to ensure validity, reliability, and reproducibility. LLJ Cards addresses this need by providing a structured framework for applying these principles in the design and reporting of automated evaluations.