Portrait de Hong Kang n'est pas disponible

Hong Kang

hong.kang@mila.quebec
Doctorat - McGill
Superviseur⋅e principal⋅e
Xue (Steve) Liu
Sujets de recherche
Apprentissage décentralisé
Blockchain

Publications

Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization
Weixu Zhang
Ye Yuan
Changjiang Han
Yuxing Tian
Zipeng Sun
Linfeng Du
Jikun Kang
Hong Kang
Xue Liu
Haolun Wu
Large Language Models (LLMs) exhibit strong implicit personalization ability, yet most existing approaches treat this behavior as a black bo… (voir plus)x, relying on prompt engineering or fine tuning on user data. In this work, we adopt a mechanistic interpretability perspective and hypothesize the existence of a sparse set of Preference Heads, attention heads that encode user specific stylistic and topical preferences and exert a causal influence on generation. We introduce Differential Preference Steering (DPS), a training free framework that (1) identifies Preference Heads through causal masking analysis and (2) leverages them for controllable and interpretable personalization at inference time. DPS computes a Preference Contribution Score (PCS) for each attention head, directly measuring its causal impact on user aligned outputs. During decoding, we contrast model predictions with and without Preference Heads, amplifying the difference between personalized and generic logits to selectively strengthen preference aligned continuations. Experiments on widely used personalization benchmarks across multiple LLMs demonstrate consistent gains in personalization fidelity while preserving content coherence and low computational overhead. Beyond empirical improvements, DPS provides a mechanistic explanation of where and how personalization emerges within transformer architectures. Our implementation is publicly available.
2026-04-23
arXiv (prépublication)
doi.org
arxiv.org