Portrait de Weixu Zhang

Weixu Zhang

weixu.zhang@mila.quebec
Doctorat - McGill
Superviseur⋅e principal⋅e
Xue (Steve) Liu
Sujets de recherche
Agent basé sur un LLM
Grands modèles de langage (LLM)
Traitement du langage naturel

Publications

Discrete Diffusion Models: A Unified Framework from Tokenization to Generation
Ye Yuan
Weien Li
Rui Song
Zeyu Li
Haochen Liu
Xiangyu Kong
Zixuan Dong
Linfeng Du
Zipeng Sun
Weixu Zhang
Jiaxin Huang
Changjiang Han
Yonghan Yang
Zichen Zhao
Xiuyuan Hu
Haolun Wu
Yankai Chen
Fengran Mo
Jikun Kang
Bowei He … (voir 2 de plus)
Philip S. Yu
Xue Liu
Discrete denoising diffusion models (DDMs) have recently emerged as a compelling alternative to autoregressive (AR) modeling for discrete da… (voir plus)ta, offering parallel generation and iterative global refinement capabilities. Unlike continuous diffusion, where the state space is fixed, DDMs are fundamentally shaped by how the discrete state space is constructed: the tokenization scheme, the vocabulary topology, and domain-specific structural alphabets. This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space. Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space. The framework further exposes common design trade-offs across training objectives, inference algorithms, scaling behavior, systems optimization, and evaluation protocols, suggesting several promising directions for future research.
2026-07-14
arXiv (prépublication)
doi.org
arxiv.org
Context-Fidelity Boosting: Enhancing Faithful Generation through Watermark-Inspired Decoding
Weixu Zhang
Fanghua Ye
Qiang Gao
Jian Li
Haolun Wu
Yuxing Tian
Sijing Duan
Nan Du
Xiaolong Li
Xue Liu
Large language models (LLMs) often produce content that contradicts or overlooks information provided in the input context, a phenomenon kno… (voir plus)wn as faithfulness hallucination. In this paper, we propose Context-Fidelity Boosting (CFB), a lightweight and general decoding-time framework that reduces such hallucinations by increasing the generation probability of source-supported tokens. Motivated by logit-shaping principles from watermarking techniques, CFB applies additive token-level logit adjustments based on a token's degree of support from the input context. Specifically, we develop three boosting strategies: static boosting, which applies a fixed bias to source-supported tokens; context-aware boosting, which scales this bias using the divergence between next-token distributions with and without context; and token-aware boosting, which further redistributes the adaptive bias according to local relevance estimated from source-position attention and source-scoped semantic similarity. CFB requires no retraining or architectural changes, making it compatible with a wide range of LLMs. Experiments on summarization and question answering tasks across multiple open-source LLMs show that CFB consistently improves faithfulness metrics with minimal generation overhead. Our implementation is fully open-sourced.
2026-04-23
arXiv (prépublication)
doi.org
arxiv.org
Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization
Weixu Zhang
Ye Yuan
Changjiang Han
Yuxing Tian
Zipeng Sun
Linfeng Du
Jikun Kang
Hong Kang
Xue Liu
Haolun Wu
Large Language Models (LLMs) exhibit strong implicit personalization ability, yet most existing approaches treat this behavior as a black bo… (voir plus)x, relying on prompt engineering or fine tuning on user data. In this work, we adopt a mechanistic interpretability perspective and hypothesize the existence of a sparse set of Preference Heads, attention heads that encode user specific stylistic and topical preferences and exert a causal influence on generation. We introduce Differential Preference Steering (DPS), a training free framework that (1) identifies Preference Heads through causal masking analysis and (2) leverages them for controllable and interpretable personalization at inference time. DPS computes a Preference Contribution Score (PCS) for each attention head, directly measuring its causal impact on user aligned outputs. During decoding, we contrast model predictions with and without Preference Heads, amplifying the difference between personalized and generic logits to selectively strengthen preference aligned continuations. Experiments on widely used personalization benchmarks across multiple LLMs demonstrate consistent gains in personalization fidelity while preserving content coherence and low computational overhead. Beyond empirical improvements, DPS provides a mechanistic explanation of where and how personalization emerges within transformer architectures. Our implementation is publicly available.
2026-04-23
arXiv (prépublication)
doi.org
arxiv.org