Portrait of Rose Nayoung Kwon is unavailable

Rose Nayoung Kwon

PhD - Université Laval
Supervisor
Christian Gagné
Research Topics
Deep Learning
Interpretability
Machine Learning Theory

Publications

Hidden-State Similarity Predicts Re-Elicitation After Inoculation Prompting
Rose Nayoung Kwon
Samy Mammeri
Sabyasachi Sahoo
Christian Gagné
Thomas Jiralerspong
Fine-tuning on narrow harmful tasks can cause emergent misalignment, where models generalize harmful behavior beyond the training distributi… (see more)on. Inoculation prompting can reduce this effect by explicitly eliciting the undesired behavior during training, but recent work shows that the behavior can reappear when evaluation prompts contain cues from the training context. We study what makes such prompts effective triggers. We find that textual similarity to the inoculation prompt is an incomplete predictor: prompts are more likely to re-elicit suppressed behavior when they induce activation states similar to those produced by the inoculation context. These findings advance our understanding of how inoculation prompting modulates conditional misalignment, and suggest that activation-space analysis can help identify when suppressed behaviors remain accessible under eval-time prompts.
2026-06-10
ICML.cc/2026/Workshop/Mech_Interp (poster)
openreview.net
openreview.net
When Does Interleaving Prevent Emergent Misalignment?
Samy Mammeri
Rose Nayoung Kwon
Chen Sun
Sabyasachi Sahoo
Christian Gagné
Thomas Jiralerspong
Large language models finetuned on narrow harmful tasks are prone to emergent misalignment (EM), where harmful behavior generalizes beyond t… (see more)he training distribution. Interleaving benign data during finetuning has been proposed as a mitigation, but recent work disagrees on whether it prevents EM. In this paper, we investigate this disagreement on Qwen-2.5 7B and 32B, and find that no single property of the interleaved data, taken in isolation, accounts for the gap. Instead, much of it traces to the evaluation itself, as the standard EM benchmark is sensitive to the length of the prompts it uses, and lengthening the evaluation prompts substantially shifts measured misalignment across model sizes. We then identify a region in the model's activations that predicts whether a given interleaved set will prevent EM, and show that reformulating benign data to fall within it substantially reduces EM on both 7B and 32B. This suggests that the standard EM benchmark, which relies on short prompts, may misrepresent the effectiveness of proposed mitigations.
2026-06-10
ICML.cc/2026/Workshop/Mech_Interp (poster)
openreview.net
openreview.net