Portrait of Sihui Wei is unavailable

Sihui Wei

Undergraduate - McGill University
Supervisor
Irina Rish
Research Topics
Natural Language Processing

Publications

Deep neural networks divide and conquer dihedral multiplication
Sihui Wei
Gavin McCracken
Gabriela Moisescu-Pareja
Harley Wiltzer
Doina Precup
Irina Rish
Jonathan Love
We find multilayer perceptrons and transformers both universally learn an instantiation of the same divide-and-conquer algorithm that requir… (see more)es only a logarithmic number of neural representations to solve dihedral multiplication. Clustering neurons based on similar activation behaviour reveals remarkably clear structure: each neural representation corresponds to a Cayley graph. To our knowledge, this is the first work that fully characterizes and describes all neural representations that are learnable on a dataset, while prior work on group multiplications studied neuron-level behavior, or preliminarily investigated cluster behavior. Thus, we can understand the algorithm networks universally learn at three levels of abstraction: 1) Neurons activate on coset or approximate coset structure of the dihedral group. 2) Groups of neurons together form neural representations that act to divide the dataset into different subproblems, being Cayley graphs, where the equivalence class of the answer is computed. 3) The global algorithm then linearly combines each neural representation (subproblem) together at the logits. This work provides a deep case study and provides the community with a very well understood toy model for interpretability, as well as makes steps toward proving the conjecture that DNNs will divide and conquer all group multiplication tasks.
2025-12-31
International Conference on Machine Learning (Accept (regular))
openreview.net
openreview.net