Marawan Gamal

marawan.gamal@mila.quebec

PhD - Université de Montréal

Supervisor

Guillaume Rabusseau

Research Topics

Computer Vision

Natural Language Processing

Tensor Factorization

Website

Google Scholar

GitHub

Publications

Merging Adapted Models via Data-Free Covariance Estimation

Marawan Gamal

Derek Tam

Pascal Junior Tikeng Notsawo

Colin Raffel

Guillaume Rabusseau

Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While s… (see more)ome merging methods can approach the performance of multitask training, they are often heuristically motivated and lack theoretical justification. A principled alternative is to pose model merging as a layer-wise optimization problem that directly minimizes interference between tasks. However, this formulation requires estimating per-layer covariance matrices from data, which may not be available when performing merging. In contrast, many of the heuristically-motivated methods do not require auxiliary data, making them practically advantageous. In this work, we revisit the interference minimization framework and show that, under certain conditions, covariance matrices can be estimated directly from difference matrices, eliminating the need for data while also reducing computational costs. We validate our approach across vision and language benchmarks on models ranging from

2026-08-03

CBW @ Conference on Language Modeling (poster)

openreview.net