Nizar Islah

What Does Layer-Importance Reveal About Transformers and State-Space Models?

Transformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the anal… (voir plus)ytical knowledge built for transformers transfers to SSMs. We address this through the lens of layer importance which underpins compression, selective fine-tuning, and interpretability across both families. We decompose layer importance into two distinct notions. \emph{Necessity} captures how much the pretrained model depends on a layer's existing contribution, measured by the loss increase from bypassing it. \emph{Plasticity} captures where the model absorbs new information during fine-tuning, measured by the magnitude of task-specific weight updates. Our analysis reveals that the two families behave fundamentally differently: in every evaluated residual transformer up to

2026-09-14

arXiv (prépublication)

doi.org

arxiv.org

Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)

When post-trained language models fail on reasoning problems, the common test-time-scaling response is to spend more compute on additional a… (voir plus)ttempts, and the failed traces play no further role. We argue this discards a crucial signal; some failures come from unlucky sampling, where more rollouts help, while others are structural and resist resampling regardless of budget. We propose that failed traces encode recoverability structure: the inference-time signature of which test-time interventions can rescue a given failure. Three problem-level trajectory features, derived from the structure of available interventions, recover this structure from the distributional signature of failed rollouts, not their text. They cluster failures into stable regimes, characterize the failure topography of different post-training methods (

2026-06-02

arXiv (prépublication)

doi.org

arxiv.org

GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities

Victor May

Samira Ebrahimi Kahou

Massimo Caccia

The rapid evolution of software libraries poses a considerable hurdle for code generation, necessitating continuous adaptation to frequent v… (voir plus)ersion updates while preserving backward compatibility. While existing code evolution benchmarks provide valuable insights, they typically lack execution-based evaluation for generating code compliant with specific library versions. To address this, we introduce GitChameleon 2.0, a novel, meticulously curated dataset comprising 328 Python code completion problems, each conditioned on specific library versions and accompanied by executable unit tests. GitChameleon 2.0 rigorously evaluates the capacity of contemporary large language models (LLMs), LLM-powered agents, code assistants, and RAG systems to perform version-conditioned code generation that demonstrates functional accuracy through execution. Our extensive evaluations indicate that state-of-the-art systems encounter significant challenges with this task; enterprise models achieving baseline success rates in the 48-51% range, underscoring the intricacy of the problem. By offering an execution-based benchmark emphasizing the dynamic nature of code libraries, GitChameleon 2.0 enables a clearer understanding of this challenge and helps guide the development of more adaptable and dependable AI code generation methods. We make the dataset and evaluation code publicly available at https://github.com/mrcabbage972/GitChameleonBenchmark.

2025-12-31

Association for Computational Linguistics (publié)

doi.org

openreview.net

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models

Matthew D Riemer

Tsuguchika Tabaru

Hiroaki Kingetsu

A. Chandar

Irina Rish

2025-09-21

NeurIPS.cc/2025/Workshop/WiML (publié)

doi.org

openreview.net

Learning to combine top-down context and feed-forward representations under ambiguity with apical and basal dendrites

Guillaume Etter

Busra Tugce Gurbuz

One of the hallmark features of neocortical anatomy is the presence of extensive top-down projections into primary sensory areas, with many … (voir plus)impinging on the distal apical dendrites of pyramidal neurons. While it is known that they exert a modulatory effect, altering the gain of responses, their functional role remains an active area of research. It is hypothesized that these top-down projections carry contextual information that can help animals to resolve ambiguities in sensory data. One proposed mechanism of contextual integration is a non-linear integration of distinct input streams at apical and basal dendrites of pyramidal neurons. Computationally, however, it is yet to be demonstrated how such an architecture could leverage distinct compartments for flexible contextual integration and sensory processing when both sensory and context signals can be unreliable. Here, we implement an augmented deep neural network with distinct apical and basal compartments that integrates a) contextual information from top-down projections to apical compartments, and b) sensory representations driven by bottom-up projections to basal compartments, via a biophysically inspired rule. In addition, we develop a new multi-scenario contextual integration task using a generative image modeling approach. In addition to generalizing previous contextual integration tasks, it better captures the diversity of scenarios where neither contextual nor sensory information are fully reliable. To solve this task, this model successfully learns to select among integration strategies. We find that our model outperforms those without the "apical prior" when contextual information contradicts sensory input. Altogether, this suggests that the apical prior and biophysically inspired integration rule could be key components necessary for handling the ambiguities that animals encounter in the diverse contexts of the real world.

2025-06-22

Cerebral Cortex (New York, NY) (publié)

doi.org

arxiv.org

GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models

Terry Yue Zhuo

Massimo Caccia