Videogames provide a promising framework to understand brain activity in a rich, engaging, and active environment, in contrast to mostly pas
… (voir plus)sive tasks currently dominating the field, such as image viewing. Analyzing videogames neuroimaging data is however challenging, and relies on time-intensive manual annotations of game events, based on somewhat arbitrary rules. Here, we introduce an innovative approach using Artificial Neural networks (ANN) and brain encoding techniques to generate activation maps associated with videogame behaviour using functional magnetic resonance imaging (fMRI). As individual behavior is highly variable across subjects in complex environments, we hypothesized that ANNs need to account for subject-specific behavior to properly capture brain dynamics. In this study, we used data collected while subjects played Shinobi III: Return of the Ninja Master (Sega, 1993), an action-platformer videogame. Using imitation learning, we trained an ANN to play the game while closely replicating the unique gameplay style of individual participants. We found that hidden layers of our imitation learning model successfully encoded task-relevant neural representations, and predicted individual brain dynamics with higher accuracy than models trained on other subjects’ gameplay. Individual-specific models also outperformed a number of baselines to predict brain activity, such as pixel inputs, or button presses. The highest correlations between layer activations and brain signals were observed in biologically plausible brain areas, i.e. somatosensory, attention, and visual networks. Our results demonstrate that training subject-specific ANNs can successfully uncover brain correlates of complex behaviour. This new method combining imitation learning, brain imaging, and videogames opens new research avenues to study decision-making and psychomotor task solving in naturalistic and complex environments.
