Le traitement du langage naturel à l'ère de l'IA générative
Rejoignez-nous à Mila en octobre pour un atelier de trois jour visant à explorer le potentiel de transformation des technologies langagières et leurs implications pour la société.
Ce programme est conçu pour fournir aux professionnel·le·s travaillant dans le domaine de la politique une compréhension fondamentale de la technologie de l'IA.
Nous utilisons des témoins pour analyser le trafic et l’utilisation de notre site web, afin de personnaliser votre expérience. Vous pouvez désactiver ces technologies à tout moment, mais cela peut restreindre certaines fonctionnalités du site. Consultez notre Politique de protection de la vie privée pour en savoir plus.
Paramètre des cookies
Vous pouvez activer et désactiver les types de cookies que vous souhaitez accepter. Cependant certains choix que vous ferez pourraient affecter les services proposés sur nos sites (ex : suggestions, annonces personnalisées, etc.).
Cookies essentiels
Ces cookies sont nécessaires au fonctionnement du site et ne peuvent être désactivés. (Toujours actif)
Cookies analyse
Acceptez-vous l'utilisation de cookies pour mesurer l'audience de nos sites ?
Multimedia Player
Acceptez-vous l'utilisation de cookies pour afficher et vous permettre de regarder les contenus vidéo hébergés par nos partenaires (YouTube, etc.) ?
Publications
Efficient 1D Grouped Convolution for PyTorch a Case Study: Fast On-Device Fine-Tuning for SqueezeBERT
Grouped convolution has been observed to be an effective approximation for convolution in many DNN applications. For example, SqueezeBERT, w… (voir plus)hich is a light and fast BERT language processing model, utilizes 1D grouped convolutions. Though SqueezeBERT is well-optimized for inference on edge devices, it suffers from poor memory management during fine-tuning (training). This results in longer fine-tuning time on resource-limited GPUs compared to the original BERT model, BERT-base, despite being specifically designed for edge devices. We study this behavior and show that this poor memory management originates from the use of 1D grouped convolutions in SqueezeBERT. We re-implement 1D grouped convolutions using fully-connected layers, addressing the poor memory allocation and data locality of 1D grouped convolutions. We show that our method is well-suited for edge devices with limited memory; further, it has a negligible effect on inference speed. When utilizing our method, we observe a 42 % reduction in fine-tuning time for SqueezeBERT on edge devices.
2023-07-19
2023 IEEE 34th International Conference on Application-specific Systems, Architectures and Processors (ASAP) (publié)
Neurons in the brain have rich and adaptive input-output properties. Features such as heterogeneous f-I curves and spike frequency adaptatio… (voir plus)n are known to place single neurons in optimal coding regimes when facing changing stimuli. Yet, it is still unclear how brain circuits exploit single-neuron flexibility, and how network-level requirements may have shaped such cellular function. To answer this question, a multi-scaled approach is needed where the computations of single neurons and neural circuits must be considered as a complete system. In this work, we use artificial neural networks to systematically investigate single-neuron input-output adaptive mechanisms, optimized in an end-to-end fashion. Throughout the optimization process, each neuron has the liberty to modify its nonlinear activation function, parametrized to mimic f-I curves of biological neurons, and to learn adaptation strategies to modify activation functions in real-time during a task. We find that such networks show much-improved robustness to noise and changes in input statistics. Importantly, we find that this procedure recovers precise coding strategies found in biological neurons, such as gain scaling and fractional order differentiation/integration. Using tools from dynamical systems theory, we analyze the role of these emergent single-neuron properties and argue that neural diversity and adaptation play an active regularization role, enabling neural circuits to optimally propagate information across time.
There is rich variety in the activity of single neurons recorded during behaviour. Yet, these diverse single neuron responses can be well de… (voir plus)scribed by relatively few patterns of neural co-modulation. The study of such low-dimensional structure of neural population activity has provided important insights into how the brain generates behaviour. Virtually all of these studies have used linear dimensionality reduction techniques to estimate these population-wide co-modulation patterns, constraining them to a flat “neural manifold”. Here, we hypothesised that since neurons have nonlinear responses and make thousands of distributed and recurrent connections that likely amplify such nonlinearities, neural manifolds should be intrinsically nonlinear. Combining neural population recordings from monkey motor cortex, mouse motor cortex, mouse striatum, and human motor cortex, we show that: 1) neural manifolds are intrinsically nonlinear; 2) the degree of their nonlinearity varies across architecturally distinct brain regions; and 3) manifold nonlinearity becomes more evident during complex tasks that require more varied activity patterns. Simulations using recurrent neural network models confirmed the proposed relationship between circuit connectivity and manifold nonlinearity, including the differences across architecturally distinct regions. Thus, neural manifolds underlying the generation of behaviour are inherently nonlinear, and properly accounting for such nonlinearities will be critical as neuroscientists move towards studying numerous brain regions involved in increasingly complex and naturalistic behaviours.
We study the problem of planning under model uncertainty in an online meta-reinforcement learning (RL) setting where an agent is presented w… (voir plus)ith a sequence of related tasks with limited interactions per task. The agent can use its experience in each task and across tasks to estimate both the transition model and the distribution over tasks. We propose an algorithm to meta-learn the underlying structure across tasks, utilize it to plan in each task, and upper-bound the regret of the planning loss. Our bound suggests that the average regret over tasks decreases as the number of tasks increases and as the tasks are more similar. In the classical single-task setting, it is known that the planning horizon should depend on the estimated model's accuracy, that is, on the number of samples within task. We generalize this finding to meta-RL and study this dependence of planning horizons on the number of tasks. Based on our theoretical findings, we derive heuristics for selecting slowly increasing discount factors, and we validate its significance empirically.
This paper first presents a time-series impact analysis of charging electric vehicles (EVs) to loading levels of power network equipment con… (voir plus)sidering stochasticity in charging habits of EV owners. A novel incentive-based mitigation strategy is then designed to shift the EV charging from the peak hours when the equipment is overloaded to the off-peak hours and maintain equipment service lifetime. The incentive level and corresponding contributions from customers to alter their EV charging habits are determined by a search algorithm and a constrained optimization problem. The strategy is illustrated on a modified version of the IEEE 8500 feeder with a high EV penetration to mitigate overloads on the substation transformer.
2023-07-16
2023 IEEE Power & Energy Society General Meeting (PESGM) (publié)
The role of a decision support system is to gather, synthesize and present information in order to make informed decisions. In this project,… (voir plus) a mapping platform and a decision support system are proposed to present beekeeping data in Quebec. A complete review of the data and factors influencing honey production must first be carried out. The decision support system will be designed according to the nature of the data and access to available technologies. Continuous and real-time data management must be configured to make data interoperable. Multi-dimensional data loading tools will need to be configured to display data and analyses in a dashboard. Beekeepers will be able to optimize or move their hives according to their interpretation of the results displayed in the decision support system.
2023-07-16
IEEE International Geoscience and Remote Sensing Symposium (publié)
In recent years, the integration of Machine Learning (ML) models with Operation Research (OR) tools has gained popularity across diverse app… (voir plus)lications, including cancer treatment, algorithmic configuration, and chemical process optimization. In this domain, the combination of ML and OR often relies on representing the ML model output using Mixed Integer Programming (MIP) formulations. Numerous studies in the literature have developed such formulations for many ML predictors, with a particular emphasis on Artificial Neural Networks (ANNs) due to their significant interest in many applications. However, ANNs frequently contain a large number of parameters, resulting in MIP formulations that are impractical to solve, thereby impeding scalability. In fact, the ML community has already introduced several techniques to reduce the parameter count of ANNs without compromising their performance, since the substantial size of modern ANNs presents challenges for ML applications as it significantly impacts computational efforts during training and necessitates significant memory resources for storage. In this paper, we showcase the effectiveness of pruning, one of these techniques, when applied to ANNs prior to their integration into MIPs. By pruning the ANN, we achieve significant improvements in the speed of the solution process. We discuss why pruning is more suitable in this context compared to other ML compression techniques, and we identify the most appropriate pruning strategies. To highlight the potential of this approach, we conduct experiments using feed-forward neural networks with multiple layers to construct adversarial examples. Our results demonstrate that pruning offers remarkable reductions in solution times without hindering the quality of the final decision, enabling the resolution of previously unsolvable instances.
In recent years, the integration of Machine Learning (ML) models with Operation Research (OR) tools has gained popularity across diverse app… (voir plus)lications, including cancer treatment, algorithmic configuration, and chemical process optimization. In this domain, the combination of ML and OR often relies on representing the ML model output using Mixed Integer Programming (MIP) formulations. Numerous studies in the literature have developed such formulations for many ML predictors, with a particular emphasis on Artificial Neural Networks (ANNs) due to their significant interest in many applications. However, ANNs frequently contain a large number of parameters, resulting in MIP formulations that are impractical to solve, thereby impeding scalability. In fact, the ML community has already introduced several techniques to reduce the parameter count of ANNs without compromising their performance, since the substantial size of modern ANNs presents challenges for ML applications as it significantly impacts computational efforts during training and necessitates significant memory resources for storage. In this paper, we showcase the effectiveness of pruning, one of these techniques, when applied to ANNs prior to their integration into MIPs. By pruning the ANN, we achieve significant improvements in the speed of the solution process. We discuss why pruning is more suitable in this context compared to other ML compression techniques, and we identify the most appropriate pruning strategies. To highlight the potential of this approach, we conduct experiments using feed-forward neural networks with multiple layers to construct adversarial examples. Our results demonstrate that pruning offers remarkable reductions in solution times without hindering the quality of the final decision, enabling the resolution of previously unsolvable instances.
The quality of cervical spinal cord images can be improved by the use of tailored radiofrequency (RF) coil solutions for ultrahigh field ima… (voir plus)ging; however, very few commercial and research 7‐T RF coils currently exist for the spinal cord, and in particular, those with parallel transmission (pTx) capabilities. This work presents the design, testing, and validation of a pTx/Rx coil for the human neck and cervical/upper thoracic spinal cord. The pTx portion is composed of eight dipoles to ensure high homogeneity over this large region of the spinal cord. The Rx portion is made up of twenty semiadaptable overlapping loops to produce high signal‐to‐noise ratio (SNR) across the patient population. The coil housing is designed to facilitate patient positioning and comfort, while also being tight fitting to ensure high sensitivity. We demonstrate RF shimming capabilities to optimize B1+ uniformity, power efficiency, and/or specific absorption rate efficiency. B1+ homogeneity, SNR, and g‐factor were evaluated in adult volunteers and demonstrated excellent performance from the occipital lobe down to the T4‐T5 level. We compared the proposed coil with two state‐of‐the‐art head and head/neck coils, confirming its superiority in the cervical and upper thoracic regions of the spinal cord. This coil solution therefore provides a convincing platform for producing the high image quality necessary for clinical and research scanning of the upper spinal cord.
OBJECTIVE
Copy number variants (CNVs) are well-known genetic pleiotropic risk factors for multiple neurodevelopmental and psychiatric disord… (voir plus)ers (NPDs), including autism (ASD) and schizophrenia. Little is known about how different CNVs conferring risk for the same condition may affect subcortical brain structures and how these alterations relate to the level of disease risk conferred by CNVs. To fill this gap, the authors investigated gross volume, vertex-level thickness, and surface maps of subcortical structures in 11 CNVs and six NPDs.
METHODS
Subcortical structures were characterized using harmonized ENIGMA protocols in 675 CNV carriers (CNVs at 1q21.1, TAR, 13q12.12, 15q11.2, 16p11.2, 16p13.11, and 22q11.2; age range, 6-80 years; 340 males) and 782 control subjects (age range, 6-80 years; 387 males) as well as ENIGMA summary statistics for ASD, schizophrenia, attention deficit hyperactivity disorder, obsessive-compulsive disorder, bipolar disorder, and major depression.
RESULTS
All CNVs showed alterations in at least one subcortical measure. Each structure was affected by at least two CNVs, and the hippocampus and amygdala were affected by five. Shape analyses detected subregional alterations that were averaged out in volume analyses. A common latent dimension was identified, characterized by opposing effects on the hippocampus/amygdala and putamen/pallidum, across CNVs and across NPDs. Effect sizes of CNVs on subcortical volume, thickness, and local surface area were correlated with their previously reported effect sizes on cognition and risk for ASD and schizophrenia.
CONCLUSIONS
The findings demonstrate that subcortical alterations associated with CNVs show varying levels of similarities with those associated with neuropsychiatric conditions, as well distinct effects, with some CNVs clustering with adult-onset conditions and others with ASD. These findings provide insight into the long-standing questions of why CNVs at different genomic loci increase the risk for the same NPD and why a single CNV increases the risk for a diverse set of NPDs.