NLP in the era of generative AI, cognitive sciences, and societal transformation
Join us at Mila in October for a three-day workshop to explore the transformative potential of language technologies and their implications for society.
This program is designed to provide decision-makers, policymakers and professional working in policy with a foundational understanding of AI technology.
We use cookies to analyze the browsing and usage of our website and to personalize your experience. You can disable these technologies at any time, but this may limit certain functionalities of the site. Read our Privacy Policy for more information.
Setting cookies
You can enable and disable the types of cookies you wish to accept. However certain choices you make could affect the services offered on our sites (e.g. suggestions, personalised ads, etc.).
Essential cookies
These cookies are necessary for the operation of the site and cannot be deactivated. (Still active)
Analytics cookies
Do you accept the use of cookies to measure the audience of our sites?
Multimedia Player
Do you accept the use of cookies to display and allow you to watch the video content hosted by our partners (YouTube, etc.)?
Publications
Smart Futures Based Resource Trading and Coalition Formation for Real-Time Mobile Data Processing
Collaboration among mobile devices (MDs) is becoming more important, as it could augment computing capacity at the network edge through peer… (see more)-to-peer service provisioning, and directly enhance real-time computational performance in smart Internet-of-Things applications. As an important aspect of collaboration mechanism, conventional resource trading (RT) among MDs relies on an onsite interaction process, i.e., price negotiation between service providers and requesters, which, however, inevitably incurs excessive latency and degrades RT efficiency. To overcome this challenge, this article adopts the concept of futures contract (FC) used in financial market, and proposes a smart futures for low latency RT. This new technique enables MDs to form trading coalitions and negotiate multilateral forward contracts applied to a collaboration term in the future. To maximize the benefits of self-interested MDs, the negotiation process of FC is modelled as a coalition formation game comprised of three components executed in an iterative manner, i.e., futures resource allocation, revenue sharing and payment allocation, and distributed decision-making of individual MD. Additionally, a FC enforcement scheme is implemented to efficiently manage the onsite resource sharing via recording resource balances of different task-types and MDs. Simulation results prove the superiority of smart futures in RT latency reduction and trading fairness provisioning.
2021-02-18
IEEE Transactions on Services Computing (published)
Collaboration among mobile devices (MDs) is becoming more important, as it could augment computing capacity at the network edge through peer… (see more)-to-peer service provisioning, and directly enhance real-time computational performance in smart Internet-of-Things applications. As an important aspect of collaboration mechanism, conventional resource trading (RT) among MDs relies on an onsite interaction process, i.e., price negotiation between service providers and requesters, which, however, inevitably incurs excessive latency and degrades RT efficiency. To overcome this challenge, this article adopts the concept of futures contract (FC) used in financial market, and proposes a smart futures for low latency RT. This new technique enables MDs to form trading coalitions and negotiate multilateral forward contracts applied to a collaboration term in the future. To maximize the benefits of self-interested MDs, the negotiation process of FC is modelled as a coalition formation game comprised of three components executed in an iterative manner, i.e., futures resource allocation, revenue sharing and payment allocation, and distributed decision-making of individual MD. Additionally, a FC enforcement scheme is implemented to efficiently manage the onsite resource sharing via recording resource balances of different task-types and MDs. Simulation results prove the superiority of smart futures in RT latency reduction and trading fairness provisioning.
2021-02-18
IEEE Transactions on Services Computing (published)
Nested dropout is a variant of dropout operation that is able to order network parameters or features based on the pre-defined importance du… (see more)ring training. It has been explored for: I. Constructing nested nets Cui et al. 2020, Cui et al. 2021: the nested nets are neural networks whose architectures can be adjusted instantly during testing time, e.g., based on computational constraints. The nested dropout implicitly ranks the network parameters, generating a set of sub-networks such that any smaller sub-network forms the basis of a larger one. II. Learning ordered representation Rippel et al. 2014: the nested dropout applied to the latent representation of a generative model (e.g., auto-encoder) ranks the features, enforcing explicit order of the dense representation over dimensions. However, the dropout rate is fixed as a hyper-parameter during the whole training process. For nested nets, when network parameters are removed, the performance decays in a human-specified trajectory rather than in a trajectory learned from data. For generative models, the importance of features is specified as a constant vector, restraining the flexibility of representation learning. To address the problem, we focus on the probabilistic counterpart of the nested dropout. We propose a variational nested dropout (VND) operation that draws samples of multi-dimensional ordered masks at a low cost, providing useful gradients to the parameters of nested dropout. Based on this approach, we design a Bayesian nested neural network that learns the order knowledge of the parameter distributions. We further exploit the VND under different generative models for learning ordered latent distributions. In experiments, we show that the proposed approach outperforms the nested network in terms of accuracy, calibration, and out-of-domain detection in classification tasks. It also outperforms the related generative models on data generation tasks.
Correction to: The patient advisor, an organizational resource as a lever for an enhanced oncology patient experience (PAROLEonco): a longitudinal multiple case study protocol
We study the learning performance of gradient descent when the empirical risk is weakly convex, namely, the smallest negative eigenvalue of … (see more)the empirical risk's Hessian is bounded in magnitude. By showing that this eigenvalue can control the stability of gradient descent, generalisation error bounds are proven that hold under a wider range of step sizes compared to previous work. Out of sample guarantees are then achieved by decomposing the test error into generalisation, optimisation and approximation errors, each of which can be bounded and traded off with respect to algorithmic parameters, sample size and magnitude of this eigenvalue. In the case of a two layer neural network, we demonstrate that the empirical risk can satisfy a notion of local weak convexity, specifically, the Hessian's smallest eigenvalue during training can be controlled by the normalisation of the layers, i.e., network scaling. This allows test error guarantees to then be achieved when the population risk minimiser satisfies a complexity assumption. By trading off the network complexity and scaling, insights are gained into the implicit bias of neural network scaling, which are further supported by experimental findings.
The human brain differs from that of other primates, but the genetic basis of these differences remains unclear. We investigated the evoluti… (see more)onary pressures acting on almost all human protein-coding genes (N = 11,667; 1:1 orthologs in primates) based on their divergence from those of early hominins, such as Neanderthals, and non-human primates. We confirm that genes encoding brain-related proteins are among the most strongly conserved protein-coding genes in the human genome. Combining our evolutionary pressure metrics for the protein-coding genome with recent data sets, we found that this conservation applied to genes functionally associated with the synapse and expressed in brain structures such as the prefrontal cortex and the cerebellum. Conversely, several genes presenting signatures commonly associated with positive selection appear as causing brain diseases or conditions, such as micro/macrocephaly, Joubert syndrome, dyslexia, and autism. Among those, a number of DNA damage response genes associated with microcephaly in humans such as BRCA1, NHEJ1, TOP3A, and RNF168 show strong signs of positive selection and might have played a role in human brain size expansion during primate evolution. We also showed that cerebellum granule neurons express a set of genes also presenting signatures of positive selection and that may have contributed to the emergence of fine motor skills and social cognition in humans. This resource is available online and can be used to estimate evolutionary constraints acting on a set of genes and to explore their relative contributions to human traits.
Denoising Score Matching with Annealed Langevin Sampling (DSM-ALS) has recently found success in generative modeling. The approach works by … (see more)first training a neural network to estimate the score of a distribution, and then using Langevin dynamics to sample from the data distribution assumed by the score network. Despite the convincing visual quality of samples, this method appears to perform worse than Generative Adversarial Networks (GANs) under the Fréchet Inception Distance, a standard metric for generative models. We show that this apparent gap vanishes when denoising the final Langevin samples using the score network.
In addition, we propose two improvements to DSM-ALS: 1) Consistent Annealed Sampling as a more stable alternative to Annealed Langevin Sampling, and 2) a hybrid training formulation, composed of both Denoising Score Matching and adversarial objectives. By combining these two techniques and exploring different network architectures, we elevate score matching methods and obtain results competitive with state-of-the-art image generation on CIFAR-10.