Arkil Patel

Doctorat - McGill

Superviseur⋅e principal⋅e

Dzmitry Bahdanau

Co-supervisor

Siva Reddy

Sujets de recherche

Apprentissage profond

Traitement du langage naturel

Site web

Google Scholar

GitHub

Publications

Universal Adversarial Triggers Are Not Universal

Nicholas Meade

Arkil Patel

Siva Reddy

2024-04-24

ArXiv (prépublication)

doi.org

arxiv.org

Evaluating In-Context Learning of Libraries for Code Generation

Arkil Patel

Siva Reddy

Dzmitry Bahdanau

Pradeep Dasigi

Contemporary Large Language Models (LLMs) exhibit a high degree of code generation and comprehension capability. A particularly promising ar… (voir plus)ea is their ability to interpret code modules from unfamiliar libraries for solving user-instructed tasks. Recent work has shown that large proprietary LLMs can learn novel library usage in-context from demonstrations. These results raise several open questions: whether demonstrations of library usage is required, whether smaller (and more open) models also possess such capabilities, etc. In this work, we take a broader approach by systematically evaluating a diverse array of LLMs across three scenarios reflecting varying levels of domain specialization to understand their abilities and limitations in generating code based on libraries defined in-context. Our results show that even smaller open-source LLMs like Llama-2 and StarCoder demonstrate an adept understanding of novel code libraries based on specification presented in-context. Our findings further reveal that LLMs exhibit a surprisingly high proficiency in learning novel library modules even when provided with just natural language descriptions or raw code implementations of the functions, which are often cheaper to obtain than demonstrations. Overall, our results pave the way for harnessing LLMs in more adaptable and dynamic coding environments.

2024-01-01

North American Chapter of the Association for Computational Linguistics (publié)

doi.org

MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations

Arkil Patel

Satwik Bhattamishra

Siva Reddy

Dzmitry Bahdanau

2023-01-01

EMNLP (publié)

doi.org

openreview.net