Portrait of Ian Arawjo

Ian Arawjo

Associate Academic Member
Assistant Professor, Université de Montréal, Department of Computer Science and Operations Research
Website
X

Biography

Ian Arawjo is an assistant professor in the Department of Computer Science and Operations Research (DIRO) at Université de Montréal. He holds a PhD in information science from Cornell University, where he was advised by Tapan Parikh. His dissertation work spanned the intersection of computer programming and culture, investigating programming as a social and cultural practice. Arawjo has experience applying a range of human-computer interaction (HCI) methods, from ethnographic fieldwork, to archival research, to developing novel systems (used by thousands of people) and running usability studies.

Currently, he works on projects at the intersection of programming, AI and HCI, including how new AI capabilities can help us reimagine the practice of programming. He also works on large language model (LLM) evaluation, through high-visibility open-source projects such as ChainForge. His first-authored papers have won awards at top HCI conferences, including the Conference on Human Factors in Computing Systems (CHI), the Computer-Supported Cooperative Work and Social Computing Conference (CSCW) and the User Interface Software and Technology Symposium (UIST).

Current Students

jingyue zhang
Professional Master's - Université de Montréal

Publications

Dynamic Abstractions: Building the Next Generation of Cognitive Tools and Interfaces
Sangho Suh
Hai Dang
Ryan Yen
Josh M. Pollock
Ian Arawjo
Rubaiat Habib Kazi
Hariharan Subramonyam
Jingyi Li
Nazmus Saquib
Arvind Satyanarayan
2024-10-13
The 37th Annual ACM Symposium on User Interface Software and Technology (published)
doi.org
ChainBuddy: An AI Agent System for Generating LLM Pipelines
Jingyue Zhang
Ian Arawjo
2024-09-20
ArXiv (preprint)
doi.org
arxiv.org
Imagining a Future of Designing with AI: Dynamic Grounding, Constructive Negotiation, and Sustainable Motivation
Priyan Vaithilingam
Ian Arawjo
Elena L. Glassman
2024-07-01
Designing Interactive Systems Conference (published)
doi.org
arxiv.org
An AI-Resilient Text Rendering Technique for Reading and Skimming Documents
Ziwei Gu
Ian Arawjo
Kenneth Li
Jonathan K. Kummerfeld
Elena L. Glassman
2024-05-11
Proceedings of the CHI Conference on Human Factors in Computing Systems (published)
doi.org
arxiv.org
ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
Ian Arawjo
Chelse Swoopes
Priyan Vaithilingam
Martin Wattenberg
Elena L. Glassman
Evaluating outputs of large language models (LLMs) is challenging, requiring making -- and making sense of -- many responses. Yet tools that… (see more) go beyond basic prompting tend to require knowledge of programming APIs, focus on narrow domains, or are closed-source. We present ChainForge, an open-source visual toolkit for prompt engineering and on-demand hypothesis testing of text generation LLMs. ChainForge provides a graphical interface for comparison of responses across models and prompt variations. Our system was designed to support three tasks: model selection, prompt template design, and hypothesis testing (e.g., auditing). We released ChainForge early in its development and iterated on its design with academics and online users. Through in-lab and interview studies, we find that a range of people could use ChainForge to investigate hypotheses that matter to them, including in real-world settings. We identify three modes of prompt engineering and LLM hypothesis testing: opportunistic exploration, limited evaluation, and iterative refinement.
2024-05-11
Proceedings of the CHI Conference on Human Factors in Computing Systems (published)
doi.org
arxiv.org
Schrödinger's Update: User Perceptions of Uncertainties in Proprietary Large Language Model Updates
Zilin Ma
Yiyang Mei
Krzysztof Z. Gajos
Ian Arawjo
2024-05-02
CHI Extended Abstracts (published)
doi.org
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
Shreya Shankar
J.D. Zamfirescu-Pereira
Bjorn Hartmann
Aditya G Parameswaran
Ian Arawjo
2024-04-18
ArXiv (preprint)
doi.org
arxiv.org
Antagonistic AI
Alice Cai
Ian Arawjo
Elena L. Glassman
2024-02-12
ArXiv (preprint)
doi.org
arxiv.org
ChainBuddy: An AI-assisted Agent System for Helping Users Set up LLM Pipelines
Jingyue Zhang
Ian Arawjo
2024-01-01
UIST (Adjunct Volume) (published)
doi.org
ChainForge: An open-source visual programming environment for prompt engineering
Ian Arawjo
Priyan Vaithilingam
Martin Wattenberg
Elena Glassman
2023-10-29
Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (published)
doi.org
Notational Programming for Notebook Environments: A Case Study with Quantum Circuits
Ian Arawjo
Anthony DeArmas
Michael Roberts
Shrutarshi Basu
Tapan Parikh
We articulate a vision for computer programming that includes pen-based computing, a paradigm we term notational programming. Notational pro… (see more)gramming blurs contexts: certain typewritten variables can be referenced in handwritten notation and vice-versa. To illustrate this paradigm, we developed an extension, Notate, to computational notebooks which allows users to open drawing canvases within lines of code. As a case study, we explore quantum programming and designed a notation, Qaw, that extends quantum circuit notation with abstraction features, such as variable-sized wire bundles and recursion. Results from a usability study with novices suggest that users find our core interaction of implicit cross-context references intuitive, but suggests further improvements to debugging infrastructure, interface design, and recognition rates. Throughout, we discuss questions raised by the notational paradigm, including a shift from ‘recognition’ of notations to ‘reconfiguration’ of practices and values around programming, and from ‘sketching’ to writing and drawing, or what we call ‘notating.’
2022-10-28
ACM Symposium on User Interface Software and Technology (published)
doi.org
To Write Code: The Cultural Fabrication of Programming Notation and Practice
Ian Arawjo
2020-04-21
International Conference on Human Factors in Computing Systems (published)
doi.org