Ian Arawjo

Kenneth Li

Jonathan K. Kummerfeld

Elena L. Glassman

2024-05-11

Proceedings of the CHI Conference on Human Factors in Computing Systems (publié)

ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing

Chelse Swoopes

Priyan Vaithilingam

Martin Wattenberg

Elena L. Glassman

Evaluating outputs of large language models (LLMs) is challenging, requiring making -- and making sense of -- many responses. Yet tools that… (voir plus) go beyond basic prompting tend to require knowledge of programming APIs, focus on narrow domains, or are closed-source. We present ChainForge, an open-source visual toolkit for prompt engineering and on-demand hypothesis testing of text generation LLMs. ChainForge provides a graphical interface for comparison of responses across models and prompt variations. Our system was designed to support three tasks: model selection, prompt template design, and hypothesis testing (e.g., auditing). We released ChainForge early in its development and iterated on its design with academics and online users. Through in-lab and interview studies, we find that a range of people could use ChainForge to investigate hypotheses that matter to them, including in real-world settings. We identify three modes of prompt engineering and LLM hypothesis testing: opportunistic exploration, limited evaluation, and iterative refinement.

2024-05-11

Proceedings of the CHI Conference on Human Factors in Computing Systems (publié)

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

Shreya Shankar

J.D. Zamfirescu-Pereira

Bjorn Hartmann

Aditya G Parameswaran

2024-04-18

ArXiv (prépublication)

Antagonistic AI

Alice Cai

Elena L. Glassman

2024-02-12

ArXiv (prépublication)

Imagining a Future of Designing with AI: Dynamic Grounding, Constructive Negotiation, and Sustainable Motivation

Priyan Vaithilingam

Elena L. Glassman

2024-02-12

ArXiv (prépublication)

ChainForge: An open-source visual programming environment for prompt engineering

Priyan Vaithilingam

Martin Wattenberg

Elena Glassman

2023-10-29

Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (publié)

Notational Programming for Notebook Environments: A Case Study with Quantum Circuits

Anthony DeArmas

Michael Roberts

Shrutarshi Basu

Tapan Parikh

We articulate a vision for computer programming that includes pen-based computing, a paradigm we term notational programming. Notational pro… (voir plus)gramming blurs contexts: certain typewritten variables can be referenced in handwritten notation and vice-versa. To illustrate this paradigm, we developed an extension, Notate, to computational notebooks which allows users to open drawing canvases within lines of code. As a case study, we explore quantum programming and designed a notation, Qaw, that extends quantum circuit notation with abstraction features, such as variable-sized wire bundles and recursion. Results from a usability study with novices suggest that users find our core interaction of implicit cross-context references intuitive, but suggests further improvements to debugging infrastructure, interface design, and recognition rates. Throughout, we discuss questions raised by the notational paradigm, including a shift from ‘recognition’ of notations to ‘reconfiguration’ of practices and values around programming, and from ‘sketching’ to writing and drawing, or what we call ‘notating.’

2022-10-28

ACM Symposium on User Interface Software and Technology (publié)

To Write Code: The Cultural Fabrication of Programming Notation and Practice

Writing and its means have become detached. Unlike written and drawn practices developed prior to the 20th century, notation for programming… (voir plus) computers developed in concert and conflict with discretizing infrastructure such as the shift-key typewriter and data processing pipelines. In this paper, I recall the emergence of high-level notation for representing computation. I show how the earliest inventors of programming notations borrowed from various written cultural practices, some of which came into conflict with the constraints of digitizing machines, most prominently the typewriter. As such, I trace how practices of "writing code" were fabricated along social, cultural, and material lines at the time of their emergence. By juxtaposing early visions with the modern status quo, I question long-standing terminology, dichotomies, and epistemological tendencies in the field of computer programming. Finally, I argue that translation work is a fundamental property of the practice of writing code by advancing an intercultural lens on programming practice rooted in history.

2020-04-21

International Conference on Human Factors in Computing Systems (publié)

To Write Code: The Cultural Fabrication of Programming Notation and Practice

2020-04-21

International Conference on Human Factors in Computing Systems (publié)