Visibility into AI Agents
Alan Chan
Carson Ezell
Max Kaufmann
Kevin Wei
Lewis Hammond
Herbie Bradley
Emma Bluemke
Nitarshan Rajkumar
Noam Kolt
Lennart Heim
Markus Anderljung
Increased delegation of commercial, scientific, governmental, and personal activities to AI agents—systems capable of pursuing complex goa… (voir plus)ls with limited supervision—may exacerbate existing societal risks and introduce new risks. Understanding and mitigating these risks involves critically evaluating existing governance structures, revising and adapting these structures where needed, and ensuring accountability of key stakeholders. Information about where, why, how, and by whom certain AI agents are used, which we refer to as visibility, is critical to these objectives. In this paper, we assess three categories of measures to increase visibility into AI agents: agent identifiers, real-time monitoring, and activity logging. For each, we outline potential implementations that vary in intrusiveness and informativeness. We analyze how the measures apply across a spectrum of centralized through decentralized deployment contexts, accounting for various actors in the supply chain including hardware and software service providers. Finally, we discuss the implications of our measures for privacy and concentration of power. Further work into understanding the measures and mitigating their negative impacts can help to build a foundation for the governance of AI agents.
Connectome-based reservoir computing with the conn2res toolbox
Laura E. Suárez
Agoston Mihalik
Filip Milisav
Kenji Marshall
Mingze Li
Petra E. Vértes
Bratislav Mišić
RapidBrachyTG43: A Geant4‐based TG‐43 parameter and dose calculation module for brachytherapy dosimetry
Jonathan Kalinowski
Transnational conservation to anticipate future plant shifts in Europe
Yohann Chauvier-Mendes
Peter H. Verburg
Dirk N. Karger
Loïc Pellissier
Sébastien Lavergne
Niklaus E. Zimmermann
Wilfried Thuiller
Transnational conservation to anticipate future plant shifts in Europe
Yohann Chauvier-Mendes
Peter H. Verburg
Dirk N. Karger
Loïc Pellissier
Sébastien Lavergne
Niklaus E. Zimmermann
Wilfried Thuiller
Transnational conservation to anticipate future plant shifts in Europe
Yohann Chauvier-Mendes
Peter H. Verburg
Dirk N. Karger
Loïc Pellissier
Sébastien Lavergne
Niklaus E. Zimmermann
Wilfried Thuiller
Gaining Biological Insights through Supervised Data Visualization
Jake S. Rhodes
Adrien Aumon
Sacha Morin
Marc Girard
Catherine Larochelle
Boaz Lahav
Elsa Brunet-Ratnasingham
Amélie Pagliuzza
Lorie Marchitto
Wei Zhang
Adele Cutler
F. Grand'Maison
Anhong Zhou
Andrés Finzi
Nicolas Chomont
Daniel E. Kaufmann
Stephanie Zandee
Alexandre Prat
Kevin R. Moon
Dimensionality reduction-based data visualization is pivotal in comprehending complex biological data. The most common methods, such as PHAT… (voir plus)E, t-SNE, and UMAP, are unsupervised and therefore reflect the dominant structure in the data, which may be independent of expert-provided labels. Here we introduce a supervised data visualization method called RF-PHATE, which integrates expert knowledge for further exploration of the data. RF-PHATE leverages random forests to capture intricate featurelabel relationships. Extracting information from the forest, RF-PHATE generates low-dimensional visualizations that highlight relevant data relationships while disregarding extraneous features. This approach scales to large datasets and applies to classification and regression. We illustrate RF-PHATE’s prowess through three case studies. In a multiple sclerosis study using longitudinal clinical and imaging data, RF-PHATE unveils a sub-group of patients with non-benign relapsingremitting Multiple Sclerosis, demonstrating its aptitude for time-series data. In the context of Raman spectral data, RF-PHATE effectively showcases the impact of antioxidants on diesel exhaust-exposed lung cells, highlighting its proficiency in noisy environments. Furthermore, RF-PHATE aligns established geometric structures with COVID-19 patient outcomes, enriching interpretability in a hierarchical manner. RF-PHATE bridges expert insights and visualizations, promising knowledge generation. Its adaptability, scalability, and noise tolerance underscore its potential for widespread adoption.
Gaining Biological Insights through Supervised Data Visualization
Jake S. Rhodes
Adrien Aumon
Sacha Morin
Marc Girard
Catherine Larochelle
Elsa Brunet-Ratnasingham
Amélie Pagliuzza
Lorie Marchitto
Wei Zhang
Adele Cutler
F. Grand'Maison
Anhong Zhou
Andrés Finzi
Nicolas Chomont
Daniel E. Kaufmann
Stephanie Zandee
Alexandre Prat
Kevin R. Moon
Dimensionality reduction-based data visualization is pivotal in comprehending complex biological data. The most common methods, such as PHAT… (voir plus)E, t-SNE, and UMAP, are unsupervised and therefore reflect the dominant structure in the data, which may be independent of expert-provided labels. Here we introduce a supervised data visualization method called RF-PHATE, which integrates expert knowledge for further exploration of the data. RF-PHATE leverages random forests to capture intricate featurelabel relationships. Extracting information from the forest, RF-PHATE generates low-dimensional visualizations that highlight relevant data relationships while disregarding extraneous features. This approach scales to large datasets and applies to classification and regression. We illustrate RF-PHATE’s prowess through three case studies. In a multiple sclerosis study using longitudinal clinical and imaging data, RF-PHATE unveils a sub-group of patients with non-benign relapsingremitting Multiple Sclerosis, demonstrating its aptitude for time-series data. In the context of Raman spectral data, RF-PHATE effectively showcases the impact of antioxidants on diesel exhaust-exposed lung cells, highlighting its proficiency in noisy environments. Furthermore, RF-PHATE aligns established geometric structures with COVID-19 patient outcomes, enriching interpretability in a hierarchical manner. RF-PHATE bridges expert insights and visualizations, promising knowledge generation. Its adaptability, scalability, and noise tolerance underscore its potential for widespread adoption.
Cross-Task Affinity Learning for Multitask Dense Scene Predictions
Dimitrios Sinodinos
EMA-Net: Efficient Multitask Affinity Learning for Dense Scene Predictions
Dimitrios Sinodinos
PhotoBot: Reference-Guided Interactive Photography via Natural Language
Oliver Limoyo
Jimmy Li
Dmitriy Rivkin
Jonathan Kelly
We introduce PhotoBot, a framework for fully automated photo acquisition based on an interplay between high-level human language guidance an… (voir plus)d a robot photographer. We propose to communicate photography suggestions to the user via reference images that are selected from a curated gallery. We leverage a visual language model (VLM) and an object detector to characterize the reference images via textual descriptions and then use a large language model (LLM) to retrieve relevant reference images based on a user’s language query through text-based reasoning. To correspond the reference image and the observed scene, we exploit pretrained features from a vision transformer capable of capturing semantic similarity across marked appearance variations. Using these features, we compute suggested pose adjustments for an RGB-D camera by solving a perspective-n-point (PnP) problem. We demonstrate our approach using a manipulator equipped with a wrist camera. Our user studies show that photos taken by PhotoBot are often more aesthetically pleasing than those taken by users themselves, as measured by human feedback. We also show that PhotoBot can generalize to other reference sources such as paintings.
PhotoBot: Reference-Guided Interactive Photography via Natural Language
Oliver Limoyo
Jimmy Li
Dmitriy Rivkin
Jonathan Kelly
We introduce PhotoBot, a framework for fully automated photo acquisition based on an interplay between high-level human language guidance an… (voir plus)d a robot photographer. We propose to communicate photography suggestions to the user via reference images that are selected from a curated gallery. We leverage a visual language model (VLM) and an object detector to characterize the reference images via textual descriptions and then use a large language model (LLM) to retrieve relevant reference images based on a user’s language query through text-based reasoning. To correspond the reference image and the observed scene, we exploit pretrained features from a vision transformer capable of capturing semantic similarity across marked appearance variations. Using these features, we compute suggested pose adjustments for an RGB-D camera by solving a perspective-n-point (PnP) problem. We demonstrate our approach using a manipulator equipped with a wrist camera. Our user studies show that photos taken by PhotoBot are often more aesthetically pleasing than those taken by users themselves, as measured by human feedback. We also show that PhotoBot can generalize to other reference sources such as paintings.