Autonomy Eval
tl;dr
Open evaluation environments for testing whether human autonomy preservation in AI systems is primarily a model-behavior problem, an interface-design problem, or a systems problem requiring both.
Previous work studying human autonomy and agency in AI systems approaches it as either a model-behavior problem or an interface-design problem shaped by a person's pattern of AI use.
HumanAgencyBench, for instance, tests whether models behave in an autonomy-preserving manner by asking clarifying questions and deferring important decisions. The skill-formation study frames autonomy as an interaction-pattern problem and shows that different patterns of AI use lead to different learning outcomes.
My hypothesis is that autonomy preservation in AI systems requires both. The goals of this project are to:
- Determine the extent to which autonomy preservation is a model-behavior problem, an interface-design problem, or both.
- Identify the specific model behaviors and interface-design approaches that promote autonomy preservation.
To achieve this, I plan to build interactive environments in which users work with AI systems on a variety of creative tasks. These environments will measure autonomy preservation while controlling model behavior, interface design, and combinations of the two.
My goal is to find the simplest possible gaming and creative environments that enable these measurements, then use what they reveal to infer how autonomy-preservation studies should scale into increasingly complex, real-world knowledge-work environments.
Motivation
I was inspired to work on this project through my own projects related to AI-native games and creative interfaces.
The specific research questions emerged from my recent project, WonderLoom: a multimodal canvas designed to help children develop narrative thinking, literacy, self-expression, and creative agency through storytelling.
The idea of using gaming environments to measure and study these questions also came from my previous work on social deduction agents and AI-native games, where I developed agents that excel at Mafia and gaming environments in which people can play against those agents. (Read Mafia: On the Design of Social Deduction Agents and AI-Native Games)
Project Outputs
- Open-source interactive evaluation environment(s).
- Open-source datasets and post-trained models for testing autonomy preservation as a function of model behavior.
- Project research report(s).
Theory of Impact
The aim of this work is to reduce human-autonomy x-risk, particularly gradual disempowerment, by providing simple environments for studying and measuring autonomy-preservation methods, as well as mechanisms for scaling those studies into increasingly complex knowledge-work environments. We hope the long-term result is a reduction in this risk as AI diffuses throughout the global economy.
Funding Asks
Because the project is still exploratory, funding will primarily support the technical experimentation needed to identify a workable evaluation design. This includes GPU compute for post-training and testing open-source models, frontier-model API costs, hosting and infrastructure for the interactive evaluation environments, and development time for building, testing, collecting data from, and iterating on those environments.
Minimum funding: $5,000
This would be sufficient to build and release an initial version of the project: at least one open-source evaluation environment, baseline experiments using prompted and open-source models, an initial dataset of interaction trajectories, and a public report documenting the methodology, results, and limitations.
Ideal funding: $10,000
This would support more iterations and broader experimentation. It would provide additional compute for post-training, enable comparisons across more models and intervention designs, and support more extensive testing of both model-level and interface-level approaches. It would also create room to improve the environments based on early results.