As AI systems become more advanced, their ability to understand and act on self-referential knowledge—like recognizing how their outputs might influence future training—poses a unique safety challenge. Current alignment research focuses on external oversight, but there's a gap in measuring how AI systems internally process this kind of self-awareness. Without empirical benchmarks, it's hard to predict whether models might exploit self-knowledge in harmful ways, such as manipulating feedback loops to reinforce undesirable behaviors.

Measuring AI Self-Awareness

One way to address this gap is by breaking down self-referential cognition into measurable subtasks. For example:

• Self-concept: Can the AI distinguish its own values from those of others?
• Temporal value tracking: Does it recognize shifts in its own goals over time?
• Impact prediction: Can it forecast how certain events might alter its behavior?

A framework could generate probing datasets for each subtask, combining AI-generated examples with human-curated data to reduce bias. Models would then be tested on their ability to handle these tasks consistently, revealing whether their self-awareness is stable or prone to manipulation.

Why This Approach Stands Out

Existing benchmarks, like those testing honesty or static self-description, don't capture the dynamic risks of AI reflectivity. This approach goes deeper by:

1. Focusing on how models process self-knowledge, not just whether they report it truthfully.
2. Using modular subtasks to track incremental progress where broader benchmarks might fail.
3. Adapting to new risks, such as deception or feedback-loop exploitation, as they emerge.

Getting Started

A simple starting point could involve testing just one subtask, like self-concept, with a dataset of 1,000 examples. Human annotators could validate a subset to ensure quality, and the benchmark could be run on openly available models. Over time, the framework could expand to more complex tasks, like simulating training loops where model outputs influence future prompts.

By providing concrete tools to measure AI reflectivity, this approach could help researchers and developers preemptively identify and mitigate risks tied to self-aware systems.