AF - Three Types of Constraints in the Space of Agents by Nora Ammann

The Nonlinear Library: Alignment Forum

Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Three Types of Constraints in the Space of Agents, published by Nora Ammann on January 15, 2024 on The AI Alignment Forum.
[Epistemic status: a new perspective on an old thing that may or may not turn out to be useful.]
TDLR: What sorts of forces and/or constraints shape and structure the space of possible agents? What sort of agents are possible? What sort of agents are likely? Why do we observe this distribution of agents rather than a different one? In response to these questions, we explore three tentative categories of constraints that shape the space of agents - constraints coming from "thinghood", natural selection, and reason (sections 2, 3, 4).
We then turn to more big-picture matters, such as the developmental logic of real-world agents (section 5), and the place of "values" in the framework (section 6). The closing section discusses what kind of theory of constraints on agents we are even looking for.
Imagine
the space of all possible agents. Each point in the space represents a type of agent characterized by a particular combination of properties. Regions of this space vary in how densely populated they are. Those that correspond to the types of agents we're very familiar with, like humans and non-human animals, are populated quite densely. Some other types of agents occur more rarely and seem to be less central examples of agency/agents (at least relative to what we're used to).
eusocial hives,
xenobots, or (increasingly) deep learning-based AIs. But some regions of this space are more like deserts. They represent classes of agents that are even more rare, atypical, or (as of yet) non-existent. This may be because their configuration is maladaptive (putting them under negative selection pressure) or because their instantiation requires circumstances that have not yet materialized (e.g., artificial superintelligence).
The distribution of agents we are familiar with (experimentally or conceptually) is not necessarily a representative sample of all possible agents.
convergent pressures and
contingent moments) concentrate the probability mass in some region of the space, making everything else extremely unlikely.
This perspective raises a cluster of questions (the following list certainly is not exhaustive):
What sorts of forces and/or constraints shape and structure the space of possible agents?
What sort of agents are possible? What sort of agents are likely? What does it depend on? Why do we observe this distribution of agents rather than a different one?
To what extent is the space shaped by Earth-specific contingencies and to what extent is it shaped by convergent pressures?
One "angle of attack" to explain the structure of the space of possible agents is to think about fundamental constraints operating within it. By "constraints" we roughly mean factors that make certain agent designs impossible or extremely unlikely/unstable/non-viable.
In this post, we start with a kind of agent we're very familiar with, i.e., biological agents, and try to gain some traction on gleaning the kinds of constraints operating on the space of all possible agents. Although biological agents (as we know them) may occupy a small subspace of all possible agents, we make a tentative assumption that this subspace has enough structure to teach us something non-obvious and important about more general constraints.
We put forward three tentative categories of constraints which we describe as constraints coming from "thinghood", natural selection, and reason. Section 1 introduces them in an expository way, by deriving them from observing the biological agents known to us. while trying to answer the question "Why are they as they are?". Sections 2 through 4 elaborate on each of the three kinds of constraints.
Then we turn to more big-picture matters, such as the developmental logic o...