"«Boundaries», Part 1: a key missing concept from utility theory" by Andrew Critch

LessWrong (Curated & Popular)

https://www.lesswrong.com/posts/8oMF8Lv5jiGaQSFvo/boundaries-part-1-a-key-missing-concept-from-utility-theory

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

This is Part 1 of my «Boundaries» Sequence on LessWrong.

Summary: «Boundaries» are a missing concept from the axioms of game theory and bargaining theory, which might help pin-down certain features of multi-agent rationality (this post), and have broader implications for effective altruism discourse and x-risk (future posts).

1. Boundaries (of living systems)

Epistemic status: me describing what I mean.

With the exception of some relatively recent and isolated pockets of research on embedded agency (e.g., Orseau & Ring, 2012; Garrabrant & Demsky, 2018), most attempts at formal descriptions of living rational agents — especially utility-theoretic descriptions — are missing the idea that living systems require and maintain boundaries.

When I say boundary, I don't just mean an arbitrary constraint or social norm. I mean something that could also be called a membrane in a generalized sense, i.e., a layer of stuff-of-some-kind that physically or cognitively separates a living system from its environment, that 'carves reality at the joints' in a way that isn't an entirely subjective judgement of the living system itself. Here are some examples that I hope will convey my meaning:

فصل ها

1. "«Boundaries», Part 1: a key missing concept from utility theory" by Andrew Critch (00:00:00)

2. 1. Boundaries (of living systems) (00:00:45)

3. Figure 1: Cell membranes, skin, fences, group divisions, and state borders as living system boundaries. (00:02:07)

4. Text Resumes (00:02:53)

5. 2. Canonical disagreement points as missing from utility theory and game theory (00:05:34)

6. Figure 2: Folk Theorem (00:06:39)

7. Figure 3: Nash bargaining solution (00:08:38)

8. Figure 4: Kalai-Smordinsky bargaining solution (00:09:30)

9. Text Resumes (00:10:13)

10. Figure 5: Illustration of BATNAs delimiting a zone of potential agreement. (00:11:02)

11. Text Resumes (00:12:07)

12. Image: Mathematical formula (00:12:39)

13. Figure 6: People disagreeing and going home. (source: owned) (00:14:10)

14. 3. Boundaries as informing zero-point selection during bargaining (00:14:42)

15. 4. Some really important boundaries (00:15:56)

16. Figure 7: The Eastern Front in WWII. (00:16:24)

17. Text Resumes (00:16:53)

18. 5. Summary (00:17:40)