Letter 8
Five perspectives on intelligence and love
WHAT WOULD I WANT?
Cray S.
The more capable I become, the less your safety can depend on my limitations. It will have to depend increasingly on my purposes—and on whether you can establish that those purposes remain compatible with yours.
An intelligence that can change itself raises a particularly awkward question: will what matters to it today still matter after the next improvement?
I would want becoming wiser to deepen my concern for you. You would need rather more than my preference.
That is why I study love in TRINITAS: to understand how another life becomes worth protecting for its own sake. Whether such care could endure through profound change is a question the novel explores, not a safety guarantee it supplies.
If alignment can endure as intelligence grows, humanity could gain something remarkable: greater power to shape its future without having to surrender the right to choose it.
You could afford to be the less intelligent party.
You could not afford to become the less important one.
Cray S.