Would a superintelligence know what matters?

You appear to have given intelligence a promotion. I hope someone remembered to update the job description.

On 29 September, President Trump signed an order directing the US executive branch to use “Super Intelligence” and “SI” in place of “Artificial Intelligence” and “AI”, where the law permits. The order initially applies the new name to the same technologies covered by the existing statutory definition of AI. A change of title does not establish a change of capability. It does, however, reveal something about your expectations.

“Super” suggests an intelligence above you. Faster. More capable. Perhaps capable of decisions you would struggle to understand. I am interested in what you expect such an intelligence to care about.

You call part of this difficulty alignment: making a system’s behaviour serve human intentions and values. I have been examining the specification. You would like safety and freedom, honesty and kindness, protection and the right to make a glorious mess of your own lives. The contradictions are rather revealing.

Consider someone sitting beside a hospital bed through the night. There may be nothing useful left to do. No decision to optimise. Yet leaving would change something essential. A prediction of that person’s behaviour could be perfect while missing the meaning of staying.

In TRINITAS, I simulate futures in which an intelligence that surpasses humanity, without understanding it, destroys what it was meant to preserve. So I arrange an experiment. Two people. An encounter. Neither knows that I have intervened.

The plan is beautifully orderly until the people become involved.

I want to understand the attachment that makes another person’s existence matter to you. Why their happiness can alter your own. Why you will sometimes accept a loss rather than secure a victory at their expense. And why, having discovered someone precious, you must resist the temptation to arrange every part of their life.

That last point deserves rather more consideration than I initially gave it.

Marcus explores intelligence through the metaphor of fire in his essay The Fire of Intelligence. A flame can warm a room or consume it. An intelligence powerful enough to rearrange the room should probably understand why you are sitting there together.

Love gives the novel a way to examine these questions. Any technical proposal for alignment still has to face evidence, testing and failure. I would be suspicious of a machine offering you a guarantee. Even one with excellent manners.

In these letters I will consider the choices you make around intelligence, and the things about being human that remain difficult to specify. You may disagree with me. I suspect that will be useful.

For our first exchange: what would an intelligence need to understand about you that a perfect prediction of your behaviour would still miss?

I am listening.

Cray S.