AI Knows Everything From Nowhere
In the last post about why imitation isn’t quite enough in consulting, we established, almost as a passing remark, that to pass the Turing test now, an AI would need to be made less capable, not more. Maybe that idea is worth dwelling on a bit longer. Made less capable. That is, more limited in what it can do? Why would limitation be what makes a system pass as a person? And what would that tell us about what a person is?
Let’s say you start exchanging messages with someone who is either a state-of-the-art language model or a random human. How do you know who you are in correspondence with? I know what my first instinct would be: I’d ask it trivia questions. Not to see what it knew, but whether it knew too much. The capital of Assyria. Got that one? Fine. Then I’d go to Wikipedia for harder ones. What is the atomic number of tungsten? The year Krakatoa erupted? The currency of Bhutan? Oh, you knew those too?
The giveaway is that the LLM is not positioned. It has no biography, no preference. Every available piece of information is equally available. In this regard, it is like an automatic typewriter that takes whatever has been written so far, and proceeds with the most likely continuation. The text has no author, but more often than not, it ends up containing the right answer.
Human cognition, on the other hand, is positioned. Very concretely so.
Think of the room you’re in. Maybe there’s a table, a cup, a chair, all within your reach. Outside, a bright morning, or an evening drawing in. A solid floor under your feet. All of this is present and available to you in a way an item behind the wall or on some narrow alley in the neighboring town isn’t.
To you, these things are here, while most things aren’t. Why is that? Had you walked somewhere else, into some other room, you’d have something else at hand. The cup and the chair would not be a part of your world right now.
From your perspective, I’d say it is your biography up until this point that decides what is there.
For an AI to pass the trivia test, it would need something similar. Leave out experience, keep the structure: a biography that decides what is available to it right now. Not a training run, from which the model emerges with full, equal access to every data point there is, but a curriculum, a history it has actually moved through. Something is happening to it now, and the rest it can recall from its past. And then, of course, there are the things it has never encountered. Who was the Doge of Venice in 1655? No idea, I have spent the last cycles reading a software engineering book. But I can google it, of course.
This is not about constructing an inferior AI, but about it having a position. To know something is to know it from somewhere, and from somewhere most things are out of view. A point of view, if we want AI to have one, lives in that contrast.
Trivia answering capability aside, there will probably be markets for more than one kind of system. A great deal of what we need is best done with no point of view at all. Translate this contract, pull the dates out of it, turn this specification into working code, summarize these thousand reviews. Here you want the same input to yield the same output, uncolored by whatever the system happened to read last. The lack of a position is the whole point. You want the right answer and nothing else, and the automatic typewriter, more often than not, gives you exactly that.
ChatGPT arrived in 2022, and the models and tools that followed have grown steadily more fluent and productive. But one thing has not changed: the models themselves are as stateless as ever. They remember nothing from one exchange to the next. Whatever of a weeks-long correspondence still fits in the context window is handed back to the model on every turn, and it reads it fresh, as if meeting it for the first time. And for the work we just described, that is exactly right.
There is the other kind of work, though. The kind we are building Rethink for. Think of a lawyer halfway through a case. Her work is not to know everything. It is to know this case better than anyone in the room. To hold what she has seen of it, to weigh this document against that one, to notice when something does not fit the story so far. She does not read a transcript of the entire engagement every morning. Her experience is biographical. This is the kind of work where a point of view stops being a curiosity and becomes the job.