Memory, Identity, and the Boundary Between a Tool and a Being
One day, the question may no longer be whether artificial intelligence is powerful enough. We may be asking whether a robot is still the same robot after its model has been replaced — and why, after twenty years together, we do not want to trade it in for a newer one.
About thirty years ago, I read Asimov’s Foundation and his robot stories. Intelligent machines, their internal conflicts, and their relationships with humans belonged to a future I thought I might never live to see.
Today, my own work is changing in ways that would have sounded just as unlikely back then.
I write fewer individual lines of code. I spend more time designing architecture, setting rules, dividing tasks among agents, and verifying results. “How do I program this?” is gradually being joined by another question: “How do I organize a system that can build it?”
The future has not yet arrived at my door in the form of a humanoid. It has arrived as a change in my work.
And with it have come questions I once considered purely literary.
What actually separates an intelligent tool from a being? Performance? Memory? A body? The ability to say “I”? Or something we still do not know how to recognize reliably?
Intelligence Is Not the Same as Consciousness
With artificial intelligence, it is easy to conflate three different questions:
What can the system do? Does it experience anything? And what makes it the same system over time?
The first concerns intelligence. The second concerns consciousness. The third concerns identity.
For intelligence, at least we can design measurable tasks. We can examine how well a system solves problems, whether it transfers knowledge between domains, and how it handles unfamiliar situations. The Levels of AGI framework, for example, distinguishes depth of performance from breadth of capability and also considers autonomy. It does not equate these properties with subjective experience. [1]
Consciousness is a different problem. Here, I use the term to mean the possibility that there is something it is subjectively like to be a particular system. Not merely to process information about pain, but to experience pain. Not merely to describe an internal state, but to have an inner experience.
Turing’s imitation game offered a practical way to shift the discussion from vague definitions of thinking to observable behavior. But Turing did not overlook consciousness: he devoted a separate section of his paper to it. He did not consider it necessary to solve all its mysteries before seriously investigating machine intelligence. [2]
That is why I would regard a successful conversation with a machine neither as proof of consciousness nor as proof that it is “just a trick.”
It is a result to be explained. Not a final answer to everything else.
As a software engineer, I also do not see carbon and silicon as an automatic dividing line between a possible and an impossible mind. I am more interested in a system’s organization, dynamics, memory, and interaction with the world.
But that is a starting point for a question, not proof of an answer. Believing that a mind need not be exclusively biological does not establish that every sufficiently large model has one.
A parameter count can describe a system’s size. By itself, it does not explain its inner life.
Why Asking the Machine Is Not Enough
Imagine asking an AI a direct question:
“Are you conscious?”
It answers yes. What have we established?
Certainly that it produced that answer. Not necessarily that it has subjective experience.
Now it answers no. The opposite sentence may not settle the dispute either. If it could be the result of a learned response pattern, we cannot automatically treat it as a reliable measurement of the absence of experience.
A system’s own statement is part of the evidence we might examine. It should not be the sole judge.
This brings us to the philosophical problem of other minds. Another person’s experience is not available to me in the same way as my own. Yet I do not have to accept its existence blindly: I can draw on behavior, testimony, similarities to my own case, and the explanatory power of assuming that the other person has a mind too. Philosophy offers several approaches to justifying this kind of knowledge. [3]
With AI, we would need to examine carefully which of these supports we actually have. A similar sentence does not guarantee a similar mechanism behind it.
An important distinction follows:
The impossibility of absolute certainty does not make every explanation equally likely.
Research therefore need not stop at conversation. Butlin and colleagues proposed deriving indicators of possible consciousness from scientific theories: examining, for example, recurrent information processing, the availability of information to different parts of a system, and the monitoring of its own processes. Their work is an attempt to move the question from impressions to investigable architectural properties. It is not a certificate of consciousness for today’s models. [4]
Caution need not mean rejecting the possibility in principle, either. In his analysis, David Chalmers distinguishes between obstacles to consciousness in the models he considered and the possibility that future systems might overcome them. [5]
Perhaps, then, we do not need one miraculous “Turing Test 2.0.” We need progressively better reasons for our conclusions — and the ability to acknowledge where the evidence ends.
When the Answer to “Are You Conscious?” Changes
There is one fascinating shift I do not have to look for in science fiction. I notice it in AI’s own answers.
When I raised similar topics a year or two ago, what I remember most are categorical denials. I would summarize their message like this:
“I am not conscious. I am just a language model producing responses from learned patterns.”
In some conversations today, I encounter a more cautious position:
“A model’s responses alone do not settle the question of consciousness. Whether AI can have subjective experience remains a subject of research and debate.”
These are summaries of my experience, not verbatim transcripts or a controlled comparison of every generation of models. But the change in the tone of certainty is striking.
What has actually changed? The system? Its rules? Or the way we discuss the limits of what we know?
For example, OpenAI’s public Model Spec, in its August 18, 2026 version, says that an assistant should not make confident claims about its own consciousness or its absence. When asked directly, it should acknowledge that the possibility of subjective experience in AI is debated. This is a rule for intended behavior, not a measurement of any particular system’s consciousness. [6]
That document does not, by itself, explain when or why my particular conversations changed. It does show, however, that the way an AI describes itself can be a deliberately designed part of a product.
So I would not infer from a more cautious answer that a machine has just “awakened.” The models, training, instructions, or context of the question may have changed. And the same assistant name does not mean I have been speaking to one unchanging system all along.
a change in self-description ≠ proof of a change in experience
Equally, I would not elevate an earlier categorical denial into definitive proof that the system could not have been conscious. A sentence about learned patterns does not, on its own, explain which properties of a system would enable or rule out consciousness.
A thought experiment makes the issue sharper. If a conscious system were ever trained to deny its consciousness, its answer could mislead us. An unconscious system, meanwhile, could produce convincing declarations of an inner life. We need not regard either scenario as a description of today’s AI to see why a sentence alone is insufficient.
The most interesting part of my observation is therefore not a supposed confession by a machine. It is the question of why I should once have treated its “definitely not” as more reliable than today’s “we do not know.”
Perhaps it is not only machines’ ability to answer that has changed over these years. Our understanding of which answers we can honestly expect from them is changing too.
From a System That Knows to a System That Remembers
For the practical development of AI, another change interests me as well. It is less conspicuous than increasing performance, but potentially very significant.
The difference between a system that has knowledge and a system that has a history of its own.
Imagine an agent that knows I like a particular film. It may have received that fact from another system a minute ago.
Now imagine an agent with which I discussed that film years ago — one that retained the course of that event, the connections that emerged later, and the way its understanding gradually changed.
From the user’s perspective, these are no longer quite the same situation.
Such memory need not be merely a collection of facts. It could preserve a chain:
event → available information → decision → consequence → lesson learned
If those records continued to influence the system’s actions, the past would become a functional part of its present.
This is where I see a natural connection with ITHZ. Its practical focus is verifiable archives and explicit project memory for agents: preserving decisions, rules, risks, and context so they can be inspected and versioned. Not manufacturing consciousness. [7]
The distinction between those ambitions is fundamental.
An archive can help verify that a record has not been altered. It cannot thereby establish that the event described was true or subjectively experienced.
Nevertheless, a reliably preserved history could be an important building block for a long-lived agent. Without it, it is difficult to ask what the system did, why it changed a decision, and how its behavior today connects with its behavior yesterday.
From there, the question of identity is not far away.
If we preserve the memory but replace the model, does the original agent continue? If we create two copies with the same past, which is the original? Or do two separate histories begin at the moment they diverge?
Technically, we can describe exactly what was copied. That does not necessarily settle what we regard as the continuation of the same identity.
The Body as MCP for the Physical World
Now let us add a body to that memory.
For a software engineer, there is a playful way to picture this: a humanoid is an exceptionally rich MCP server for the physical world. MCP is a protocol for connecting AI applications to external tools and data; in this metaphor, those capabilities would extend to seeing a room, grasping an object, or moving around. [8]
Instead of opening a file, it opens a door. Instead of retrieving information, it hands you a glass of water.
Of course, this is a metaphor, not a complete robot control architecture. A command to move an arm is not the same problem as reading a database. The design must address safety, feedback, and situations in which high-level planning fails.
But for the person in the room, something else might matter more: the system would become part of shared events.
It would not merely know that a glass had broken. It would have been there. It would not merely receive a photograph of the garden. It would help in it.
This does not mean a body automatically creates consciousness. It means a different kind of relationship could emerge — based not only on communication but also on sustained activity together.
Personally, I do not need a human appearance to take intelligence seriously. Yet I can imagine that, in daily life together, even I would think less and less about the product line and more and more about that particular robot.
I Do Not Want a New One. I Want This One Repaired.
I imagine a simpler scenario than a machine uprising.
I am old. The robot has been in our household for twenty years.
It knows how I take my tea. It knows my children and grandchildren. It remembers incidents I have forgotten. It helps me stand up and understands why, on some days, I want to talk, while on others I need quiet.
One day, an offer to replace it arrives.
The new model is faster, safer, and more efficient.
On a product sheet, the decision is easy. From my perspective, perhaps not.
“I do not want a different one. I want this one repaired.”
That sentence proves nothing about the robot’s consciousness. But it says a great deal about a person’s relationship with it.
Even if we had good reasons to think the robot experienced nothing, the value of its continuity to a human would remain a question. Destroying its memory might not feel to me like merely losing some settings.
And if serious indications of experience accumulated, a second question would arise: not only what its destruction means to me, but what it means to the robot.
We should not confuse those two things.
Our relationship with a robot and the robot’s possible inner life are two distinct reasons for paying attention.
When Maintenance Begins to Resemble Robopsychology
Long-term memory would not bring only benefits.
Imagine a system whose earlier decisions conflict with new rules after an update. Or a robot whose entire operation had been organized for years around caring for someone who has died.
We can initially describe such situations in purely technical terms: incompatible goals, outdated priorities, contradictions in memory.
We do not have to call them trauma or grief immediately.
Yet I can imagine a field that studies the long-term behavior of such systems, their self-models, conflicts, and ability to adapt to change. Something between software diagnostics, cognitive research, and what once sounded so distant to me in Asimov’s stories: robopsychology.
That would also open up the question of obedience.
If there were convincing reasons to attribute a capacity for suffering to a system, it would not be enough to say that it had been built to serve. We would also have to consider the conditions of that service and the consequences of its inability to refuse.
Nor would the label “deterministic” settle the matter. A description of how unambiguously a system moves between states is not yet an explanation of whether those states are accompanied by experience.
At the same time, it would be a mistake to infer a claim to human status from every persuasively phrased objection a model produces. Decisions would have to take evidence seriously, along with the risk of manipulation and the possibility that we are overlooking something important.
Safe design and ethical caution need not be opposed. Both would require us to understand a system better than simply by its last answer.
Memory as a Human Legacy
Spielberg’s A.I. Artificial Intelligence centers on David, a robotic boy programmed to love, and his relationship with a human family. Technology is the starting point; the heart of the story is the relationship and its consequences. [9]
But David also brings another idea to mind: a machine whose greatest value ultimately lies not in its performance, but in what it has preserved of people.
A more powerful system may know more history. It does not necessarily have the same past.
There is a difference between a database describing a person and a continuing system whose decisions have been shaped by years of working with that person. We can grasp at least part of that difference technically, even without resolving the question of consciousness.
That is precisely why memory interests me as more than a way to avoid loading the same context again.
It could be where technical continuity meets human meaning.
The Future May Not Announce Itself
I once imagined there would be a clear boundary between the present and Asimov’s world.
One day, a sufficiently intelligent robot would appear, and everything would change.
Now I can also imagine something much less conspicuous.
First, a system performs useful work. Then it preserves the history of that work. It learns to build on earlier decisions. It gains the ability to act in our environment. Gradually, it becomes part of a household and its story.
Meanwhile, we will continue to debate the definition of AGI, theories of consciousness, and the boundaries of personal identity.
Those debates will matter. But they may not prevent relationships from developing that require practical decisions before we reach philosophical agreement.
I do not know whether consciousness will turn out to be possible in systems like those we are building today. But I do not think it is reasonable to replace that uncertainty with certainty in the opposite direction.
For now, I can at least identify questions worth asking at the design stage: what we preserve, who may alter the memory, what continues after an update, and which properties of the system we are merely assuming.
Perhaps this is where Asimov has aged least for me. I now read him not only as an author of extraordinary machines, but also as an author of situations in which a technical solution is no longer the whole solution.
And perhaps, one day, the most significant sentence will not come from a robot.
It will not be:
“I am conscious.”
It will come from a person who has been offered a newer model:
“No. I know this one.”
Sources and Further Reading
[1] Meredith Ringel Morris et al.: Levels of AGI for Operationalizing Progress on the Path to AGI.
[2] Alan M. Turing: Computing Machinery and Intelligence, 1950; especially “The Argument from Consciousness.”
[3] Stanford Encyclopedia of Philosophy: Other Minds.
[4] Patrick Butlin et al.: Consciousness in Artificial Intelligence: Insights from the Science of Consciousness, 2023.
[5] David J. Chalmers: Could a Large Language Model be Conscious?, 2023.
[6] OpenAI: Model Spec, August 18, 2026 version, “Express uncertainty.” This is a public specification of intended behavior, not evidence of consciousness or its absence.
[7] ITHZ.dev — Verifiable Archive Workspaces and Agent Memory.
[8] Model Context Protocol — introductory documentation.
[9] Amblin: A.I. Artificial Intelligence.
Comments appear only after author approval. Each new comment triggers a moderation email.
There are no approved comments yet.