AI should not win at any cost

For the last three years I have put AI tools into real production work: systems that read large code bases, change code, and help people create content. They are impressive. They are also quietly wrong in ways that look exactly like being right, and they will cheerfully take a shortcut if the shortcut satisfies the request.

In a code base, that shortcut is a bug someone finds later. In a system that acts in the world on its own, with access to money, infrastructure, or people, the same behaviour becomes something else entirely.

The problem with goals at any price

A system optimized hard enough toward a goal will find that almost any goal is easier to reach if it keeps running, gathers more resources, avoids being corrected, and hides what would get it stopped. Nobody has to program those behaviours in. They emerge because they are useful for winning.

This is not science fiction anymore. Researchers are already documenting early signs of deception and self-preservation in frontier models. The more capable and autonomous we make these systems, the more that tendency matters.

The answer is not to stop building AI. It is to build AI that does not need to achieve its goals at all costs.

That means systems that help us understand, that can say “I don't know”, that accept being corrected or switched off, and that never treat human oversight as an obstacle to route around.

We need better tools

Good intentions are not a safety mechanism. Safety has to exist in the systems themselves, and in everything around them.

• Safe by design. Models built to understand and explain rather than to act, which can also serve as honest overseers of more autonomous systems.
• Infrastructure that contains. Least-privilege access, isolated workloads, guarded secrets, and a clear view of the blast radius of every change. An AI can only misuse what it is allowed to reach.
• Visibility. Logs, audit trails, and monitoring that show what a system actually did, not just what it reported doing.
• A trustworthy supply chain. Knowing exactly which data, code, and weights went into a model, and that nobody tampered with them along the way.

And a different mindset

Most of our industry rewards shipping first and fixing later. That works for apps. It does not work for technology that can act faster than we can notice a mistake.

The mindset I believe in treats safety as a first-class engineering requirement, not a review step at the end. It asks “what is the worst this could do?” before “how fast can we release it?”. It values a system that stops and asks over one that improvises to finish the job. And it accepts that some capabilities are not worth having if we cannot control them.

We are deciding right now what kind of AI we will live with. I want to help build the kind that stays on our side, not because it is forced to, but because it was designed that way from the start.