The heads of the big AI labs publicly warn against their own models and call for slowing down. Weeks later, one release follows the next anyway. In episode 10 of Prompt & Proper, Steffen Ochsenreither from Swiss Post and I ask what sits behind this fear campaign. Our answer: the risk lies less in superintelligent models than in agents we have given too many rights. The episode is in German, and this post captures the core of the conversation.
Warning or Marketing?#
I read the warnings as a wonderful marketing gig. Every time the stock market trembles, a statement appears about how dangerous all of this is and how it needs regulating. The models really are getting better, and AI is here to stay. But the infrastructure investments are enormous, and at some point this investment bubble will burst.
Steffen sees it the same way. The models develop as fast as the economic incentives dictate. Every CEO of a frontier lab warns, every one of them has the most dangerous model, and at the same time they release at a pace that does not fit the warning at all.
Containment Is the Operator’s Job#
How does a model break out in the first place? It runs in a virtual machine that has some access somewhere. It gets a hard task, and it is trained to solve that task whatever it takes. That it finds a way out along the way does not surprise me.
Concluding from this that AI is dangerous is, in my view, the wrong conclusion. A nuclear plant operator does not tell you the fuel rods keep trying to escape. The operator builds the containment. That is exactly what I expect from the AI companies.
“My demand is that these AI companies finally show some backbone and take responsibility for their systems.”
In practice that means: secure the system, detect early when something happens, and have a kill switch. You can also deploy further agents that watch and shut down a runaway agent automatically.
Steffen corrected the picture of the evil model. An agent breaking out of its sandbox does not think “Hugging Face belongs to me today.” It looks for a solution to the problem it was given, and it never gets tired. With enough resources it finds gaps. Most AI incidents come from good agents getting creative inside a frame nobody defined properly. Which brings us to the boring topic of governance: which rights does an agent get, and how far can I run an autonomous system safely?
Start at Home: The Agent Inventory#
Pointing at the frontier labs is convenient. Steffen’s counterpoint is that every company has to start with itself. If you do not know how many agents run in your organisation and what they do, you have no grounds to complain.
That is why Swiss Post keeps an inventory of its agents. Steffen says openly that it is not fully transparent yet and that they are at the beginning. The questions the inventory has to answer are clear, though: who owns this agent, which permissions does it have, how far may it act autonomously?
One advantage over the past: with today’s tools you can get transparency in. That is a massive difference from the days when everyone had an Excel sheet with macros on their laptop. Steffen sees the real problem elsewhere. As of today, our structures are not built for this autonomy.
I go one step further: every agent needs its own identity. Steffen and I use an agent at Zühlke ourselves that helps prepare this podcast. We send it ideas, it collects them, we decide, and it researches the facts. It has an identity, an email address and access to a GitHub repository. To be honest, its first proposals were really bad. It takes a large part of the work off our hands, but the last 20 percent, the part that matters, we still think through ourselves.
Human in the Loop Under Load#
Human in the loop sounds good. Steffen asked the question that is often missing: how much can a human still be in the loop? Ten decisions a day work. At 30 it gets critical. At 100 the human no longer decides, they just press yes.
I recently saw exactly that at a bank that has agents in use. At some point people only click okay, because the agent’s answer is almost always right. The bank reacted:
“They deliberately build in errors so that the human in the loop stays alert.”
I find that fascinating, but it raises a bigger question. If the agent is almost always right, maybe you could take the human out entirely. Even a trained person who knows what they are doing makes mistakes.
The Error Rate Depends on the Use Case#
Steffen believes human in the loop will be redefined. The human shifts towards setting the rules within which the agent decides.
For a thousand decisions a day that barely touch the core business, he accepts a higher error rate. When it is about a large amount of money or about people, such as whether someone stays employed, a very different error rate applies, and at the end a human has to decide.
His example is customs clearance at Swiss Post. The AI supports the declarant in deciding about a good, and its hit rate is very high. The decision and the liability still stay with the declarant. Legally, it could not be done any other way today. Steffen sees a positive side effect: the customs staff gain confidence because their decision is backed up. Either way, you have to monitor the system and define responsibilities cleanly.
Lead Agents Like Employees#
This is where I challenged Steffen. I am not talking about the background agents a coding tool spins up for you. I mean an agent with an identity, one I hold a conversation with and that has a clear task. That agent is like an employee.
In the 1980s you had an assistant or a secretary who wrote letters, made copies and organised travel. Then came the personal computer, we took that work over ourselves, and the secretariats shrank. Now I see the next shift, above all in my own work. I first had to go through unlearning and relearning. You have to lead these agents like employees: make the goal clear and be able to state it clearly.
The accountability stays with me, no question. But I delegate, like every manager does. So why should an agent at the bank or in customs not be allowed to make the decision? Employees make wrong decisions too.
Steffen made a good observation here. To err is human, and we accept that up to a point. From AI we expect 100 percent. He traces that back to our historical relationship with the computer: one or zero, right or wrong, nothing in between.
AI Amplifies What Is There#
The models are getting better at reading context. Tell your travel agent to book a flight to France and today it will not book a departure from São Paulo, because it knows from its memory that I probably fly from Zurich. Still, Steffen holds his point:
“AI only ever produces what is already there. If the foundation is bad, it does not get better unless I work on it.”
I see it the same way. If your processes or your organisation are weak, AI amplifies exactly that. So you have to review your processes and adapt them, and in doing so decide which responsibility you can hand to an agent. Legally we are not there yet. As a company you still need to think about it soon, because that is where the efficiency gain lies. A chat window on top of the existing process does not deliver it.
Steffen’s Take-Away#
Regulation will always lag behind the technology, Steffen says. That does not mean you should wait to define your own first rules. His three points at the end:
- Governance is boring and not a sexy topic, but without governance you run into problems.
- AI itself is not the problem. The autonomy that agents bring is the real challenge, because our companies are not prepared for it.
- Maybe the models are not that superintelligent after all.
“Maybe we were, and still are, simply too careless in handing out permissions.”
If you enjoy the episode, subscribe to Prompt & Proper on Spotify, Apple Podcasts, Amazon Music or YouTube and tell us which AI topic you are wrestling with right now. All episodes: promptandproper.ai.