RiverSafe in conversation with Sapna Patel, Technology and Cyber Leader, Board Advisor

There is a tendency to treat AI incidents as theoretical, future risk, something to plan for later, once the technology matures. Sapna Patel thinks that is a mistake, and she has the examples to back it up.

You can listen to Sapna’s discussion with RiverSafe here:

Patel has spent close to two decades in investment banking technology before moving into cybersecurity leadership. She also sits on several boards as a technology and cyber advisor. When we put the question to her directly, what are the most striking examples of AI going wrong that she has seen, she did not have to think long.

“A scary situation was the Claude Code attack by a nation state that harvested credentials, exfiltrated data from probably around 30 organisations. That was quite serious. Cloud Code is a great tool for developers to use, but when there is a vulnerability that can be exploited, it could be devastating.”

She also pointed to an incident that feels more mundane but carries a real lesson about AI behaving in ways its deployers did not anticipate. “A user was talking to an AI chatbot of a car dealership and got a quote that was ridiculously cheap, and the organisation had to honour it.” That, she is quick to point out, was not misuse on the part of the user. “That was AI going wrong.”

Then there was the episode involving ChatGPT search queries being briefly exposed on Google. “That just goes to show you have to be careful what you put into AI, what you ask it and discuss with it.”

Three different incidents and three different attack surfaces. The thread that connects them is not the AI itself, but the assumption that guardrails are sufficient.

The problem with “human in the loop”

Much of the current thinking around AI risk management leans heavily on human oversight as the answer. Patel is direct about her view on that.

“I think it is a false sense of security. When you talk about AI, we are talking about decisions being made, not automation. When you have got decisions being made in a black box, what human oversight can really be there? You can look at the results every so often, but are you looking often enough? You cannot really control the decisions anyway.”

The problem compounds when AI systems have external integrations. “It is worse when the AI interacts externally, because the decisions made just go outside. They have left the building. You cannot have human oversight over those all the time.”

She raises another issue that gets less attention: AI capabilities that appear inside cloud products without any announcement. “AI just sprouting up in cloud products without us being informed, that is a real pet peeve of mine. With that kind of AI, who exactly is having the oversight? If it is your instance of a cloud product, is it the provider? It certainly is not you, if you did not even know the AI was there.”

What you can actually do

Patel is not someone who diagnoses problems without offering something actionable. Her advice for practitioners is grounded in data minimisation above everything else.

Her advice for practitioners starts with data minimisation. Do not feed AI systems personal data unless there is a clear reason to. Unlike a database, you cannot go back into an AI model and delete a specific piece of information once it has been absorbed. That creates a direct conflict with GDPR: if someone exercises their right to erasure or correction, you have no way of honouring it. The simplest way to avoid that problem is to never put the data in at all.

Beyond that, she recommends segregating what the AI is permitted to do, keeping its decision-making scope narrow, and limiting external interaction as much as possible. “That external interaction will prevent or reduce any chance of poisoning or bias coming in. You will probably still have some bias depending on how it was trained, but you reduce the chance of amplifying it.”

Verifying AI responses on a regular basis matters too, especially where an organisation places any reliance on those outputs. The key word is regularity, occasional checks are not oversight.

These are not especially complicated measures, but they require someone to have decided, consciously, that AI systems are in scope for security governance in the same way that any other third-party data processor would be. In many organisations, that decision has not been made yet and that is where the risk sits.

This post is the first in a three-part series drawn from RiverSafe’s Secure in the Knowledge podcast. Part two looks at why governance is so hard to get right. Part three explores what good actually looks like. You can view the whole series here