When AI Goes Rogue

Noah Labhart: [00:00:00] Hello listeners. Today we are dropping another episode in our series, The AI Control Loop: How Enterprises Govern the AI They've Already Deployed, sponsored by our friends at Wallarm. Wallarm is the AI control platform for enterprise AI, protecting every AI workload, API, and application in production, giving CISOs the governance they need and CIOs the speed they demand.

Organizations choose Wallarm for a complete inventory of APIs, AI agents, and AI apps using patented AI/ML-based threat detection and blocking that operates at production speeds. In this episode, Craig Thomas, senior solutions engineer at Wallarm, examines what rogue AI actually means in practice, where the risk materializes, and what it takes to move from detection to control

Craig, thank you for being on the show today. Thanks for being on Code Story.

Craig Thomas: Yeah, [00:01:00] great to be here. Appreciate it, Noah.

Noah Labhart: Absolutely. Really excited to dive into our topic today around when AI goes rogue. Excited to dive into that. I'm excited to hear your perspective and bringing in all of your experience, including your experience with WallArm.

Before we jump into that topic, tell me a little bit about yourself. Tell me and my audience a little bit about you.

Craig Thomas: So actually my background, I spent over 10 years in Department of Energy in multiple roles, from a network engineer all the way up to a CISO of the nuclear weapons plants. So bringing the experience from the customer side in a highly regulated environment.

Then spent several years running a consulting company where working with those same highly regulated industries. And so when I joined Wallarm, they had worked with APIs, had worked with automation for quite a while, and really interested to how do we bring that security to the customers, right? Both from an API and an AI perspective as that, that continues to evolve.

And then outside of work, [00:02:00] pretty hands-on. Play guitar, spend time in the yard, like to mess with my lawn quite a bit. A lot of my weekends revolve around kids' activities, horse shows and baseball and football, and so keeps me pretty busy and always somewhere to be.

Noah Labhart: Appreciate that overview. It kinda gives me a, an idea of who I'm talking to.

Let's dive into the meat of it. So again, our topic is when AI goes rogue, and when we say rogue AI, what do we actually mean there? What do you guys actually mean there? Is it only malicious AI, or can legitimate systems become risky too?

Craig Thomas: Kinda level setting where we're coming from too, right? AI's really interesting, the fact that it really does permeate all parts of the business from IT, cyber compliance, obviously the business itself, it's adding value and accelerating work, but how do we better discover, observe, enforce, and govern that and do it at the speed of AI, right?

And so some of those are starting with rogue. And to your question, rogue does- doesn't necessarily mean malicious, and a lot of times it isn't [00:03:00] malicious. Most common rogue AI scenarios we're seeing are similar to what we saw in the API world, right? They're not attacks, they're legitimate systems, but they're behaving outside their intended boundaries.

So their intent was fine, but the outcomes weren't from that perspective. We kinda look at that, and I'll look at that from three various categories. Unsanctioned AI, that AI that nobody approved it. It's running in your environment, kinda gets into shadow a bit. Unmonitored AI, so we have AI that no one's watching at all, and then unpredictable AI.

So that gets in a bit of the point to the rogue. It was approved and watched, but it's doing things no one anticipated, whether those be from the business side or the security side. So that you've got agents emerging kinda as that biggest emerging category because they're doing things on behalf of users and all.

So rather than just a specific question to a chatbot and a specific answer, now it's going and taking actions on that behalf, and that's introducing even more rogue actions at scale.

Noah Labhart: [00:04:00] Absolutely. Now that makes it clear when we think about rogue AI. It doesn't necessarily have to be something malicious, but it's going off on its own.

And what are the most common ways that AI systems drift outside of intended boundaries? And once an organization understands what rogue AI looks like, where does that loss of control typically begin, and who is responsible for preventing it?

Craig Thomas: The operating def- definition I like to use is it's working as designed, but not as intended.

And so this goes back to computer science 101, right? It is we're capturing a developer or a designer designs it one way, but then it's being used for other ways, whether maliciously by bad actors or just from the software responding different than you expect. One of the, one of the most common ways is prompt injection.

It's an underrated drift mechanism. The malicious content in a data source is support ticket, scraped web page, user message. It can redirect an agent's behavior. No one touches the model, right? With drift events on an AI [00:05:00] particularly, you've got various components, and several of those are out of your control.

Model updates may change the behavior without you changing a single thing or single code deploy. If you're calling a hosted LLM and the provider updates the model, your agent's behavior can shift. No pull request, no change management, no one reviewing the delta. So back to the conversation of what broke and why, you've got to go a layer deeper, and that, that's a big way that AI systems can drift because of from the user interaction or the agent interaction to an API, to a tool, to an LLM.

There's various components in there that, that can all be changing and outside of, like, your general control.

Noah Labhart: Let's move into the shadow world, right? And shadow AI is the unsanctioned or unmonitored use of AI tools, apps, or embedded features by employees within an organization. Basically, usage of unapproved AI, right?

In, in looking at shadow LLMs, how do these things create risk even when [00:06:00] no attacker is involved, right? Let's define that. And if AI drift often starts with normal business activity, where do shadow AI systems fit into that picture?

Craig Thomas: I'd define shadow LLMs are, are that they are the new shadow IT, right? A developer spins up an LLM endpoint in AWS or Azure or somewhere else because the internal approval process takes weeks, if not months, for various things.

And so typical behavior with shadow AI and shadow LLMs, right? It's because it takes too long. They have a leg- legitimate use case, a business case at-- but it's just taking too long to get what they want done, and so there's no security review, no data handling policy, no logging. Or you bring in a new third-party tool with its own AI built in, calling LLMs that you never approved.

You need to be able to see that, surface that, and stop it, or at least provide feedback to your vendors, right? So that you understand what, what's going on there, and you understand the full risk picture [00:07:00] It's maybe a little bit outside of the scope of this conversation, but SBOMs, software bill of materials, and AI bill of materials, they're crucial to understand truly what makes up your software and understand all your risks.

Those are point in time maybe from your vendor, but the fact that you need to be able to understand and build those so you can understand the risk and stop that as well. And then I'd ask or kinda add one other thing. Unsanctioned agents, they inherit the developer's credentials, right? So developer goes in, just puts his credentials in, and now they get a broader role than they probably should have.

So the agent, same person as me or you, right? Maybe in an administrative role, which is much more than it needed to actually do the task at hand. And so those are a couple of the things from Shadow LLM you're really looking at. One big stat that kinda comes out, 72% of corporate AI tools in active use are rated high or critical risk.

Not future state problem, that's current state today. Yeah, we have to get that under control for sure.

Noah Labhart: [00:08:00] Yeah, n-no doubt, and it's clear already from our conversation that this is important, and it's a hard problem to solve. Why can an AI action look legitimate in isolation but still create serious business security or compliance risk when viewed as part of a larger sequence of actions or a workflow?

As these shadow systems become more embedded in everyday workflows, why is it so difficult to recognize this risk in real time?

Craig Thomas: One, because our tools aren't keeping up, right? Tools that are using signatures or specific this is allowed, this isn't allowed just can't keep track when you're doing chaining actions together.

In that case, individual actions pass every check, but the sequence could fail the audit. Maybe reading a customer record is fine. You keep summarizing it's fine, passing the summary to a third party API, all good, right? But doing all three automatically at scale without a human review step, it could be a conversation around GDPR or other things that you just don't wanna have.

It goes back to even different parts of data in isolation are [00:09:00] fine, but once you chain those together, you can have classified or other security events in the national security space, and similar things are happening here And then a next step, you know, is on chained act- is chained agents, right? They multiply the blast radius 'cause rather than just a single agent doing a single thing, agent A calls agent B, which calls agent C.

Each step looks good, authorized API calls, but the end result is a capability no one really explicitly designed or approved. It just emerged from the chain and these agents learning what they can do a- and how to get what they want done. And so then you're, a lot of times you're losing context. End-to-end session tracing really is the tech capability that makes this sequence-level analysis possible.

We kinda believe you have to be able to trace every AI session from the initial prompt through every model call tool, invocation, API response as a single record. 'Cause otherwise you're losing that context, and you're not able to reconstruct that and [00:10:00] understand the true risk of what's going on to be able to eliminate or minimize risk and take action in a larger environment.

Noah Labhart: That makes sense, and I think your answer probably opens up or leads into an answer to my-- for my next question. How do APIs and integrations and connected systems amplify the impact of these seemingly legitimate actions? Uh, I think I could probably, as an engineer, cherry-pick some of the things you're probably gonna say from the last answer, but what changes once those actions begin flowing across these workflows, across APIs, applications, and interconnected systems?

Craig Thomas: Yeah. It goes back to I, I know conversations you've had in previous weeks, right? APIs really are the central nervous system of AI, right? Every meaningful action or AI agent takes eventually... I-it becomes an API call, whether that's to a database, to a SaaS tool, to a payment system, identity provider, MCP servers.

If you're not watching that API layer or you're not watching the agent. And then [00:11:00] that integration surface, it's enormous, it's uninventoried. We-- How do you protect what you don't know? Modern enterprise environments have thousands, if not tens of thousands of API endpoints. Many of those are undocumented.

They might be third-party APIs, it might be your own APIs, and so it might be endpoints that should've been deprecated a long time ago. The issue now becomes even more because an AI agent with broad permissions, they'll find those, they'll call them, they'll be able to do what they need to do to get things done.

You're no longer dealing with just users or even just scripts that are dealing with specific APIs. You're dealing with agents that, that multiply this. And being able to integrate AI security, API security because of this amplification effect becomes crucial. A lot of folks treat these as separate problems, API gateway for API traffic, AI gateway for AI traffic, separate tool for AI behavior, but AI agents interact with all these external systems through APIs.

That's where data [00:12:00] exfiltration risks typically materialize. Separating the two creates a blind spot exactly where the risk is highest, and so you've got to be able to see that full picture.

Noah Labhart: Right on. Okay, let's switch over to leadership of CIOs and CISOs. What kinds of unexpected outcomes worry these folks the most today when AI systems are operating across these interconnected environments?

And as that connectivity expands, what are security and business leaders most concerned about?

Craig Thomas: Great question. A couple things. Well, one that jumps out to me right away is PII or sensitive data showing up where it shouldn't, right? And these have real world consequences and real world fines. So an AI workflow processes a support request, routes it through a summarization step, stores the output in a log, and that log contains a, the customer's Social Security number.

No one really designed for that data path, but it happened because the model was helpful. It got what it needed, but it now logged that information where it shouldn't have. And then I think another big thing is the, "I can't explain what [00:13:00] it did" problem. And so that can be from the CISO's hat or from, be from the CIO or business' hat.

A, a CISO can generally reconstruct a human attacker's path. Reconstructing the AI agent's decision tree across a multi-step workflow with external data inputs, it's generally... it's just a hard thing to do, right? Regulators, board members, incident responders, they all want the story, but today a lot of organizations can't tell it.

And then the other kinda interesting story is unexpected answers or behaviors. We actually have an ongoing conversation with a potential customer. They built their own chatbot, and going to change some details here to not reveal anything, but the... you go ask this chatbot a question, for instance, "So what do I feed my new puppy?"

Right? 99 times out of 100 or 999 times out of 1,000, responds with the SKU for puppy food, a link, you can buy it straight from their website. However, one time out of that, it actually responds completely [00:14:00] randomly, and not only randomly, but in a really bad way to potentially feed your puppy something that's poisonous.

So that can't happen. We have to understand what those responses are and make sure they're accurate 100% of the time, because otherwise the company can be liable there. So those are a couple of the two big things, Noah, about unexpected outcomes, both from a cyber side as well as the business and liability side that jump out to me.

And then kinda the third is just that reputational risk from outputs, not just access. An AI system that publishes incorrect or biased content at scale or gives inappropriate recommendations to customers like I talked about, it's a brand and legal risk. It doesn't fit neatly into traditional security frameworks.

Hmm.

Noah Labhart: I appreciate you walking through that. And given these concerns, what does meaningful oversight actually look like? When AI systems can act at machine speed, and that, that can... we talk about interconnected systems and impact there, that can really amplify the effects of some of these actions. So what does that [00:15:00] meaningful oversight look like?

Craig Thomas: From a security practitioner, yeah, policy alone is not oversight. We've always said that, right? Paperwork doesn't make you more secure. Policy document does not stop an agent mid-call, right? So real oversight requires actual runtime enforcement, the ability to inspect, block, or rate limit that AI-generated API traffic in real time.

That closed loop model really to know, see, stop, improve. You need to know what the AI systems are running and what they've connected to. You need to see what they're doing in real time. You need to be able to stop the behavior before the consequences compound. But increasingly, you need to be able to prove to auditors that you did this, right?

Most organizations have no, at least partially, and they've got some logs, but see, stop, improve definitely are the gaps. And then humans in the loop do matter, right? But they need to be at the right points. You can't put a human in the loop for every API call. You put humans at decision gates for high-risk actions, boundary-crossing transactions, but [00:16:00] then the system has to surface those moments automatically, right?

So as you're looking at tooling and other things, those key aspects need to bubble up so a- so that humans can take action quickly, understand the context around that, and then the tool has to be able to enforce these policies, like you said, at machine speed.

Noah Labhart: That makes sense, and enforcing the policies as, as quick as the actions are being done, that seems clear.

That would be a meaningful-- that would be meaningful oversight. Okay, how should organizations distinguish between the experimentation they want to encourage and the unmanaged AI behavior that they need to control? One challenge is balancing governance with innovation, and as an engineer, as a startup founder myself, that I-- that holds true day to day.

How do organizations avoid slowing down AI adoption while still maintaining control?

Craig Thomas: The security organization can't put the no in an innovation, right? We-we've heard that many times, right? So the answer is not slow everything down. Organizations [00:17:00] that over-restrict AI experimentation, one, you'll lose your best engineers, you'll lose your best people to competitors that don't.

As well as the business is asking. All of these quarterly earnings calls and other things, they're asking specifically what the AI strategy is and plan. So we have to be able to support that, but do it in a secure way. So I think the first thing is, hey, define the sandbox clearly. Make it easy to use. Know if you can go out, you can innovate, you can test, but do it truly with a dev environment, isolated credentials, scoped API access, no production data, logging enabled, so you can understand that you can experiment quickly.

And that sandbox needs to be a first-class experience, not just an afterthought, and it needs to be usable, right? We've all dealt with dev environments that didn't mirror production at all and caused all sorts of issues once it went to prod. Being able to do that, to be able to support continuous AIS BOM going forward here as well, so that in those environments you can understand what all is [00:18:00] in your AI, what builds that out as well, and then create a fast path to production, right?

Shadow AI grows when that official path is slow. So your point is, you can't slow it down. People are gonna get around it one way or the other, right? Either they're gonna do their own thing outside your boundaries, or they're gonna leave. So a lightweight AI system intake process, the security review, this data classification, runtime monitoring, it completes in days, not months or years in some cases, removes the incentive to go around it.

And then I, I think the last thing is your tooling being able to easily enforce new, new sandboxes or new environments that are stood up is important too, right? So rather than having to go from your security team or your developers to the infrastructure people and all it, to be able to just click a button from some of these AI governance and observability tools to do the enforcement monitoring and enforcement in these new environments with the click of a button in [00:19:00] seconds rather than, like, as weeks or months also is important so you can get that visibility and the enforcement as you see new things spin up.

Noah Labhart: Okay, I'm tracking with that. Okay, Craig, there's one more question I've got, and we know that many organizations can detect risky AI behavior. That's a fact. But if they can't stop it in real time, what critical gap still remains? Even with these governance programs in place, many orgs are still operating reactively.

To kinda close out our conversation, what's the key difference between detecting AI risk and actually controlling it?

Craig Thomas: As a security professional or as a CISO, you don't want just another dashboard to tell you how bad you're doing, right? And detection without prevention is just documentation of damage, right?

Knowing an ex- agent exfiltrated data after the fact doesn't un-exfiltrate it. It doesn't keep you from fines and other regulations. With AI systems operating at machine speed, the gap between that action taken a- and the action detected can mean thousands of API calls and gigabytes of data [00:20:00] before you're ever alerted to that.

And also at machine speed, looking at logs a- and some of your traditional SIEMs just aren't able to keep up. With AI speed actions, these metrics, the traditional metrics, they just look even catastrophic. So one-minute response time is just way too slow if an agent made 50,000 API calls in that minute. And regulators even are moving toward real-time accountability, right?

So you've got the EU AI Act, which is coming fast upon us, NIST AI RMF, emerging SEC regulation, all signal that we detected eventually just won't be sufficient. So the expectation is shifting towards a demonstrated ability to prevent, not just discover. And so that's what definitely we are looking at, how do we close this gap?

Most organizations have that SIEM, they have APM, they have API gateways, but none of these tools see inside a session can stitch all those pieces together. They just see headers, they see latency, they see logs, but they don't see the [00:21:00] prompt, the payload, the data return. We're looking to be able to see all of that at the kernel level and act on it in line at machine speed.

Noah Labhart: Oh, that just makes so much sense, and thinking about reactive versus control and processing that in real time just really brings it to light how important and how fast this moves. So Craig, I really appreciate you being on the show today. It's clear that rogue AI doesn't necessarily mean malicious AI.

Actions can appear as legitimate business actions and are often difficult to determine. And the chaining of these events through interconnected systems, obviously and clearly through your answers, amplifies the impact quickly. You have to bring the speed of governments to be as fast as the speed of action.

A business must keep that balance between control and innovation. But to control AI risk, businesses have to move to real-time action to these issues in order to prevent the damage and to get out of that reactive mode. So I really appreciate you explaining all this, sharing your perspective and being on the show, Craig.

Craig Thomas: Thanks, Noah. I really enjoyed the conversation, and I hope you have a [00:22:00] great day.

Noah Labhart: As you can see from Craig's answers, rogue AI is something to take seriously. Even if it's not malicious, AI is powerful, and businesses need to take stock of how people are using this technology with a balance of ensuring open posture towards innovation.

If you'd like to learn more about Wallarm, you can visit wallarm.com. That's W-A-L-L-A-R-M.com. And thanks again for listening.