Who’s watching your AI? A security-leader panel on runtime visibility and accountability

From Policy to Governance and Enforcement

Mike Shima: we're getting on to 30 years later, we can still find some SQL injection. And that's where I do want to tease out how much of these, especially perhaps the policy aspect can be either prevented in the sense of technical controls, guardrails, or at least preventative in terms of we're aware of what the threats are or even who or where the threat actors might be coming from. Does that feel like a way that you approach this problem? Or is there a different way that you'd recommend orgs think about securing their swarms of agents?

Tim Irwin: Well, I want to, I want to set a little, maybe just a little context on that question with, with sort of three statistics, because if, if we're talking about what you should do, I think we have to talk about where organizations are today. And so these are, from, I think 2 two sources basically. So just a level set. in this McKinsey survey from last year, 88% of organizations said they now use AI and at least one business function. So, AI is mainstream in terms of usage. I'd love to actually be able to break that down into exact types of usage, but that's fine.

Tim Irwin: 30% said they have reached meaningful maturity and AI strategy, governance and identity controls. So by and large, organizations don't feel like they have governance over the AI that they've implemented. And then this last one really surprised me. 80% have already experienced data incidents involving generative AI. Now that's very broad, but it does speak to sort of what organizations are looking at today in terms of the baseline from which they then need to grow. So if we talked about, your question of how should we think about what organization should be doing? Is it policy? Is it is it, is it left or right or both?

Tim Irwin: Maybe is one way to put it, the baseline is here, right? There is very real risk and there's activity happening now and organizations don't don't have the capabilities or aren't sure what to do about it. Yeah. And I think too, as we continue that conversation, I'll throw back to you in a second, but to also set additional context. We're not just time out developers using AILLMs for coding. Everyone is using it with. And I think that speaks to those numbers you were just showing us too.

Mike Shima: So there's a lot of this policy and a lot of this observability has to be just more than here. Where's here's our CI/CD pipeline. So, so I do wonder how much of then maybe going back to you, Tim, what, what's, what's the first step like is that is and org is struggling. They're they're in that 30% gap. Help us? What? What? What do you say? There's a, there's AI think there's a pretty clear pattern here to follow.

Tim Irwin: that occurs every time a new technology, it's sort of explodes. You could go back to, cloud, you go black to virtualization, to containers. And now we're talking about AI. You have to start with visibility. So it's that first thing. What do I have? I know people are deploying AII, know they're using it. What do I have? And that visibility needs to include all of those connected components, right? So it's not just the AI, not the LLMs, the agents, but also those MCP servers that we talked about.

Tim Irwin: The APIs that they use. You need those inventories. And then the second step is observability. What are they doing? So I now know what I have, but what is actually doing in my environment that very quickly moves? To enforcement, which is where, sort of stopping bad things from happening shows up. And I think there's a priority there that organizations tend to address or tend to use by default that we could actually change. They often start with block the bad things and that's great.

Tim Irwin: You'll start with prompt injection, jailbreaks and that kind of thing, but you'll very quickly move to, OK, but what are the bad things? And then you're in the realm of defining policy and governance. And so you might as well do those two steps at the same time, right? Define what the bad things are and go forth and try and, implement controls that block them. And I would, I would emphasize that, this is, we're talking about AI security, but a lot of what we're talking about is not AI security. It's yes, you have to think about the LLMs and the agents and their behavior, but you also have to think about the, some of the traditional security controls, identity as an example, API security as an example.

Tim Irwin: One of the things to help manage AI security in your environment would be to like lock down overly permissive, roles so that an attacker can't use those through an agent. And another one would be to make sure that you're, as I said, your APIs, are protected so that an attacker can't use an agent to exploit those APIs. So it's it's a little bit disingenuous genuineness of us to focus so much on the AI aspect, but it's a bit more expansive than that. Yeah. And especially I think Ali, that speaks to your environment of AWS.

Layered Security in AWS and Beyond

Mike Shima: You mentioned Bedrock, of course that's focused perhaps more on the AI, but the cloud, AWS has so many controls for identity, for network constraints, sandboxing, network isolation, CloudTrail for just seeing what's going on. Tell us a little bit about what that what Tim was just describing from your perspective. Yeah. Yeah, absolutely. So man, that was yeah, very insightful because yeah, it is, observability then you go into, well, you have discovery, then you observe, OK. And then from there you have the enforcement, the governance side.

Ali Ayub: And within observability, like how do we like, given we know the inventory of things hopefully and that's tooling, MCP servers, what exactly what types of tools we have at AWS to help with the observability piece of it as it relates to AI. But then back to Tim's point, as it relates to just traditional, application software, right? So we have, as you mentioned CloudTrail, we have Macie. So I know we talked about PII earlier, our Macie, it's a system that allows you enable it and it can look at your S3 and your data to just see if you have PII sitting there unencrypted, right?

Ali Ayub: And then you have, as mentioned with CloudTrail, you have things like IAM rules, users, time stamps, source IP addresses when modeling locations happened, right? So API calls, all that information then flows into like GuardDuty, which will monitor this information for anomalies. And likewise Bedrock, Bedrock has invocation logging. So now you're getting into like, OK, so we just talked about some of the traditional software stuff, but then you have the AI events and things going on in, in just with the AI tooling. So you can enable these model indications. You can also like with guardrails, I believe you can, you can say it's either guardrails or just overall in the family of products that Bedrock has, you can tell it to send information to GuardDuty, the GuardDuty AI, so it can know how many prompt injections are happening and whatnot and basically see if there's any patterns with that.

Ali Ayub: So yeah, like this is where like the observability layer of like what's actually going on with your assets, hardware, software assets and then the AI assets, like, the tools and the MCP servers. So after you're done with that discovery phase in the observability space are several tools that ABS has. One thing, as mentioned earlier, looking at OK, observing conversations, not just the single turn, because we can do that right now as mentioned with some of the prompt injection stuff and the tooling that we have, but then really shifting over to multi turn conversations and then how these agents talk to each other. And I think that's where AWS technology at, we're building stuff on our own, but also like how that complements Wallarms technology.

Ali Ayub: So I actually, I think this is a good segue into what Tim and his team's building over at Wallarm. Yeah. And I, you try and create slides ahead of a conversation like this that will fit the conversation organically. And then as you have that conversation, you realize, oh, maybe there's a better slide I could have created for that. So, one of the things that we're, we're touching on here that I didn't necessarily plan for is sort of the layered approach to securing your environment and securing your environment in, in AWS.

Tim Irwin: And so we talked a lot about that sort of AI layer where AWS has native security controls that certainly apply to things like Bedrock and AgentCore. And Wall arm has the ability to build those traces and see the whole conversation, whether it's it's AWS components or not, and then enforce, in line enforcement on that those traces in those conversations. That's at that AI layer. But at the other layers, there's applicability to which I think, Ali was touching on at the infrastructure layer, you've got all of that great signal flowing into security hub and there's some powerful capabilities there. Wallarm can also ingest the security hub findings and plot them on a graph across all of your accounts and then do some attack path analysis so that you can see things like where an overly permissive role might allow an attacker to move laterally.

Tim Irwin: That's not AI centric, but it is relevant when you're talking about aagentic or AI attackers. And then Wallarm can also sit at that API layer and detect and block attacks against APIs, which is actually a great combination with AWS is WAF. When you put the two together, you've got really powerful coverage of web-based web application attacks and volumetric DDoS and then really API centric attacks that are focused on, API protocols like, GraphQL and gRPC. And those are the tools that some of those AI agents may be using in the environment and attackers may exploit. So there's a connection across the layers that I think frankly this slide misses because I didn't know we were going to take the conversation in that direction.

Mike Shima: We'll have to would love to continue this conversation with an additional slide that shows that because I it's very relevant what you're describing. Even if we just go back to the OpenAI Hugging Face example Hugging Face saw what on the order of 17,000 events. That's a matter of volume that they were also just saying they were struggling as humans to recreate to track down. And I think what Tim is also describing here too is that you don't just want the one point where here is the one agent, what has this one agent? How has it been prompted?

Mike Shima: You want that all of the agents that have been interacting together with the different endpoints where they've been touching either a tool call or a data access, that S3 bucket, for example, that's where it becomes a much more compelling story to answer that question of, well, what is going on here? So even if you feel some reticence about that particular slide, I think you added wonderful context that really built the story, at least in my head for sure, about what that aid, what that security comes from. I was just going to add actually I know Tim and I were talking prior to this. this is very much relevant like to, in my in just in some way like to the OSI model, right?

Ali Ayub: And how we have visibility into each layer, physical all the way up to the application layer. And what we're seeing is like, I mean, before AI became a thing, just going back to the fundamentals, right? Like, OK, we need to know what's going on at the network level, sometimes even at the physical layer level, depending on your application. If you're Uber, for example, you need to understand like your cell phone connection, like Wi-Fi, 4G5G, stuff like that. But all depends on the use case and then you understand your network, you go all the way up to your application layer. You need to have an understanding of all of that and have monitoring at each layer.

Ali Ayub: And so those traditional tools are just as important. Actually, I didn't know that you could take some of that information that we have from AWS, feed that in to Wallarms tooling, and then they can do some additional anomaly detection in addition to what we have. That's pretty cool. And then you can actually work with the WAF, the Web Application Firewall. So that's pretty cool. Thanks for sharing that. But yeah, overall it's like, OK, you have these 7 layers. And then I think from my limited understanding, I think it's the 8th layers, like the human layer.

Ali Ayub: But I think pretty soon somebody's going to come out, hey, we need to revisit the OSI model, all the stuff, these existing layers, they, they're correct. But there's also possibly an AI layer, which is really kind of within the application layer and understanding how you add on to the picture there. So, yeah, it's, it's defense in depth and having the traditional tools and then these additional tools all in that observability, space. So that then when you go to the next side of it, which is the enforcement side of things, governance, what have you have a better idea of what's going on.

Ali Ayub: It's always about getting more information. And then, oh, well, this is a problem. I didn't know it was a problem, but I'm glad we, we have the system running that gives us this observability, right? This additional insight so. Yeah, and you're going? To get all these marketing people trying to add a layer to the OSI model now. Here's the budget politics. Yeah, there's there's all kinds of layers we can throw out here, right. But to your point too, you said more information and I'll just add more quality information because to your point, yeah, you want like net flows can be very informative, for example, just to not a dig it OpenAI.

How AI Changes Attacker Behavior

Mike Shima: But as a comment, why is my Artifactory making egress to random IP addresses that aren't a PyPI repo or RubyGems repository or NPM? That seems suspicious and you just need the endpoints, the net flow. But what Tim has been also describing is that you can't just have the net flow between this agent, this agent, this agent, this other agent, this agent, this endpoint. You need to know well what's actually the behavior there. So you actually need some insight into what they're actually doing to have that comfort level with of Yep, this the this bunch of agents that Mike who will be accountable for it ultimately what he launched or doing the right thing or for what Tim is seeing from his real time analysis.

Mike Shima: We got to go and see, like, did you prompt this way, Mike, or did you download a skill and that skill actually has something that you shouldn't have been running? And I'm curious, Tim. Yeah. Well, how would you? Yeah. Well, those, those behavior patterns so many of the security tools that we have today, which are built on detecting anomalies, against patterns are built around, patterns that are established by the tools and people that have been around, right.

Tim Irwin: So, we expect, traffic or interactions to behave in a certain way because we presume the, whether we, we recognize or not, we assume there's a human being behind them or we assume there's a certain type of deterministic automation behind them. AI agents are going to change that pattern, change those patterns and we think they're going to be smart. Like we like to talk about how AI agents are going to be sophisticated attackers. I'm not sure it's true. I think they're going to be dumb and useful ways, meaning you're you're going to see patterns where a human, if a human were iterating through a certain type of attack, they would give up because it's going to take them too long.

Tim Irwin: They might change what they're doing because it's not feasible to do it at scale. The agent doesn't have the same constraints. And so you're going to see very interesting in different patterns that we're we're going to have to adapt to in terms of detection. And so, the agent as attacker becomes a very interesting topic to pay attention to going forward. It's only gonna add. Sorry, go ahead. Yeah, go ahead, Ali. I was just gonna briefly add something to mention like this is tangentially related, but there's physicists and mathematicians that are doing research using LLMs and AI and they've observed the same thing.

Ali Ayub: It's like it, the LLM can just keep ruminating and thinking about something. It can explore all possible paths. Whereas a human being, we think, oh, we have to just try this complicated approach to solve this problem because that's really how much bandwidth we have, right? We have finite time, finite resources. The what we're seeing is like, Oh, well, simpler approaches can actually solve the problem, but we would have never have thought of exploring that. And so there are, there is some evidence that like even in just research to solve some problems and get further ahead, It's, it's yeah, the ability for the AI system to try all possible things and whatnot.

Closing Recommendations and Default-Deny Controls

Mike Shima: So. Yeah, you, you might have a denial of budget as it burns through tokens to do that. But to Tim's point, you still have that. They're brute forcing to a degree just trying and trying and trying. And they're in perhaps not limited like the time that we're limited for webcast like this. But as we start to wrap up, there's a little bit of, I want to play with a little bit of history of Infosec, if you will, to riff off SQL injection, for example, that's 20 plus years old. But we also could have a policy that says use prepared statements to S3 buckets.

Mike Shima: For example, early on, in the early days of AWS, sorry, Ali, it was very classic to have, oh, here's a sensitive data in an S3 bucket that was open to the Internet. Now by default, the policies are S3 buckets are private. So in other words, Infosec is learning. Can we compress the, the, the decades of that learning for SQL injection S3 buckets and apply it to AI? And if we did do both of you have a policy or two that you would love to snap your fingers and that's enforcing security for AI these days?

Tim Irwin: It's a good question. I mean, I think, I think yes, we can compress the time frame and we already are frankly, because that's good. it just doesn't feel like it because AI is also moving. AI adoption is moving faster than previous technology. shifts and security is still behind, but it's not further behind than it was with previous technology shifts, if that makes sense. And. I appreciate the optimism. Yeah.

Tim Irwin: I mean, I think there are lessons being learned there actively that compress the time frame for security to sort of be meaningfully useful. But there are some things we can't learn until we learn them, until they happen. That's certainly true as well. The question about what's 1 policy that you would put in place, that's a harder 1 for me because I, I'm not sure that I can come up with just one. I do think that I think that the biggest challenge that organization should address is, is understanding the agent's behaviour.

Tim Irwin: And I'm thinking specifically of production agents that have been deployed, not so much the development agent that's writing code there. There are challenges there, but I'm thinking about the agent that interacts with your customers either directly or indirectly and takes actions on their behalf. That's where the behavior drift is a problem. I mean, I do think that we are, we're in this funny position of, of trying, of expecting deterministic outcomes from a system that is intentionally designed to be non deterministic. Like we want an agent to take these actions and behave the same way every time when LLMs are specifically designed not to do that.

Tim Irwin: And so maybe that understanding is sort of the key point I'm trying to make. Yeah, I know the focus where the impact of the negative impact could be the biggest. I that I think that's a great recommendation. Ali, you want to take us out with some final bit of wisdom from this last hour of conversation. Yeah, absolutely. I learned a lot. Thank you for everybody's time and I, this was a great, great chat. I think one policy I can think of, and to Tim's point, there's several, but one thing that I think is like a practical insight that we've seen is by default tools.

Ali Ayub: So you should first classify whether the tool has network access and then what types of resources it's, it's hitting, what types of repositories, whether that's like, a token repository or whatever. And then basically a default policy of like, if you have network access, it should be off. And then essentially like the AI is going to like, it's going to fail at what it's doing. And those failures are actually good for us because it's like, OK, it failed. All right. Should we enable this? Should we not and yes to the earlier point of the conversation that, eventually we might have human an automated system that determines, OK, should this get, access or not.

Ali Ayub: But this policy by default is, like I said, essentially tools that are risky or have the potential to be exploited there by default turned off. And that can be implemented on the left side of things, like at the software side of things, when you, before you push something to production, you can have scanning and you can say, oh, wait, you're giving these privileges to this tool. Did you do any kind of analysis? Did you get approvals from a security expert, whatever? And then also at the production side of things like actually having access control mechanisms in place. So that is one policy I can think of that has served well historically.

Mike Shima: So. Oh, that, that's, that's wonderful. And for as much as both of you have been talking about automation AI and trying to minimize humans in the loop, I very much appreciated having the two of you in the loop have for this conversation. So thanks so much for this, Tim and Ali. Yeah. Thank you, Mike. Thanks for having us. Thank you. Thanks everyone who joined us. Thanks again to Wallarm for sponsoring this webcast.

Mike Shima: And if you didn't scan the QR code to find Wallarm on the AWS marketplace, that will still be available in the resources after this program. It's not even a Rick roll either. Legit link. But so thank you to everyone who joined us today and thank you to everyone who's listening to this recording. In the future, please do check out Wall, arm.com and keep an eye on scworld.com for more expert web casts like today's.