Securing AI Systems at Runtime: Visibility and Governance

The article highlights a critical shift in AI security concerns, moving from model selection and prompt engineering to the challenges of securing AI systems after deployment. Traditional security models, built on assumptions of static services and clear ownership, are ill-equipped to handle the dynamic and often unpredictable behavior of AI agents, LLMs, and MCPs in production environments. The key issues identified include agents calling unexpected tools, LLMs accessing internal APIs through unforeseen chains, and execution identities diverging from user identities. Addressing these challenges requires a new approach focused on runtime visibility and governance. This involves understanding what AI components are doing, who is acting on whose behalf, what tools are being invoked, and where data is flowing during real executions. Levo.ai's efforts to document and ship solutions in this area represent a significant step towards addressing these emerging security concerns. The shift towards runtime security aligns with the broader trend of DevSecOps, where security is integrated throughout the development lifecycle. The EU AI Act Art. 50 emphasizes the need for transparency and accountability in AI systems, and runtime monitoring plays a crucial role in ensuring compliance with these requirements. By providing insights into the behavior of AI systems in production, organizations can better understand and mitigate potential risks, fostering greater trust and confidence in AI technologies. This proactive approach to security is essential for enabling the responsible and sustainable adoption of AI across various industries.