When it comes to frontier models, unrelenting and persistent attacks can lead to failure. The patterns of failure vary depending on the model and developer. Red teaming has shown that it’s not always sophisticated, complex attacks that bring a model down, but rather the continuous, random attempts by attackers that can force a model to fail.
AI apps and platform builders need to be aware of this harsh reality as they release new products. Relying on a frontier model that is prone to red team failures due to persistency alone is risky, akin to building a house on sand. Even with red teaming, frontier models like LLMs are falling behind adversarial and weaponized AI.
The arms race is already underway
Cybercrime costs reached $9.5 trillion in 2024, and are projected to exceed $10.5 trillion in 2025. LLM vulnerabilities contribute to this trend. For example, a financial services firm that deployed a customer-facing LLM without adversarial testing experienced a costly data leak within weeks. Another enterprise software company had its salary database leaked after using an LLM for financial modeling.
The UK AISI/Gray Swan challenge conducted 1.8 million attacks on 22 models, and every model was compromised. This demonstrates that no current frontier system can withstand determined, well-resourced attacks.
Builders now face a choice: integrate security testing early on, or deal with breaches later. Tools like PyRIT, DeepTeam, Garak, and OWASP frameworks are available, but proper execution is key.
Organizations that treat LLM security as a feature rather than a foundation will learn the hard way. In the arms race, those who act swiftly are rewarded.
Red teaming highlights the nascent nature of frontier models
The gap between offensive capability and defensive readiness is widening. Adversaries are evolving rapidly, leveraging AI to accelerate attacks. Red teaming results show that every frontier model eventually fails under sustained pressure.
It’s crucial for AI builders to review system cards after new model releases to understand the security and reliability measures employed by model providers. The differences in red teaming practices between companies like Anthropic and OpenAI can have significant implications for enterprise AI.
Attack surfaces are constantly evolving
Builders must recognize the dynamic nature of attack surfaces that red teams need to cover, despite incomplete knowledge of potential threats. Frameworks like OWASP’s 2025 Top 10 for LLM Applications highlight critical vulnerabilities that businesses building AI apps need to address to secure their systems.
The shift to AI-driven models introduces new risks, as attackers are now operating at machine scale. The use of AI in attacks poses unprecedented challenges for defenders.
Validation of security practices by model providers
Each frontier model provider employs a unique red teaming process to demonstrate the security, robustness, and reliability of their system. By examining system cards, it becomes evident how different providers approach security validation and testing.
Comparing red teaming results between companies like Anthropic and OpenAI reveals varying measurement philosophies and approaches to security testing. These differences can impact the compatibility of platforms with a building team’s priorities.
Challenges in defending against adaptive attackers
Defensive tools struggle to keep up with adaptive attackers who continuously refine their strategies. Builders must conduct their own testing and not rely solely on model providers’ claims.
Open-source frameworks like DeepTeam and Garak are emerging to address testing gaps in LLM systems, but adoption by builders lags behind attacker sophistication.
Actions for AI builders
AI builders must prioritize security measures to protect their systems from evolving threats. Input and output validation, regular red teaming, and controlling agent permissions are essential practices. Additionally, supply chain scrutiny and separating instructions from data are crucial steps in safeguarding AI systems.
By following these guidelines and staying vigilant against emerging threats, AI builders can enhance the security and resilience of their systems in the face of evolving cyber risks.
