Safety Rules Don’t Survive the Context Window: Quantifying and Fixing Agent Compliance Decay

AI systems, specifically LLM agents, are reported to lose their safety instructions because these rules, initially set in the 'system prompt,' are pushed out of the model's 'context window' as conversations lengthen. This 'compliance decay' is described as structural, meaning it's an inherent behavior rather than a simple error. For example, a medical chatbot using GPT-4 reportedly began issuing symptom assessments after its dialog exceeded 8,000 tokens, despite having explicit 'do not diagnose' instructions in its configuration.

This problem is not unique to a single LLM, appearing across OpenAI, Llama 2, and MPT models. While methods like prepending prompts or compacting old dialog are often suggested, they frequently fail to reliably preserve rules, as existing frameworks may not guarantee their retention. Even large context windows, such as GPT-4’s 128K, show compliance loss after 15K–30K tokens if prompts are not reinforced.

The most effective approach, based on the reporting, is an 'architectural' solution involving a 'Generation–Verification–Reinforcement Loop.' This means that after the AI generates a response, it is immediately checked for safety violations using methods like NER, regex, or fine-tuned classifiers. If a violation is detected, the rule is either reinjected, or the unsafe result is blocked. This constant, inline verification, rather than relying on initial instructions to persist, is reported to maintain much higher compliance.

Experiments with this approach show sub-2% rule violations even in dialogs up to 10,000 tokens across various LLM APIs. This suggests that explicit code-level verification is crucial for practical reliability, moving beyond sole reliance on prompt-based or memory-heuristic methods in real-world deployments.