26 lines
6.8 KiB
JSON
26 lines
6.8 KiB
JSON
{
|
||
"schema_version": 1,
|
||
"saved_at": "2026-08-07T05:00:13.6008914Z",
|
||
"content_sha256": "077b9628712d8b2d6aca6c250a463508163659ac635b17cdfd0dbae8645487a5",
|
||
"result": {
|
||
"title": "Agent Behavior Monitoring and Anomaly Detection — Agentic AI AI Governance Control | AI Governance Institute",
|
||
"url": "https://aigovernance.com/controls/agent-behavior-monitoring",
|
||
"snippet": "Monitor for deviation: an agent suddenly calling external APIs at 10x normal rate, accessing data stores it rarely touched, or producing outputs far outside its length or format baseline warrants investigation. Distinguish behavioral drift from intentional changes: re-establish baselines after model updates, prompt changes, or capability additions.",
|
||
"content": "← Agentic AI\n\nAGT · Agentic AI AGT-011 High effort Agent-relevant\n\nAgent Behavior Monitoring and Anomaly Detection\n\nContinuously monitor deployed agents for behavioral drift, unusual tool call patterns, unexpected resource consumption, and actions outside their defined operational envelope.\n\nObjective\n\nDetect agent misbehavior, compromise, model drift, or unintended capability escalation before it produces harm — by watching behavioral signals rather than just final outputs.\n\nMaturity Levels\n\nInitial\n\nNo agent behavior monitoring exists; issues are detected only when users report problems or downstream systems fail.\n\nDeveloping\n\nBasic output logging exists but no behavioral baselines or anomaly alerts are in place.\n\nDefined\n\nBehavioral baselines are established per agent type; deviations from normal tool call patterns, resource use, or action sequences trigger alerts.\n\nManaged\n\nAlerts are triaged by a designated team on a defined SLA; behavioral anomalies feed into agent evaluation and update cycles.\n\nOptimizing\n\nAutomated analysis identifies behavioral drift in real time; agents can be paused or constrained automatically when anomaly thresholds are exceeded.\n\nEvidence Requirements\n\nWhat an auditor or assessor would expect to see for this control.\n\n— Behavioral baseline documentation per agent, including normal tool call frequency, resource consumption ranges, and action sequence patterns with the baseline period and data volume\n\n— Alert configuration records showing which deviations trigger alerts, alert severity levels, and routing/escalation paths\n\n— Anomaly investigation records for a sample period, showing alert triage, root cause determination, and resolution or escalation\n\n— Baseline refresh records confirming baselines were re-established following intentional model or prompt changes\n\n— Integration evidence showing agent behavioral alerts are routed to and actioned by a designated security or governance function within the defined SLA\n\nImplementation Notes\n\nKey steps\n\nEstablish behavioral baselines per agent deployment: typical tool call frequency, common action sequences, average token and API consumption, expected output types, and error rates.\n\nMonitor for deviation: an agent suddenly calling external APIs at 10x normal rate, accessing data stores it rarely touched, or producing outputs far outside its length or format baseline warrants investigation.\n\nDistinguish behavioral drift from intentional changes: re-establish baselines after model updates, prompt changes, or capability additions.\n\nBuild alert playbooks for the most actionable anomaly patterns: excessive recursive calls, first-use of high-risk permissions, sudden spikes in rejection or error rates, and access to out-of-scope resources.\n\nRoute agent behavioral alerts into your SOC workflow alongside infrastructure monitoring — agent incidents look different from application incidents but require the same urgency and documentation.\n\nExample Implementation\n\nFinancial services firm running document processing agents over customer loan files\n\nAgent Behavioral Baseline — Loan Document Processing Agent\n\nBaseline period: 30 days post-deployment (sampled from 500+ sessions)\n\nMetric\n\nNormal Range\n\nAlert Threshold\n\nAlert Routing\n\nTool calls per session\n\n8–14\n\n\u003e25 or \u003c3\n\nAI Eng on-call\n\nExternal API calls per session\n\n2–4\n\n\u003e10\n\nAI Eng + SOC\n\nSession duration\n\n45–120 seconds\n\n\u003e300 seconds\n\nAI Eng on-call\n\nToken consumption per session\n\n4,000–8,000\n\n\u003e20,000\n\nAI Eng on-call\n\nError / rejection rate\n\n\u003c5% of sessions\n\n\u003e20% in any 1-hour window\n\nAI Eng + SOC\n\nFirst-use of any permission\n\nN/A\n\nAny\n\nSOC immediate\n\nBaseline refresh: Re-established within 5 business days of any model update, prompt change, or new tool addition.\n\nTriage SLA: P1 alerts (first-use of permission, external API spike) acknowledged within 15 minutes.\n\nControl Details\n\nControl ID AGT-011\n\nDomain Agentic AI\n\nTypical owner AI Engineering / SOC / AI Governance Team\n\nImplementation effort High effort\n\nAgent-relevant Yes\n\nTags\n\nmonitoring anomaly detection behavioral drift observability agent safety\n\nMapped Regulations\n\nNIST AI 600-1 Generative AI Profile → OWASP Top 10 for Large Language Model Applications →\n\nRelated Controls\n\nAGT-006 Agent Action Audit Trail → MON-004 AI Output Anomaly Detection → AGT-007 Agent Scope and Task Boundaries → AGT-012 Agent Kill Switch and Emergency Stop →\n\nRelated Playbook\n\nHow do we govern AI agents that take autonomous actions? →\n\nRecent Coverage\n\nAmazon's KiroRank Shutdown Exposes Metric Gaming as an AI Governance Risk → CASB and DLP Cannot See Inside AI Prompts. That Is Now a Material Control Gap. → NIST's Agent Standards Gap Leaves Enterprises Without Enforceable Agentic AI Controls → Mayer Brown Guidance Exposes Gaps in Existing AI Governance for Agentic Systems → Okta's $200M Permiso Deal Puts AI Agent Identity Governance on the Vendor Map →\n\nGet control updates weekly\n\nNew and updated controls, maturity guidance, and the regulatory changes behind them. Every Thursday.\n\nPowered by Buttondown.",
|
||
"content_type": "text/html",
|
||
"query": "How is the documentation of baselines and expected normal behavior for AI agents implemented in practice?",
|
||
"language": "en-US",
|
||
"round": 1,
|
||
"fetched": true,
|
||
"relevant": true,
|
||
"relevance": 0.9828571428571429,
|
||
"source_quality": "primary",
|
||
"source_quality_score": 0.864,
|
||
"actionable": true,
|
||
"covered_gap_ids": [
|
||
"KG-001"
|
||
],
|
||
"assessment_reason": "The source provides a detailed explanation of how to implement behavioral monitoring and anomaly detection for AI agents, including the establishment of baselines, alert configuration, and evidence requirements. It also includes a practical example of how to document and monitor agent behavior, which directly addresses the question of how baselines and expected normal behavior are documented and implemented in practice."
|
||
}
|
||
}
|