Monitoring AI Agent Activity with Sentinel and OpenTelemetry

In this blog post Monitoring AI Agent Activity with Sentinel and OpenTelemetry we will show how to spot risky actions, control unexpected costs and understand what your AI agents are doing before a small problem becomes a serious incident.

Many businesses can tell you which AI agents they have deployed, but not what those agents did yesterday. An agent may search company files, call external services, create records or trigger another agent, yet its activity is often scattered across application logs that nobody routinely checks.

OpenTelemetry provides a consistent way to record that activity. Microsoft Sentinel, Microsoft’s cloud-based security monitoring platform, then analyses the records alongside identity, device and cloud security information to find behaviour that needs attention.

What the technology does in plain English

OpenTelemetry is an open standard for collecting traces, logs and measurements from applications. A trace is simply a timeline showing the steps taken to complete a task, such as which agent started the task, which AI model it called, which tools it used and where something failed.

Microsoft Sentinel brings security information into a central workspace and searches it for suspicious patterns. It can create an incident when a rule is triggered and use automated workflows, called playbooks, to notify the right people or begin a controlled response.

Together, these technologies answer questions that standard application monitoring often misses:

• Which agent performed the action?
• Was it acting for an employee, another agent or an automated process?
• Which files, databases, tools or business systems did it access?
• Did its behaviour match the permissions and purpose it was given?
• How many AI model calls and tokens did the task consume?
• Did the agent fail safely when an action was denied?

If you already collect agent telemetry, our guide to monitoring Claude Code and agent activity with OpenTelemetry explains the operational foundation. The next step is turning selected telemetry into useful security detections.

How the monitoring flow works

The first step is adding OpenTelemetry instrumentation to the agent application. Instrumentation means adding a small amount of configuration and code so the application records important events in a standard format.

Those records can be sent to Azure Monitor and Application Insights, which provide Microsoft’s application monitoring and troubleshooting tools. Microsoft now supports AI agent views based on OpenTelemetry’s generative AI conventions, including visibility into agent performance, tool calls, token use and errors.

Relevant security events are then made available in the Log Analytics workspace used by Microsoft Sentinel. For custom events, Azure Monitor’s Logs Ingestion API and data collection rules can place selected fields into a dedicated table, filter unnecessary information and mask sensitive values before storage.

A practical flow looks like this:

1. The AI agent begins a task and creates a trace identifier.
2. OpenTelemetry records model calls, tool use, outcomes, timing and token consumption.
3. Azure Monitor receives the telemetry and stores the required records.
4. Microsoft Sentinel compares agent activity with Microsoft Entra ID sign-ins, Defender alerts, Azure changes and other security data.
5. A detection rule creates an incident when behaviour exceeds an agreed risk threshold.
6. A playbook alerts the security team and can start a response with human approval.

Record business context, not just technical events

A log stating that an application made an API request is rarely useful to a CIO. A record stating that the payroll assistant exported employee data after its normal workflow was changed is much more meaningful.

Every important agent event should therefore include enough context to support a business decision:

• Agent identity and owner so someone is accountable for it.
• Agent version so incidents can be connected to recent changes.
• User or requesting system so delegated actions can be investigated.
• Tool and data source so you know what the agent attempted to access.
• Outcome such as successful, denied, failed or awaiting approval.
• Risk classification based on the action’s business impact.
• Trace identifier so the full chain can be reconstructed.
• Token and cost data so waste and runaway loops are visible.

For agents that communicate with other agents, preserve the same trace identifier across the conversation. Our article on monitoring A2A agent communication in Azure covers this multi-agent tracing model in more detail.

A simple instrumentation example

The following Python example is deliberately simplified. It creates a trace around a tool call and records security-relevant facts without storing the employee’s request or the tool’s full response.

from opentelemetry import trace

tracer = trace.get_tracer("finance-agent")

with tracer.start_as_current_span("agent.tool.execute") as span:
 span.set_attribute("gen_ai.agent.name", "finance-assistant")
 span.set_attribute("gen_ai.agent.version", "2.4.1")
 span.set_attribute("business.owner", "finance-operations")
 span.set_attribute("business.tool.name", "invoice-export")
 span.set_attribute("business.action", "export")
 span.set_attribute("business.risk_level", "high")
 span.set_attribute("enduser.id", "user-1842")

 result = run_invoice_export()

 span.set_attribute("business.outcome", result.status)
 span.set_attribute("business.records_affected", result.count)

The exact fields should match your governance requirements. The important point is consistency: every agent should describe identities, tools, outcomes and risk in the same way.

Use Sentinel to detect behaviour that matters

Microsoft Sentinel uses Kusto Query Language, usually called KQL, to search log data. A detection can look for repeated denied actions, unusually large exports, unexpected tools, sudden token increases or high-risk activity from a newly deployed agent version.

For example, the following illustrative query looks for agents repeatedly attempting actions that were denied:

AgentActivity_CL
| where TimeGenerated > ago(15m)
| where Outcome_s == "denied"
| summarize DeniedActions=count(),
 Tools=make_set(ToolName_s),
 TraceIds=make_set(TraceId_s)
 by AgentId_s, UserId_s, bin(TimeGenerated, 5m)
| where DeniedActions >= 5

A single denied action may show that a control worked. Five denials in a few minutes may indicate a broken workflow, an agent stuck in a retry loop or an attempt to work around its permissions.

Useful Sentinel detections for most organisations include:

• An agent using a privileged tool it has never used before.
• A large increase in model calls or tokens compared with its normal workload.
• Repeated attempts to access restricted files or systems.
• A high-risk action without the required human approval.
• An agent identity signing in from an unexpected environment.
• A new agent version followed by increased failures or unusual exports.

Do not turn monitoring into a privacy problem

Collecting every prompt and response may sound useful, but it can create unnecessary privacy, legal and security exposure. Agent conversations may contain customer details, employee information, commercial documents or credentials that should never have appeared in a prompt.

Application Insights can store generative AI content, including prompts, outputs, system instructions and tool interactions, in a dedicated table. Access to this sensitive content should be restricted separately, and organisations should avoid capturing it unless there is a clear and approved need.

Start with metadata such as agent identity, model, tool, outcome, timing, token count and risk level. Apply role-based access, suitable retention periods and data collection rules that remove fields you do not need.

A real-world business scenario

Consider a 200-person professional services company using an AI agent to prepare project reports. After an update, the agent begins retrying a failed document search hundreds of times and then calls a broader file-search tool.

Without joined-up monitoring, the first warning may be a larger Azure bill or a complaint that confidential files appeared in an unexpected report. With OpenTelemetry and Sentinel, the company can see the retry loop, connect it to the new version, identify the broader tool call and suspend the workflow before sensitive information leaves the approved process.

The outcome is not simply better logging. It is faster investigation, lower AI consumption costs and a defensible record showing what happened and how the business responded.

Start with one agent and three risks

Do not begin by sending every available event into Sentinel. That approach increases storage costs and produces alerts your team will eventually ignore.

Choose one business-critical agent and identify its three most important risks. Instrument those actions, create a small number of meaningful detections and test the response process with the agent’s business owner.

CloudPro Inc combines more than 20 years of enterprise IT experience with hands-on work across Microsoft Azure, Microsoft 365, OpenAI, Claude, Microsoft Defender and Wiz. As a Microsoft Partner and Wiz Security Integrator based in Melbourne, we help organisations build monitoring that supports real operational decisions rather than another dashboard nobody checks.

If you are not sure whether your AI agents are creating blind spots, unexpected costs or compliance gaps, we are happy to review the current setup and identify the first practical monitoring steps — no strings attached.