{"id":59662,"date":"2026-11-09T09:01:47","date_gmt":"2026-11-08T23:01:47","guid":{"rendered":"https:\/\/www.cloudproinc.com.au\/index.php\/2026\/11\/09\/observability-for-ai-agents-with-azure-monitor-and-opentelemetry\/"},"modified":"2026-11-09T09:03:31","modified_gmt":"2026-11-08T23:03:31","slug":"observability-for-ai-agents-with-azure-monitor-and-opentelemetry","status":"publish","type":"post","link":"https:\/\/www.cloudproinc.com.au\/index.php\/2026\/11\/09\/observability-for-ai-agents-with-azure-monitor-and-opentelemetry\/","title":{"rendered":"Observability for AI Agents with Azure Monitor and OpenTelemetry"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">In this blog post <strong>Observability for AI Agents with Azure Monitor and OpenTelemetry<\/strong> we will explain how to see what your AI agents are doing, why they fail and where money is being spent before small issues become expensive operational problems.<\/p>\n\n\n\n<!--more-->\n\n\n\n<p class=\"wp-block-paragraph\">An AI agent can appear healthy while quietly producing poor results. It may answer users, call business systems and complete requests, yet still choose the wrong tool, repeat the same step or consume far more AI capacity than expected.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Observability gives you a structured record of that journey. Instead of only knowing that a request succeeded or failed, you can see each model call, tool action, delay, retry and outcome that contributed to the result.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What AI agent observability actually means<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Traditional application monitoring usually focuses on availability, errors and response times. Those measurements remain important, but they do not tell you whether an AI agent made a sensible decision.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An agent might be technically available while giving customers incorrect information. It might complete a task successfully after making six unnecessary model calls, turning a low-cost interaction into an expensive one.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">OpenTelemetry is an open standard for collecting operational information from applications. It records requests as <em>traces<\/em>, with each trace containing smaller steps called <em>spans<\/em>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For an AI agent, a trace could include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The user request and the agent that received it.<\/li>\n<li>The AI model selected to process the request.<\/li>\n<li>Documents or knowledge sources searched.<\/li>\n<li>Tools, APIs or business systems called by the agent.<\/li>\n<li>Retries, errors and policy checks.<\/li>\n<li>Input and output token usage, which influences AI consumption costs.<\/li>\n<li>The final result and whether the business task was completed.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">OpenTelemetry GenAI conventions provide common labels for this information. That consistency matters because it lets Azure Monitor understand telemetry from different agent frameworks without every development team inventing its own monitoring format.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How Azure Monitor fits into the picture<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Azure Monitor is Microsoft\u2019s platform for collecting, analysing and alerting on operational data. Application Insights, a capability within Azure Monitor, stores application traces and helps teams investigate individual transactions.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When an agent is connected correctly, OpenTelemetry collects the activity and sends it to Application Insights. Your team can then move from a high-level dashboard into the detailed trace for a specific failed, slow or unusually expensive request.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For agents hosted in Microsoft Foundry, server-side tracing may provide useful visibility with minimal code changes. Client-side instrumentation can then be added around custom business logic, external services and steps that occur outside the hosted agent.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This complements the service-level monitoring covered in our guide to monitoring Azure AI services. Service monitoring tells you whether the platform is healthy. Agent tracing tells you what happened inside an individual workflow.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">A simple tracing pattern<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The following Python example configures Azure Monitor and creates a span around a support process. The span adds business context without recording the customer\u2019s full message.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import os\nfrom azure.monitor.opentelemetry import configure_azure_monitor\nfrom opentelemetry import trace\nfrom opentelemetry.trace import Status, StatusCode\n\nconfigure_azure_monitor(\n connection_string=os.environ[\n &quot;APPLICATIONINSIGHTS_CONNECTION_STRING&quot;\n ]\n)\n\ntracer = trace.get_tracer(&quot;customer-service-agent&quot;)\n\nwith tracer.start_as_current_span(&quot;process-support-request&quot;) as span:\n span.set_attribute(&quot;gen_ai.agent.name&quot;, &quot;support-agent&quot;)\n span.set_attribute(&quot;gen_ai.agent.version&quot;, &quot;2026-11&quot;)\n span.set_attribute(&quot;business.process&quot;, &quot;customer-support&quot;)\n\n try:\n result = run_support_agent()\n span.set_attribute(&quot;business.outcome&quot;, &quot;completed&quot;)\n except Exception as error:\n span.record_exception(error)\n span.set_status(Status)\n raise\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Framework instrumentation can create the more detailed spans for model calls and tool activity. The custom span above connects those technical steps to the process the business actually cares about.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That distinction is important. A dashboard showing 99.9 per cent availability means little if only 70 per cent of customer requests reach an acceptable outcome.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The measurements leaders should insist on<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Business completion rate<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Measure whether the agent completed its assigned task, not simply whether it returned an answer. For example, did it actually update the customer record, create the order or route the request correctly?<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Response time by step<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Separate time spent waiting for the model from time spent searching documents or calling business systems. This prevents teams from upgrading an AI model when the real bottleneck is an outdated internal application.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Token usage and estimated cost<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Track consumption by agent, version, department and business process. Sudden increases can reveal overly long prompts, repeated calls, unnecessary conversation history or agents stuck in retry loops.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Tool success and retry rates<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">An agent may depend on a CRM, finance platform or internal database. Monitoring each tool call shows which integration is causing failures and whether the agent is responding safely when a system becomes unavailable.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Quality and safety results<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Operational success does not guarantee a useful answer. Production monitoring should therefore include approved quality evaluations, policy failures and escalation rates alongside technical performance.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Multi-agent environments need an additional layer because one agent may pass work to several others. Our article on monitoring agent-to-agent communication in Azure explains how to follow those hand-offs without losing the original business context.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">A practical 200-person business scenario<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Consider a 200-person company introducing an agent to classify support requests, search internal documentation and create tickets. Staff initially report that the agent feels slow, while finance notices that monthly AI consumption is higher than forecast.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A trace reveals that the agent searches the same document source three times before creating each ticket. It also sends the entire conversation history to the model at every step.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The agent is not technically broken. Every request eventually succeeds, so a standard uptime dashboard shows no obvious problem.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">After removing the repeated search and limiting unnecessary conversation history, the company reduces response time and AI consumption without changing models. Observability turns a vague complaint into a specific, measurable fix.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Do not turn tracing into a privacy problem<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Detailed traces can contain prompts, responses, customer information, tool arguments and retrieved documents. Collecting everything by default can create unnecessary exposure under Australian privacy obligations and contractual confidentiality requirements.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Start with metadata such as timing, token counts, agent versions, tool names and outcomes. Capture full content only when there is a clear troubleshooting or audit need, with access controls, redaction and appropriate retention settings.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Restrict access through Azure roles and review who can read the connected Application Insights and Log Analytics data. Sampling and retention rules can also prevent telemetry costs from growing without limit.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Security teams can extend this operational view into threat detection. We cover that next step in wiring AI agent telemetry into Microsoft Sentinel, which can identify unusual tool use, prompt injection attempts and suspicious data access.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">A sensible rollout plan<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Choose one important workflow.<\/strong> Avoid trying to monitor every AI experiment at once.<\/li>\n<li><strong>Define the business outcome.<\/strong> Decide what successful completion means before building dashboards.<\/li>\n<li><strong>Connect Application Insights.<\/strong> Confirm traces are arriving and can be followed from the initial request to the final result.<\/li>\n<li><strong>Add business context.<\/strong> Record safe labels such as department, workflow, agent version and outcome.<\/li>\n<li><strong>Create meaningful alerts.<\/strong> Start with failure rates, unusual latency, retry loops and unexpected token growth.<\/li>\n<li><strong>Test privacy controls.<\/strong> Confirm that prompts, credentials and personal information are not being collected unintentionally.<\/li>\n<li><strong>Review after every release.<\/strong> Compare agent versions so you can quickly detect higher costs or lower completion rates.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Some agent monitoring features and framework integrations may still be offered as previews. Check their current production support, limitations and service commitments before making them critical to your operating model.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Make the agent accountable to the business<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Good observability is not about creating more dashboards. It is about answering practical questions: Is the agent completing useful work, what is each outcome costing, where are failures occurring and can we prove what happened?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">CloudPro Inc combines more than 20 years of enterprise IT experience with hands-on expertise across Azure, Microsoft 365, OpenAI, Claude, Defender and Wiz. As a Melbourne-based Microsoft Partner and Wiz Security Integrator, we help organisations build monitoring that is useful to technical teams and understandable to business leaders.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If your AI agents are already operating but you cannot clearly explain their cost, behaviour or business results, we are happy to review the current setup and identify the visibility gaps \u2014 no strings attached.<\/p>\n\n\n","protected":false},"excerpt":{"rendered":"<p>Learn how Azure Monitor and OpenTelemetry traces help you control AI agent costs, diagnose failures, protect sensitive data and measure real business outcomes.<\/p>\n","protected":false},"author":1,"featured_media":59664,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[80,21,13],"tags":[],"class_list":["post-59662","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-agents","category-azure-monitor","category-blog"],"jetpack-related-posts":[{"id":59289,"url":"https:\/\/www.cloudproinc.com.au\/index.php\/2026\/10\/21\/why-azure-ai-foundry-voice-agents-need-observability-before-launch\/","url_meta":{"origin":59662,"position":0},"title":"Why Azure AI Foundry Voice Agents Need Observability Before Launch","author":"CPI Staff","date":"October 21, 2026","format":false,"excerpt":"Voice agents can sound impressive while hiding costly failures. Learn which metrics, privacy controls and launch gates to establish before customers start calling.","rel":"","context":"In &quot;AI Agents&quot;","block_context":{"text":"AI Agents","link":"https:\/\/www.cloudproinc.com.au\/index.php\/category\/ai-agents\/"},"img":{"alt_text":"","src":"\/wp-content\/uploads\/2026\/10\/why-azure-ai-foundry-voice-agents-need-observability-before-launch.png","width":350,"height":200,"srcset":"\/wp-content\/uploads\/2026\/10\/why-azure-ai-foundry-voice-agents-need-observability-before-launch.png 1x, \/wp-content\/uploads\/2026\/10\/why-azure-ai-foundry-voice-agents-need-observability-before-launch.png 1.5x, \/wp-content\/uploads\/2026\/10\/why-azure-ai-foundry-voice-agents-need-observability-before-launch.png 2x, \/wp-content\/uploads\/2026\/10\/why-azure-ai-foundry-voice-agents-need-observability-before-launch.png 3x, \/wp-content\/uploads\/2026\/10\/why-azure-ai-foundry-voice-agents-need-observability-before-launch.png 4x"},"classes":[]},{"id":57880,"url":"https:\/\/www.cloudproinc.com.au\/index.php\/2026\/10\/12\/monitoring-claude-code-and-agent-activity-with-opentelemetry\/","url_meta":{"origin":59662,"position":1},"title":"Monitoring Claude Code and Agent Activity With OpenTelemetry","author":"CPI Staff","date":"October 12, 2026","format":false,"excerpt":"Claude Code can move fast, but leaders need visibility. Here is how OpenTelemetry helps monitor cost, risk, productivity, and agent behaviour without slowing teams down.","rel":"","context":"In &quot;AI Coding Agents&quot;","block_context":{"text":"AI Coding Agents","link":"https:\/\/www.cloudproinc.com.au\/index.php\/category\/ai-coding-agents\/"},"img":{"alt_text":"","src":"\/wp-content\/uploads\/2026\/07\/monitoring-claude-code-and-agent-activity-with-opentelemetry.png","width":350,"height":200,"srcset":"\/wp-content\/uploads\/2026\/07\/monitoring-claude-code-and-agent-activity-with-opentelemetry.png 1x, \/wp-content\/uploads\/2026\/07\/monitoring-claude-code-and-agent-activity-with-opentelemetry.png 1.5x, \/wp-content\/uploads\/2026\/07\/monitoring-claude-code-and-agent-activity-with-opentelemetry.png 2x, \/wp-content\/uploads\/2026\/07\/monitoring-claude-code-and-agent-activity-with-opentelemetry.png 3x, \/wp-content\/uploads\/2026\/07\/monitoring-claude-code-and-agent-activity-with-opentelemetry.png 4x"},"classes":[]},{"id":58230,"url":"https:\/\/www.cloudproinc.com.au\/index.php\/2026\/10\/12\/monitoring-ai-agent-activity-with-sentinel-and-opentelemetry\/","url_meta":{"origin":59662,"position":2},"title":"Monitoring AI Agent Activity with Sentinel and OpenTelemetry","author":"CPI Staff","date":"October 12, 2026","format":false,"excerpt":"Learn how Microsoft Sentinel and OpenTelemetry can reveal risky AI agent behaviour, control costs and give leaders a clear audit trail without collecting unnecessary sensitive data.","rel":"","context":"In &quot;AI Agents&quot;","block_context":{"text":"AI Agents","link":"https:\/\/www.cloudproinc.com.au\/index.php\/category\/ai-agents\/"},"img":{"alt_text":"","src":"\/wp-content\/uploads\/2026\/08\/monitoring-ai-agent-activity-with-sentinel-and-opentelemetry.png","width":350,"height":200,"srcset":"\/wp-content\/uploads\/2026\/08\/monitoring-ai-agent-activity-with-sentinel-and-opentelemetry.png 1x, \/wp-content\/uploads\/2026\/08\/monitoring-ai-agent-activity-with-sentinel-and-opentelemetry.png 1.5x, \/wp-content\/uploads\/2026\/08\/monitoring-ai-agent-activity-with-sentinel-and-opentelemetry.png 2x, \/wp-content\/uploads\/2026\/08\/monitoring-ai-agent-activity-with-sentinel-and-opentelemetry.png 3x, \/wp-content\/uploads\/2026\/08\/monitoring-ai-agent-activity-with-sentinel-and-opentelemetry.png 4x"},"classes":[]},{"id":58851,"url":"https:\/\/www.cloudproinc.com.au\/index.php\/2026\/10\/12\/wiring-ai-agent-telemetry-into-sentinel-to-detect-threats-early\/","url_meta":{"origin":59662,"position":3},"title":"Wiring AI Agent Telemetry into Sentinel to Detect Threats Early","author":"CPI Staff","date":"October 12, 2026","format":false,"excerpt":"Connect AI agent activity to Microsoft Sentinel so risky tool use, prompt injection and unusual data access stand out before a small anomaly becomes a costly incident.","rel":"","context":"In &quot;AI Agents&quot;","block_context":{"text":"AI Agents","link":"https:\/\/www.cloudproinc.com.au\/index.php\/category\/ai-agents\/"},"img":{"alt_text":"","src":"\/wp-content\/uploads\/2026\/09\/wiring-ai-agent-telemetry-into-sentinel-to-detect-threats-early.png","width":350,"height":200,"srcset":"\/wp-content\/uploads\/2026\/09\/wiring-ai-agent-telemetry-into-sentinel-to-detect-threats-early.png 1x, \/wp-content\/uploads\/2026\/09\/wiring-ai-agent-telemetry-into-sentinel-to-detect-threats-early.png 1.5x, \/wp-content\/uploads\/2026\/09\/wiring-ai-agent-telemetry-into-sentinel-to-detect-threats-early.png 2x, \/wp-content\/uploads\/2026\/09\/wiring-ai-agent-telemetry-into-sentinel-to-detect-threats-early.png 3x, \/wp-content\/uploads\/2026\/09\/wiring-ai-agent-telemetry-into-sentinel-to-detect-threats-early.png 4x"},"classes":[]},{"id":57266,"url":"https:\/\/www.cloudproinc.com.au\/index.php\/2026\/10\/12\/the-new-enterprise-ai-stack-identity-observability-and-control\/","url_meta":{"origin":59662,"position":4},"title":"The New Enterprise AI Stack Identity Observability and Control","author":"CPI Staff","date":"October 12, 2026","format":false,"excerpt":"AI value disappears fast when access is messy, data is exposed, and nobody can see what the model is doing. Here is the practical stack that makes enterprise AI safe and useful.","rel":"","context":"In &quot;AI&quot;","block_context":{"text":"AI","link":"https:\/\/www.cloudproinc.com.au\/index.php\/category\/ai\/"},"img":{"alt_text":"","src":"\/wp-content\/uploads\/2026\/03\/post-27.png","width":350,"height":200,"srcset":"\/wp-content\/uploads\/2026\/03\/post-27.png 1x, \/wp-content\/uploads\/2026\/03\/post-27.png 1.5x, \/wp-content\/uploads\/2026\/03\/post-27.png 2x, \/wp-content\/uploads\/2026\/03\/post-27.png 3x, \/wp-content\/uploads\/2026\/03\/post-27.png 4x"},"classes":[]},{"id":57591,"url":"https:\/\/www.cloudproinc.com.au\/index.php\/2026\/10\/12\/designing-secure-ai-agent-infrastructure-on-azure\/","url_meta":{"origin":59662,"position":5},"title":"Designing Secure AI Agent Infrastructure on Azure","author":"CPI Staff","date":"October 12, 2026","format":false,"excerpt":"AI agents are moving from demos into production workflows. They can read documents, call APIs, query databases, create tickets, summarise customer history, and trigger business processes. That power creates a new infrastructure problem for Australian organisations: an AI agent is not just a chatbot. It is an application runtime, an\u2026","rel":"","context":"In &quot;AI Governance &amp; Risk Management&quot;","block_context":{"text":"AI Governance &amp; Risk Management","link":"https:\/\/www.cloudproinc.com.au\/index.php\/category\/ai-governance-risk-management\/"},"img":{"alt_text":"","src":"\/wp-content\/uploads\/2026\/05\/designing-secure-ai-agent-infrastructure-on-azure.png","width":350,"height":200,"srcset":"\/wp-content\/uploads\/2026\/05\/designing-secure-ai-agent-infrastructure-on-azure.png 1x, \/wp-content\/uploads\/2026\/05\/designing-secure-ai-agent-infrastructure-on-azure.png 1.5x, \/wp-content\/uploads\/2026\/05\/designing-secure-ai-agent-infrastructure-on-azure.png 2x, \/wp-content\/uploads\/2026\/05\/designing-secure-ai-agent-infrastructure-on-azure.png 3x, \/wp-content\/uploads\/2026\/05\/designing-secure-ai-agent-infrastructure-on-azure.png 4x"},"classes":[]}],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"\/wp-content\/uploads\/2026\/11\/observability-for-ai-agents-with-azure-monitor-and-opentelemetry.png","_links":{"self":[{"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/posts\/59662","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/comments?post=59662"}],"version-history":[{"count":2,"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/posts\/59662\/revisions"}],"predecessor-version":[{"id":59665,"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/posts\/59662\/revisions\/59665"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/media\/59664"}],"wp:attachment":[{"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/media?parent=59662"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/categories?post=59662"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cloudproinc.com.au\/index.php\/wp-json\/wp\/v2\/tags?post=59662"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}