Production-Ready AI Agents on Microsoft Foundry Go-Live Checks

In this blog post Production-Ready AI Agents on Microsoft Foundry Go-Live Checks we will explain what Australian CIOs should verify before an AI agent is allowed to access company data, use business systems or act on behalf of employees.

The core idea is straightforward. Microsoft Foundry provides a managed environment for building, running and monitoring AI agents, but the platform cannot decide how much authority your agent should have, what business risks are acceptable or when a human must step in.

An AI agent combines an AI model with instructions, company information and tools. Unlike a basic chatbot that only answers questions, an agent may search SharePoint, update a customer record, prepare a quotation, open a support ticket or trigger a workflow.

That ability to act is where much of the business value comes from. It is also why a successful demonstration is not evidence that an agent is ready for production.

The gap between a working pilot and a business service

Pilots usually run with friendly users, clean data and close supervision. Production introduces incomplete requests, incorrect records, unavailable systems, malicious instructions and employees who assume the answer must be right because it came from AI.

Microsoft Foundry Agent Service handles much of the underlying infrastructure. It can provide managed agent hosting, identity, connections to tools, evaluation and operational monitoring, depending on the services and agent design selected.

Your organisation still owns the outcome. Before approving go-live, CIOs need evidence across seven areas.

1. Define the outcome and the agent’s authority

Start with the business problem, not the model. Write down what successful work looks like, who owns the process and how the result will be measured.

An accounts agent, for example, might reduce the time required to match invoices with purchase orders. Its success measure should be cost per completed match, processing time and exception rate—not the number of conversations it handles.

Then draw a firm line between what the agent can recommend and what it can do. An agent may draft a supplier email, while a person approves sending it. It may prepare a refund, while a manager authorises payment.

For higher-risk actions, require human approval and provide a clear way to disable the agent. If nobody can explain how to stop it quickly, it is not ready to go live.

2. Give the agent only the access it needs

Permissions are one of the biggest production risks. A pilot is often connected using a developer’s account or a broadly privileged service account because it is quick and convenient.

That approach can give the production agent access to far more information than its job requires. If the agent is confused by an instruction or manipulated through a document, those excessive permissions increase the potential damage.

Microsoft Foundry can use Microsoft Entra identities, which are controlled digital identities for users, applications and agents. Each production agent should have its own identity wherever practical, with access limited to the specific systems, records and actions it requires.

• Remove shared passwords and access keys from prompts and application code.
• Separate development, testing and production identities.
• Review every tool the agent can call and every action that tool permits.
• Use the signed-in employee’s access when the agent should not see more than that employee.
• Log sensitive actions so the business can identify what happened and why.

Our earlier guide to production-ready agents on Microsoft Agent Framework covers the broader governance checklist. At go-live, permissions should be treated as a formal release gate rather than a configuration detail.

3. Know where company information goes

An agent may process customer details, employee information, contracts, financial records and commercially sensitive material. CIOs should be able to map where that information enters the system, where it is stored and who can retrieve it.

Do not examine only the model. Check connected search indexes, file storage, conversation history, agent memory, operational logs, evaluation datasets and backups.

Tracing is particularly important. Tracing records the steps an agent took so teams can diagnose errors, but those records may also contain prompts, outputs and data returned by business systems.

Before enabling detailed tracing, set access permissions, retention periods and rules for masking sensitive information. The Office of the Australian Information Commissioner recommends assessing privacy risks when AI handles personal information, so a privacy impact assessment may be appropriate before launch.

4. Test difficult behaviour rather than polished demonstrations

A production test should represent messy business conditions. Build a test set using anonymised examples of real requests, including incomplete information, conflicting instructions and unusual exceptions.

Measure whether the agent completes the correct task, uses the right source, selects the correct tool and stops when approval is required. Test attempts to trick the agent through documents, emails and web content as well as direct user prompts.

• What happens when a connected system is unavailable?
• Can a document instruct the agent to ignore company policy?
• Does it invent missing customer or financial information?
• Will it retry an action and accidentally create a duplicate transaction?
• Can unauthorised users retrieve information through indirect questions?

Microsoft Foundry includes evaluation tools for checking answer quality, safety, factual grounding and tool use. Set minimum passing scores and test them again whenever the model, instructions, tools or business data change.

Also confirm that every Foundry feature on which the production design depends is generally available and supported in the required Azure region. Some newer capabilities may still have preview conditions or different service limits.

5. Design for failure before it happens

AI agents depend on models, networks, identity services, data sources and external applications. Any of these components can become slow, unavailable or rate-limited.

Decide what the user sees when the agent cannot complete its task. For an internal knowledge assistant, a clear error may be enough. For an agent involved in customer orders or employee onboarding, you may need a human handover and a queue of unfinished work.

Test timeouts, retries, duplicate prevention, service quotas and rollback procedures under realistic load. Our guide to hosting AI agents on Microsoft Foundry explains these operational choices in more detail.

6. Monitor business quality as well as system health

A dashboard showing that the agent is online does not tell you whether it is helping the business. Production monitoring should combine technical measures with business outcomes.

Track response time, errors and usage, but also monitor successful task completion, human corrections, escalations, rejected actions and user complaints. Watch for gradual quality changes when company information, user behaviour or model versions change.

Assign a named business owner who reviews these results. IT can operate the service, but the process owner must decide whether the agent’s work remains accurate and useful.

7. Control cost per completed task

Agent costs can be less predictable than chatbot costs because an agent may perform several model calls, searches and tool actions for one request. Retries, long conversations and unnecessary background steps can quietly increase the bill.

Include model usage, search services, data storage, monitoring logs, networking and supporting Azure resources in the forecast. Set budgets and alerts, then report the cost per successful business outcome.

An agent costing $2 per completed task may be valuable if it replaces 20 minutes of manual work. An agent costing 20 cents is still wasteful if employees regularly redo its output.

For broader scaling decisions, see what CIOs should assess before scaling Azure AI agents.

Australian security requirements are moving

The Essential Eight is the Australian government’s widely adopted cybersecurity framework for reducing common attacks. Organisations with contractual or regulatory Essential Eight requirements should continue measuring and documenting their maturity.

However, the Australian Signals Directorate launched ASD Essentials on 15 October 2026 as an evolution of that approach. It extends the focus across enterprise IT, cloud environments and AI-related risks, making it relevant to Microsoft Foundry deployments.

Your agent should therefore be assessed alongside existing controls for identity, patching, application control, backups, privileged access and incident response. Microsoft Intune, which manages and secures company devices, and Microsoft Defender, which detects threats across identities, endpoints and cloud services, may support parts of this control environment.

For cloud-level visibility, platforms such as Wiz can help identify exposed services, excessive permissions and risky connections. As a Microsoft Partner and Wiz Security Integrator, CloudProInc commonly looks for gaps between these systems rather than reviewing the AI project in isolation.

A practical go-live scenario

Consider a 180-person professional services firm introducing an agent to prepare supplier responses and create CRM records. The pilot produced strong drafts and saved employees several minutes per request.

A production review found that the agent could search every SharePoint site, traces could record personal information, retries could create duplicate CRM records and no spending alert had been configured. None of those issues appeared in the polished pilot demonstration.

The team restricted the agent’s identity, added approval for external messages, masked sensitive trace data, introduced duplicate checks and set cost alerts. The agent then launched with clearer accountability, lower risk and predictable operating costs.

The CIO’s final go-live questions

• Is there a measurable business outcome and a named owner?
• Are the agent’s actions and approval boundaries documented?
• Does it have the minimum access required?
• Have privacy, retention and data location been reviewed?
• Has it passed realistic security and quality testing?
• Can the business recover from failures and duplicate actions?
• Are quality, security and cost monitored in production?
• Can the agent be disabled quickly without disrupting the wider business?

Move to production with evidence, not optimism

Microsoft Foundry provides a strong foundation for enterprise AI agents, but production readiness comes from disciplined decisions around authority, data, testing, resilience and cost. The goal is not to remove every risk; it is to make each risk visible, controlled and owned.

CloudProInc brings more than 20 years of enterprise IT experience across Azure, Microsoft 365, Microsoft Foundry, OpenAI, Claude, Defender and Wiz. We are based in Melbourne and work hands-on with organisations across Australia and internationally.

If your AI agent looks impressive but you are not certain it is ready to touch real data or business processes, we are happy to review the design and go-live controls—no strings attached.