In this blog post How Foundry Agent Optimizer Improves AI Agent Quality at Scale we will explain how Microsoft’s optimisation technology can improve AI agents without your team repeatedly rewriting prompts by hand.
Many organisations can build an AI agent that performs well in a controlled demonstration. The problems appear later, when employees phrase requests differently, business rules change or the agent must choose between several tools.
Foundry Agent Optimizer addresses this problem by automatically creating and testing alternative agent configurations. Instead of relying on someone’s opinion about whether a rewritten prompt looks better, it measures each option against representative business tasks and defined quality standards.
What is Foundry Agent Optimizer?
Foundry Agent Optimizer is part of Microsoft Foundry Agent Service, Microsoft’s managed Azure platform for building and operating AI agents. An AI agent is software that can interpret a request, use approved information and tools, and take steps towards completing a task.
The optimizer starts with your existing agent as a baseline. It then produces alternative configurations, runs them through the same test cases and ranks the results.
Depending on the agent type, it can improve:
- Instructions that define the agent’s role, boundaries and expected behaviour.
- Skills that give a hosted agent reusable procedures for particular tasks.
- Tool descriptions that explain when and how the agent should call business systems.
- Model selection so you can compare whether a different AI model provides a better balance of quality, speed and cost.
This is broader than conventional prompt tuning. It examines several parts of the agent’s configuration rather than assuming every failure can be fixed by adding another paragraph to the prompt.
How the technology works
The optimizer uses an evaluation-driven improvement loop. In plain English, it tests the current agent, proposes changes, retests the changed versions and shows you which option performed best.
1. Establish a baseline
The baseline is the current version of your agent. Its instructions, model, tools and relevant skills become the starting point against which all proposed improvements are compared.
This matters because an impressive score is not enough on its own. Decision-makers need to know whether the proposed version is genuinely better than what is already running.
2. Test representative business tasks
The optimizer needs a dataset, which is simply a structured collection of test requests. These should reflect the work employees or customers will actually ask the agent to perform.
For a service desk agent, that could include password problems, device requests, access approvals and unusual requests that must be escalated. Good datasets include common work, difficult edge cases and situations where the agent should refuse to act.
3. Score the results using evaluators
Evaluators are automated scoring rules. They can assess whether the agent followed the task, used approved information, selected the correct tool or produced an acceptable answer.
The quality of these rules is critical. If you tell the optimizer to reward friendly answers but ignore factual accuracy, it may produce a more polished agent that is no more reliable.
Our guide to evaluating groundedness, tool accuracy and task completion explains the business measures worth testing before an agent reaches production.
4. Generate and compare candidates
The optimizer uses an optimisation model to analyse weaknesses and propose alternative configurations. Each candidate is then tested against the same dataset and scoring criteria as the baseline.
Teams can compare overall scores and inspect performance by evaluator. This helps reveal trade-offs, such as a candidate that completes more tasks but takes longer, costs more or breaks an important compliance rule.
5. Review before deployment
The winning candidate is not silently pushed into production. Your team can review the changes, examine the test results and decide whether to promote or deploy it.
That human approval step is important. The technology reduces repetitive prompt editing, but it does not remove accountability for business rules, security or customer outcomes.
Prompt agents and hosted agents work differently
For prompt agents, optimisation can be started through the Foundry portal without changing application code. The optimizer can improve instructions, function-calling tool descriptions and model selection.
Hosted agents require some initial configuration so the optimizer can access the relevant instructions, skills, tools and model setting. They offer more control for complex applications, but they also require stronger engineering and operational practices.
If you are still selecting an approach, our comparison of Foundry prompt agents and hosted agents explains the practical differences.
A realistic business scenario
Consider a 180-person professional services company building an internal policy agent. Early testing looks promising, but employees soon discover inconsistent behaviour.
The agent answers standard leave questions correctly, yet struggles with contractors, parental leave and policies that vary by location. One developer spends several hours each week adding more instructions, fixing one problem while sometimes creating another.
With Agent Optimizer, the company could assemble representative questions from approved, de-identified test cases. It could score answers for policy accuracy, correct source use, appropriate escalation and clarity.
The optimizer might find that clearer instructions improve difficult cases, while a different model achieves similar quality at a lower cost. The business outcome is not merely a “better prompt”. It is fewer incorrect answers, less developer time spent on trial and error, and stronger evidence that the agent is ready for wider use.
Where the business value comes from
- Lower development effort: Teams spend less time making small prompt changes and manually rerunning tests.
- More consistent quality: Every candidate is assessed against the same tasks and business rules.
- Better cost control: Model comparisons can show whether an expensive model produces enough additional value to justify its price.
- Safer releases: Proposed changes can be reviewed before deployment rather than introduced directly into production.
- Clearer governance: Recorded datasets, evaluation criteria and results provide evidence for internal risk reviews.
Optimisation itself consumes AI model capacity, so it is not free. Foundry can provide estimates for supported workflows, but organisations should still set budgets and monitor actual Azure usage.
Cost also depends on the wider agent design. For example, Foundry Toolboxes can reduce token costs by showing an agent only the tools relevant to each request.
What Agent Optimizer cannot fix
Agent Optimizer cannot compensate for poor business processes, unreliable source information or vague success criteria. If your organisation cannot agree on the correct outcome, the optimizer has no dependable target.
It also does not turn a simple prompt into a fully controlled business workflow. Tasks involving approvals, recovery steps and audit requirements may need structured orchestration, as explained in why prompt-only agents break down in complex business work.
Tool testing requires particular care. For prompt agents, the optimizer can improve descriptions for function-calling tools, but it cannot execute those client-side tools while optimising their descriptions. End-to-end testing is still required.
A small view of the technical process
For a hosted agent, a technical team can generate evaluation assets and start an optimisation run with Azure Developer CLI commands such as:
azd ai agent eval generate
azd ai agent optimize --config eval.yamlThe first command creates an initial test dataset and scoring rules. The second evaluates the baseline, generates candidates and ranks the results according to the chosen configuration.
Automatically generated tests are a starting point, not a finished assurance program. Business owners should review them to confirm that they represent real risks, policies and customer expectations.
Governance still matters in Australia
Evaluation datasets and agent traces may contain employee, customer or commercially sensitive information. Australian organisations should de-identify test data where possible, limit access and confirm how information is stored and retained.
The Essential Eight, the Australian Government’s cybersecurity framework that many organisations use as a security baseline, does not certify an AI agent’s answer quality. However, its controls remain relevant to the devices, administrator accounts and systems surrounding the agent.
Before optimisation begins, define who can change evaluations, approve candidates and deploy new versions. This prevents a technically higher-scoring agent from bypassing privacy, security or operational requirements.
Start with measurement rather than prompt rewriting
Foundry Agent Optimizer changes the improvement process from repeated guesswork into a measurable cycle. It can reduce manual tuning, but its real value comes from connecting agent changes to business outcomes such as accuracy, completion rates, cost and risk.
CloudProInc combines more than 20 years of enterprise IT experience with practical expertise across Microsoft Azure, Foundry, OpenAI, Claude and cloud security. As a Melbourne-based Microsoft Partner and Wiz Security Integrator, we help organisations build evaluation and governance into agents before they become business-critical.
If you are unsure whether your AI agent is genuinely improving or simply producing more polished demos, we are happy to review the current setup and testing approach with you—no strings attached.