Boost Accuracy with Azure AI Groundedness

In this blog post Boost Accuracy with Azure AI Groundedness for Cloud Apps we will explore how to make your Azure AI solutions more accurate, verifiable, and production-ready by grounding them in your data and systems.

Generative AI is powerful, but it has a habit of sounding confident even when it is wrong. Boost Accuracy with Azure AI Groundedness for Cloud Apps focuses on how to reduce these hallucinations by tying model outputs tightly to your data, APIs, and business rules using Azure AI Studio and related services.

What is groundedness and why it matters

Groundedness means that an AI system’s answers are clearly supported by reliable sources, such as your documents, databases, or APIs. A grounded response is:

• Traceable – you can see where the answer came from
• Verifiable – you can check the sources and confirm the facts
• Repeatable – the same question produces a similar answer when the data has not changed

Without groundedness, AI assistants will sometimes invent facts, misquote policies, or provide outdated information. For technical teams and technical managers, this is more than an annoyance – it can create compliance issues, bad customer experiences, and support overhead.

Azure AI tackles this using a combination of prompt design, retrieval, and evaluation tools that make it easier to keep models aligned with your data and to measure how well they are doing.

The technology behind Azure AI Groundedness

Under the hood, groundedness in Azure AI is driven by three main ideas:

1. Retrieval-Augmented Generation (RAG)

RAG is the architecture where a large language model (LLM) first retrieves relevant context from a data source, then uses that context to generate an answer. In Azure AI, this typically looks like:

1. User asks a question to your app.
2. Your app sends the question to a vector index (e.g., Azure AI Search) built over your content.
3. Top matching passages, documents, or knowledge base entries are retrieved.
4. The LLM receives both the user question and the retrieved content as context.
5. The LLM answers, instructed to rely on that context and cite it where possible.

This keeps the model from guessing beyond what your data supports, and allows you to update answers just by updating the index.

2. System prompts and safety configuration

Azure OpenAI and Azure AI Studio let you define a system message that controls how the model behaves. You can instruct the model to:

• Use only the provided sources
• Refuse to answer when information is missing
• Include citations and reference IDs in replies

Combined with Azure’s safety features (content filters, rate limits, and logging), this shapes the model into a reliable, policy-aware assistant rather than a free-form chatbot.

3. Groundedness evaluation

Azure AI also provides evaluation tools to test how grounded your system is. These tools can use another model to score your AI’s responses on dimensions such as:

• Groundedness – is the answer supported by the provided sources?
• Relevance – how well does it address the user’s question?
• Coherence – is the answer clear and logically structured?

Instead of guessing whether your AI is “good enough,” you can run batches of test queries, measure groundedness, and improve iteratively.

How groundedness fits into Azure solutions

In a typical Azure-based AI application, groundedness weaves through several services:

• Azure AI Studio – design, prototype, and evaluate your AI flow
• Azure OpenAI Service – provides GPT-family models and system prompt configuration
• Azure AI Search – stores your content as a hybrid or vector index for retrieval
• Azure Storage or Cosmos DB – keeps documents and structured data
• Application code (e.g., .NET, Python, Node) – orchestrates user requests, retrieval, and responses

CloudPro and similar Azure partners often use this pattern to build helpdesk copilots, policy assistants, and internal knowledge tools that must be right most of the time and provably so.

Design a grounded AI assistant step by step

The steps below walk through a high-level pattern you can adapt for your environment.

Step 1 – Identify critical use cases and risk

Start by choosing where accuracy matters most. Common cases:

• Employee policy Q&A (HR, IT, compliance)
• Customer-facing support and troubleshooting
• Internal engineering knowledge bases and runbooks

For each, list:

• What must be correct (numbers, dates, policy clauses)
• What can be approximate (summaries, explanations)
• What the AI must never invent (legal terms, pricing, security guidance)

Step 2 – Prepare content for retrieval

Groundedness is only as good as the content you provide. Good practices:

• Store source documents in Azure Blob Storage or SharePoint.
• Split large documents into smaller chunks (e.g., 500–1500 tokens) with clear headings.
• Enrich documents with metadata (department, region, effective date, product line).
• Index them with Azure AI Search using vector or hybrid search.

This allows your AI to retrieve focused, relevant passages rather than entire PDFs or wiki pages.

Step 3 – Build a retrieval-augmented flow

At runtime, your app should follow a consistent pattern:

1. Receive the user question.
2. Call Azure AI Search with a semantic or vector query.
3. Select top passages and optionally post-filter by metadata.
4. Construct a prompt that includes system instructions, user question, and retrieved sources.
5. Send the prompt to Azure OpenAI with instructions to stay within the context.

Here is a simplified Python example (structure, not production-ready code):

from azure.search.documents import SearchClient
from azure.identity import DefaultAzureCredential
from openai import AzureOpenAI

search_client = SearchClient(
    endpoint="https://<your-search>.search.windows.net",
    index_name="knowledge-index",
    credential=DefaultAzureCredential()
)

openai_client = AzureOpenAI(
    azure_endpoint="https://<your-openai>.openai.azure.com/",
    api_version="2024-02-15-preview",
    api_key="<your-key>"
)

SYSTEM_PROMPT = """You are a helpful assistant for our organisation.
Use only the provided context to answer. If the answer is not in the
context, say you do not know. Always reference the document IDs you used.
"""

user_question = "What is our policy for working from home in NSW?"

# 1. Retrieve context
results = search_client.search(
    search_text=user_question,
    top=5,
    query_type="semantic"
)

context_snippets = []
for doc in results:
    context_snippets.append(f"[docId={doc['id']}]\n{doc['content']}")

context = "\n\n".join(context_snippets)

# 2. Call Azure OpenAI
response = openai_client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_question},
        {"role": "system", "content": f"Context:\n{context}"}
    ],
    temperature=0.1
)

answer = response.choices[0].message.content
print(answer)

Key groundedness features in this pattern:

• The model gets explicit context from your index.
• The system prompt instructs the model to refuse to guess.
• Document IDs are passed so the model can cite sources.

Step 4 – Use Azure AI evaluation to measure groundedness

After your prototype works, you need to quantify its accuracy. Azure AI Studio supports:

• Manual playground testing – interact with your flow and inspect responses.
• Automated evaluations – run test sets of queries and let a model score responses.

In an automated evaluation, you provide:

• A set of test questions (manually created or sampled from logs)
• Reference documents or expected answers
• The model’s responses generated via your pipeline

Azure then uses an evaluator model to score groundedness, coherence, and relevance. The groundedness score answers: “Did the AI stay within the evidence we gave it?”

Step 5 – Close the loop and improve

Once you can measure groundedness, you can improve it systematically:

• Tighten prompts – be clearer that the model must not infer beyond the context.
• Improve retrieval – adjust search parameters, chunk sizes, and metadata filters.
• Curate test sets – add examples where the system previously hallucinated.
• Introduce guardrails – for high-risk questions (legal, financial, safety), escalate to a human or provide links to official documents instead of natural-language synthesis.

Practical tips to boost groundedness in Azure

1. Turn temperature down

Creative variance is not your friend when you care about accuracy. For grounded Q&A, use low temperature values (e.g., 0–0.2). This makes the model more deterministic and closer to the evidence.

2. Ask for quotes and citations

In your system prompt, ask the model to quote or reference the relevant passage and include document IDs or URLs. For example:

When you answer, list the documentId values you used at the end
under a heading "Sources". If multiple documents conflict, say so
explicitly and do not guess.

This creates traceability and builds user trust.

3. Separate facts from opinions

If your assistant sometimes needs to explain or hypothesise, clearly separate grounded facts from model opinions. For example:

First, summarise what the documents say using a heading "What the
policy says". Then, optionally provide your reasoning or suggestions
under "Additional guidance".

This makes it obvious where the AI is extrapolating beyond strict source material.

4. Handle “no answer” gracefully

A well-grounded system will sometimes say, “I don’t know based on the provided information.” Instead of treating that as a failure:

• Offer a link to a relevant search page or contact channel.
• Log the question for content gap analysis.
• Consider adding that topic to your knowledge base or FAQ.

5. Log prompts, context, and outputs

For production systems, store:

• User queries
• Retrieved documents or snippets
• Final prompts sent to the model
• Model responses and evaluation scores

This trace is invaluable for debugging and compliance, and it lets you replay real interactions when you refine your retrieval or prompts.

Where CloudPro-style partners can help

Groundedness is not just a checkbox; it is an ongoing practice across architecture, data, and operations. Organisations often need help with:

• Designing secure, compliant Azure architectures for AI workloads
• Preparing content, indexing, and metadata for high-quality retrieval
• Implementing evaluation pipelines and dashboards for groundedness
• Integrating AI assistants into existing apps, portals, and workflows

Working with an Azure-focused team that understands both the technical stack and the business context can dramatically shorten time-to-value and reduce risk.

Next steps

To move forward with groundedness in your own Azure environment:

1. Pick one high-impact, low-risk use case (e.g., internal IT FAQ).
2. Index a curated set of documents with Azure AI Search.
3. Build a simple RAG-based prototype in Azure AI Studio or code.
4. Add system prompts that enforce groundedness and citations.
5. Run evaluations, review groundedness scores, and iterate.

Done well, Azure AI Groundedness turns generative AI from a clever demo into a dependable part of your cloud applications – one that your teams can trust, verify, and steadily improve.

• Boosting Intune Machine Policy Updates from 8 Hours to 30 Minutes
• What is Supervised Fine-Tuning (SFT)
• What Are Weights in AI Models
• How to Upload Files to OpenAI Storage with C#
• Document Definition in LangChain