Customer FAQ agent deployed on website and inside the support team
An Azure AI Foundry agent grounded in Dataverse and ERP data, deployed simultaneously as a customer-facing website widget and an internal Teams knowledge tool for the support team. Two channels, two temperature profiles, one data layer.
See it working
Demo video — coming soon
The same questions, answered 80 times a week by people who have better things to do
For a B2C company with 40,000 active customers and a 7-person support team, the daily ticket queue had a predictable shape: 60–70% of incoming contacts were repeat questions. Order status. Return eligibility. Account changes. Product compatibility. The answers existed in the ERP, in the CRM, in a Dataverse knowledge base that the support team maintained but customers couldn't access.
The support team was acting as a translation layer between customers and systems. Each ticket took 8–12 minutes on average. The team was managing 400+ contacts per week. That's 55–80 hours of handling time on questions that, in principle, had known answers.
The second problem was internal. Support agents were experienced but the product range was broad. When a complex or unusual query came in, finding the relevant knowledge article, cross-referencing the ERP for order context, and checking case history took time often longer than it should, especially for newer agents.
Two separate problems, both solvable with the same data layer and the same underlying agent architecture.
One agent, two channels, two temperature profiles
The same Azure AI Foundry agent is deployed twice as a website widget for customers, and as a Teams bot for the support team. Both channels are grounded in the same Dataverse knowledge base and ERP data. The key difference is the configuration: the customer channel is tuned for consistency and safety, the internal channel for depth and synthesis.
Query intake and intent classification
Both channels classify the incoming query into one of five intent buckets: order/account lookup, returns and refunds, product/compatibility question, billing query, or general FAQ. This determines which data sources are retrieved and which response template is applied. Unclassifiable queries are routed to a human immediately rather than attempting a low-confidence answer.
Knowledge retrieval from Dataverse
Azure AI Search performs semantic retrieval across the Dataverse knowledge base product documentation, policy articles, FAQs, and resolved case summaries. Results are ranked by relevance and recency. Only articles marked as 'published and verified' in Dataverse are used as grounding sources for the customer channel; the internal channel also surfaces draft and pending-review articles, flagging their status to the agent.
Live data lookup from ERP and Dataverse CRM
For order, account, and billing queries, the agent makes a real-time lookup to Dynamics 365 Finance & Operations (order status, returns, billing history, inventory) and Dataverse (customer record, case history, entitlements). On the customer channel, the agent only accesses data for the authenticated customer's own account. On the internal channel, support agents can query any account by customer ID.
Response generation temperature-tuned per channel
The customer channel uses temperature=0.1, top_p=0.9: responses are factual, concise, and traceable to source. The model is instructed to say 'I don't have that information let me connect you with the team' rather than attempt a low-confidence answer. The internal channel uses temperature=0.4, top_p=0.95: responses are richer and more synthesised, giving support agents a fuller picture including related cases, edge-case notes, and a suggested reply draft for the customer.
Escalation, citation, and handoff
Every customer-channel response includes the source article title and a link to the full knowledge article. If the agent can't resolve the query with high confidence (confidence score below threshold), it offers a one-click escalation to a human agent, pre-populating the ticket with the query summary and retrieved context. Internal-channel responses include a 'suggested next action' send the draft reply, escalate to tier 2, or flag for knowledge base update.
50 test cases across both channels and five intent types
25 test cases per channel, stratified across all five intent categories. Customer-channel ground truth established from historical resolved tickets rated by the support team lead. Internal-channel ground truth established from senior agent reviews of the synthesised answers and draft replies.
| Metric | Score | What it measures |
|---|---|---|
| Customer channel - groundedness | 93% | Answers traceable to Dataverse knowledge articles or ERP data |
| Customer channel task completion | 89% | Query resolved without escalation to human agent |
| Internal channel answer quality | 91% | Senior agent rated synthesised answer as useful or better |
| Internal channel draft reply usability | 78% | Draft reply sent to customer with minor or no edits |
| Citation accuracy | 96% | Cited article correctly matched to answer content |
| Intent classification accuracy | 94% | Correct intent bucket assigned across all 50 test cases |
| Temperature: customer channel | 0.1 | Factual, consistent, low-variance responses safety over depth |
| Temperature: internal channel | 0.4 | Richer synthesis for agents acceptable variance for trained users |
What it doesn't handle well
The customer channel's conservative temperature (0.1) produces occasionally clipped responses for nuanced multi-part questions when a query spans two intent categories (e.g. order status + return eligibility), the agent sometimes answers only the primary intent and requires a follow-up. We're iterating on the multi-intent classification to address this. ERP data lookups add 2.4 seconds of latency for order-related queries, which is acceptable for both channels but noticeable on slower connections. The internal channel's higher temperature (0.4) occasionally produces synthesised answers that sound confident for genuinely ambiguous cases support agents are trained to verify these before sending. The agent also cannot action requests in the ERP; it reads and reports, but order modifications, refund processing, and account changes still require a human to complete the action in Dynamics 365.
What it's built on
Common questions
Why different temperature settings for the two channels?
Customer-facing responses need to be consistent and verifiable if the agent says 'your order ships in 2 days', that needs to be true and traceable to a specific ERP record. Temperature 0.1 keeps outputs close to the retrieved facts. Internal agents are trained professionals who can evaluate a synthesised answer and decide whether to use it. Temperature 0.4 gives them richer context and more useful draft replies, at the cost of slightly higher variance that they're equipped to assess.
How does the agent handle queries about orders it can't find?
If the ERP lookup returns no matching order for the customer's account, the agent tells the customer it can't find a matching order and offers immediate escalation. It doesn't guess or invent order details. The escalation ticket is pre-populated with the customer's query and the failed lookup parameters so the support agent can investigate without asking the customer to repeat themselves.
Can the internal channel agent raise cases or update records in Dataverse?
Currently, no the agent is read-only against both Dataverse and the ERP. It can suggest a case update (e.g. 'flag this article for review') but the action is taken by the agent in the Dataverse UI. A write-capable version is scoped for the next phase of the project.
How is the knowledge base kept current?
The Dataverse knowledge base is maintained by the support team as part of their existing workflow articles are updated when policies change or new products are released. The agent reads from Dataverse at query time (not from a cache), so updates are reflected immediately without any reindexing step. The internal channel also flags queries that don't match any knowledge article, which feeds back into the team's content gap backlog.
The same architecture works for any business with a knowledge base and live transactional data.
Book a 25-minute call. We'll map your support volume, your data sources, and tell you what both channels would look like for your operation.