It started with a familiar ambition

In one of my discussions with the leadership team of a retail organization, the conversation started with a fairly straightforward ambition.

“We want AI-powered customer service.”

The business leadership was looking at how AI could improve customer experience, reduce repetitive operational effort and create a more scalable customer service model.

It was a reasonable ambition. Today, it is relatively easy to demonstrate an AI assistant answering customer questions, retrieving information or generating a personalized response.

But from an Enterprise AI Architect’s perspective, I was interested in something beyond the demonstration. What would it actually take to make this work reliably inside the retailer’s existing business and technology landscape? And, equally important, what would make the investment worthwhile?

Rather than immediately discussing LLMs, Agentic AI frameworks or technology platforms, I wanted to understand the business outcome they were trying to achieve. That changed the direction of the conversation.

1. Understanding what the business really wanted

As we explored the service operations, I used a working scenario of approximately 15,000 customer contacts per week. Suppose nearly 60% of those interactions are routine enquiries involving order status, product availability, delivery schedules and return policies. That represents approximately 9,000 routine contacts every week.

For the leadership team, the opportunity was not simply to introduce a chatbot. It was to reduce repetitive service effort, improve response times and allow customer service professionals to spend more time resolving complex issues.

This is where I believe an AI Architect needs to establish alignment with the business. What is the current cost per contact? How many interactions are genuinely repetitive? Which enquiries can be resolved without human judgment? What is the impact on customer satisfaction? How would we measure whether the customer issue was actually resolved?

Because an AI assistant can answer a question without necessarily solving the customer’s problem. We can improve automation metrics and still disappoint customers.

Before progressing into the architecture, I framed the objective around three outcomes: improve genuine customer resolution, reduce the cost of routine interactions and create more capacity for complex human-led service. Everything else in the architecture would need to support those outcomes.

2. Decomposing the journey before selecting the model

Once the business problem became clearer, I started looking at the actual customer journey. Take a simple interaction:

“Where is my order? I placed it three days ago and need it before Friday.”

A customer sees one question. An AI Architect should see several different responsibilities: recognize intent, establish identity, identify the relevant order, obtain the current delivery status, evaluate delivery-risk context and communicate an accurate response. If information is unavailable or an exception exists, the system should know when to involve a human.

These responsibilities do not all need the same AI technology. Intent recognition is primarily a classification problem. For a retailer with perhaps 20–30 well-defined contact categories, a smaller discriminative model may be more efficient than repeatedly invoking a large generative model.

But I would not design a classifier that forces every request into one known category. It needs a confidence threshold and an explicit unknown-intent path. That is different from escalation for a distressed customer or sensitive complaint. One is triggered by uncertainty; the other by business risk.

Next comes information retrieval. For order status, I would use an authorized, read-only Order Management System (OMS) integration. For a return-policy enquiry, I might use Retrieval-Augmented Generation (RAG) against approved knowledge documents. For stock availability, I would query the appropriate inventory service.

The enterprise systems remain the sources of truth. The LLM does not become the system of record. Only after trustworthy information is retrieved should the generative model compose the customer-facing answer.

That gives us five core capabilities with distinct failure modes: intent classification, knowledge retrieval, response generation, escalation logic and output guardrails. This is why I prefer fit-for-purpose AI over using a frontier LLM for every step.

3. Turning the functional design into a production operating architecture

A proof of concept could be built with an LLM, a prompt and a few integrations. But a production implementation needs to behave predictably when information is missing, systems are unavailable, customers provide ambiguous requests or models produce unsupported answers.

This is where I would introduce an AI Harness, or controlled AI execution runtime. Think of it as the operating backbone coordinating each interaction. It maintains conversation state, orchestrates models and tools, applies execution policies, manages errors and decides whether the process should continue, clarify or escalate.

For the order-status enquiry, the harness establishes authenticated customer context, routes the question to classification, invokes the authorized OMS lookup and assembles verified evidence. The response model produces a clear answer, while independent checks validate facts, policy and risk before release.

If the OMS is unavailable, the model must not invent a delivery commitment. If the policy cannot be confirmed, it must not promise a refund. If retries or conversation attempts exceed limits, the harness should fail gracefully into escalation.

In the technical design, I would expect a workflow state machine or orchestration graph, model gateway, tool execution policies, versioned prompts, retry handling, timeouts, circuit breakers, token budgets and traceable events. I would also resist introducing multiple autonomous agents for predictable read-only lookups. Dependable execution matters more than maximum autonomy.

Where MCP belongs in the architecture

The Model Context Protocol (MCP) offers a standardized way for AI applications to interact with approved external tools and resources. A controlled interface could expose functions such as `get_order_status`, `check_inventory_availability`, `retrieve_return_policy` and `get_store_details`.

These tools may be served through MCP adapters while OMS, inventory, CRM and knowledge services continue to operate behind the enterprise’s established APIs and governance controls.

MCP is not a replacement for API management, identity controls or server-side authorization. Every request must still enforce authentication, customer-level access checks, least privilege, tool allowlists, schema validation and logging.

For the first phase, I would keep the AI boundary read-only. The service can retrieve and explain information, but cannot independently initiate payments, approve refunds, change orders or modify customer master data. Transactional authority can be assessed later as a separately governed capability.

4. A good prompt is not the same as good business context

One common implementation mistake is assuming that better prompts solve most reliability problems. Prompt engineering is valuable, but it cannot compensate for missing or contradictory business facts.

Consider a customer asking: “Can I return the television I purchased last week?” The correct answer could depend on product category, purchase date, seller, applicable policy version, order status, geography and promotional conditions.

That is a context engineering problem. Before inference, the platform needs to assemble the relevant authenticated customer context, active session state, trusted tool responses, applicable policy clauses, and required business rules. It should not send the entire CRM record, all prior messages and every policy document into the context window.

Selective context improves grounding, reduces privacy exposure and keeps token consumption and latency under control. Prompt engineering then defines how the model must use that context: brand voice, answer format, prohibited commitments, citations and abstention behaviour.

For production RAG, I would examine ingestion quality, document chunking, embeddings, hybrid search, metadata filters, re-ranking, source provenance, freshness and policy-version management. A semantically similar return-policy passage can still be the wrong policy for a particular geography, seller or product category.

If a policy changes, outdated passages must be invalidated or made ineligible. Retrieval should be measured for relevance and correctness, and the generated response should be assessed for grounding. RAG reduces dependence on model memory; it does not automatically eliminate hallucination.

Prompt templates and response contracts should be version-controlled and regression-tested. Instructions embedded inside customer messages or retrieved content must never override trusted application instructions; this is a core prompt-injection defence.

5. Designing memory for continuity without making memory the source of truth

Now imagine the customer continues: “I understand the delivery is delayed. If it arrives next week, can I still return it?” The system should understand the order being discussed instead of forcing the customer to repeat the details.

That is where short-term working memory helps. A managed session or state store should capture the active intent, order reference, recent turns, pending clarification, retrieved results and escalation state. Context-window management, expiration and summarization should be deliberate platform features.

When the customer returns days later, selective long-term memory may provide useful continuity, but only where there is a legitimate purpose, permission and retention basis.

The memory patterns I would distinguish

Semantic memory holds selected durable facts or preferences that remain useful across interactions. It should not be confused with the retailer’s governed policy knowledge base.

Episodic memory refers to a specific past interaction or business event, such as a damaged-item complaint, its escalation and documented resolution. A usable episode should include provenance, event reference, timestamp, outcome and retention status.

Procedural memory represents approved ways of performing recurring tasks. For enterprise use, I would retain these as version-controlled workflows and policies rather than allow an AI model to silently rewrite its operating procedures.

Short-term memory maintains the active conversation and workflow state. It is distinct from persistent customer records and should expire or compact according to defined rules.

The key boundary is that memory must never overrule an authoritative system. A conversation may remember that an order was expected on Friday, but current delivery status still comes from the OMS. Likewise, an older return-policy answer must not override a new approved policy.

The architecture needs memory write eligibility, provenance, retrieval quality, customer isolation, access control, consent where applicable, deletion, retention and audit. This gives continuity without uncontrolled personal data accumulation or stale decisions.

6. The part that matters most in production: when things go wrong

At this stage, I would shift the conversation toward the business consequences of AI failures. Imagine the assistant says “Your replacement will arrive tomorrow,” although the warehouse has not confirmed stock. That is an unsupported commitment, not simply an awkward answer.

If it retrieves an expired return policy, the issue is knowledge governance. If it discloses another customer’s order status, the issue is authorization. If the OMS fails during a peak sale and the assistant guesses, the issue is resilience and fallback design.

These failure modes require different controls. Prompt wording alone cannot fix them.

The AI harness needs rate limits, execution budgets, bounded retries, timeouts, circuit breakers and graceful fallback. The retrieval layer needs source validity, provenance and freshness. The security layer needs identity checks, least privilege, PII minimization, encryption and prompt-injection protection.

The model and validation layer must evaluate groundedness, factual consistency, response format and policy compliance. The escalation layer must recognize model uncertainty, high-risk content, customer distress and unresolved service exceptions. The observability layer must connect these events into one traceable execution path.

A model stating it is “95% confident” is not evidence that the answer is 95% likely to be correct. Thresholds need to be calibrated against empirical performance, retrieval quality, business rules and risk. High-impact decisions should not be released on model self-confidence alone.

I would also plan for peak loads, model/provider outages, vector retrieval latency, API failures, identity-provider errors and abandoned sessions. Production readiness means the service can degrade truthfully and safely, not merely respond quickly when everything works.

One integrated technical view

To make that practical for engineering, operations and security stakeholders, I consolidate the solution into one technical reference: omnichannel interaction, identity/session/context, harness orchestration, intent classification, context and memory services, governed retrieval, LLM response generation, validation and handoff. Below them sit approved MCP/API interfaces and enterprise systems of record, with security, LLMOps, monitoring and platform services across the entire stack.

7 . Returning to the CXO question: how do we prove business value?

After working through the architecture, I would bring the discussion back to measurable outcomes. That is where technology choices become accountable to the original business ambition.

The illustrative baseline is 15,000 weekly contacts, with 9,000 identified as potentially routine. A 60% overall containment ambition in 90 days may sound attractive, but if just 60% of all contacts are in scope, it would imply resolving almost all routine contacts automatically. That should be treated as an aspirational hypothesis, not an assumed outcome.

For comparison, if the platform genuinely resolves 70% of those 9,000 routine contacts, that is 6,300 AI-resolved contacts weekly, equivalent to 42% of all incoming contacts. Even then, containment alone is insufficient: a customer can remain with a bot and still leave without an answer.

Business and engineering success scorecard

BUSINESS OUTCOME

The architecture needs to support these measures directly. Higher containment depends on intent coverage and reliable retrieval. Better CSAT depends on response quality, relevant memory and seamless handoff. Lower costs require model routing, controlled token use, efficient tool calls and reduced manual rework. Stronger trust depends on grounding, policies, traceability and appropriate human approval.

I would measure total cost per genuinely resolved customer interaction, not just inference tokens. The full unit economics include model execution, retrieval, embeddings refresh, orchestration, integration, observability, knowledge operations, human escalation and corrections after AI errors.

8. How I would take the design into production

By this point, the business request has become an enterprise solution blueprint. But I would still resist launching every capability immediately. A phased approach gives the business evidence at each decision point.

Discovery and readiness assessment

Validate actual contact volumes, intent mix, service journeys, CRM/OMS readiness, API access, knowledge ownership, customer identification, regulatory obligations, agent capacity and existing support costs. Resolve critical assumptions before finalizing scope.

Controlled pilot

Start with a few well-understood intents, such as order tracking, product availability and return policies. Use read-only integrations, governed knowledge, deterministic orchestration, grounded generation, independent guardrails and human fallback. Avoid speculative autonomous actions.

Evaluation and shadow mode

Build representative, privacy-safe evaluation cases covering normal enquiries, multi-intent messages, missing records, policy exceptions, emotional complaints, adversarial inputs and out-of-scope requests. Compare AI recommendations with human resolutions before exposing the platform broadly.

Evaluate the classifier on precision, recall and unknown-intent handling; retrieval on relevance, provenance and freshness; generation on groundedness, policy compliance and communication quality; escalation on missed versus unnecessary transfers. Regression-test changes to prompts, models, retrieval indexes and policy rules.

Controlled rollout

Release gradually using feature flags, limited customer segments, observability, incident runbooks, support ownership and fast rollback. Validate actual resolution, CSAT, latency, system stability, recontact rates and unit economics.

Scale with evidence

Expand eligible intents only after demonstrating reliable performance. Higher-autonomy capabilities such as refund initiation or order modification should have separate transaction authorization, policy approval, idempotency, audit and, where appropriate, human checkpoints. Autonomy must be earned through demonstrated reliability, not assumed because tool calling is possible.

Operationally, the platform needs continuous monitoring. When a customer receives an incorrect answer, the team should be able to reconstruct the detected intent, context supplied, knowledge versions, tool calls, model/prompt versions, policy decisions and handoff outcome. “The AI generated it” is not a root-cause analysis.

9. What this discussion reinforced for me

The most useful part of this exercise was watching the conversation evolve. We started with the ambition to introduce AI into customer service. The value emerged when we moved from a technology label to the real customer journey, the operating problem, the enterprise controls and the outcomes worth measuring.

We separated classification, retrieval, generation, escalation and guardrails; established where small models, structured APIs, RAG and LLMs genuinely belong; introduced a controlled AI harness; considered MCP for governed tool access; designed context and memory deliberately; and treated hallucination, security, stale knowledge, uncertainty and operational outages as distinct architecture concerns.

More importantly, we connected those design choices to customer experience, service quality, operational efficiency, risk and total cost per resolved interaction.

That is where I believe an Enterprise AI Architect should add value. Our role is not simply to recommend an LLM or introduce the newest agentic framework. It is to help the organization decide where intelligence belongs, what needs deterministic control, where human judgment remains essential, and how the entire service can be operated and measured.

Sometimes that means introducing AI. Sometimes it means using simpler classification, retrieval or workflow capabilities. Sometimes it means keeping a human decision in the process. And sometimes it means telling the business that an integration or data dependency must be resolved first.

A successful AI demonstration proves that a technology can respond. A successful enterprise AI architecture establishes how the business can depend on that response, operate it safely and measure the value it creates.

That is the real transition from AI ambition to AI execution.