Benchmarking an AI Shopping Assistant Across 5 Critical Capability Dimensions

Key Takeaways:

• The Principle: A conversational shopping assistant is not one product, it is five overlapping conversations (clarify, close, empathize, call tools, speak the language). Optimizing for a global "quality" score hides regressions in the dimensions that actually move revenue.
• The Reality Check: When a global eCommerce technology company brought us their production shopping agent, four of its five capabilities were running at roughly a quarter of where they needed to be.
• The Data: Bain & Company projects fully agentic commerce in the U.S. could reach $300-$500 billion in revenues by 2030, representing up to one-quarter of total e-commerce.
• The Recommendation: Stop fine-tuning your shopping assistant on intuition. Build a multi-dimensional evaluation framework first, then let it dictate the data strategy, the fine-tuning method, and the path to production. This is exactly what aion's Nexus platform operationalizes.

The most expensive mistake we see in production conversational commerce: treating an AI shopping assistant as a single model with a single quality score. It is not. It is a multi-objective system whose dimensions trade off against each other, often invisibly. A model that scores higher on user satisfaction can simultaneously close fewer sales. A model that is more empathetic can become more verbose, ask more questions, and slow time-to-cart. A model fine-tuned for fluency can quietly start hallucinating product attributes during tool calls.

The stakes are high: Bain & Company estimates that by 2030, "fully agentic commerce could reach $300 billion to $500 billion in revenues in the US, representing up to one-quarter of total e-commerce." McKinsey QuantumBlack's October 2025 report The agentic commerce opportunity puts the global figure even higher: "Globally, this opportunity is projected to range from $3 trillion to $5 trillion." None of that revenue accrues to retailers whose assistants ask too many questions, miss the close, or call the wrong tool.

Real-life example: A global eCommerce technology company had built a genuinely capable assistant that handled catalog search, inventory checks, cart management, and order tracking through natural dialogue. But as usage scaled, the cracks began to spread across multiple axes at once, and the team had no rigorous way to know which axis to fix first. They reached out to us to help take their production shopping assistant from impressive demo to scalable system.

The Five Conversations Hiding Inside One Assistant

When we embedded with the client's engineering and product teams, the first thing we did was refuse to treat "the assistant" as a single artifact. Instead, we decomposed it into the five distinct conversations it was actually having with every shopper, each with its own success criteria, its own failure modes, and its own data requirements.

Capability Dimension	What It Measures	Why It Matters
Clarification discipline	Is the assistant asking the right questions at the right time, or introducing unnecessary friction?	Every extra clarifying question adds latency to time-to-cart and gives a hesitant shopper another reason to leave.
Sales closure	How effectively does the assistant guide multi-turn conversations toward purchase completion?	This is most directly linked to revenue per session and the one most often invisible in generic quality scores.
Empathy and problem handling	When things go wrong, does the assistant respond like a helpful human or a decision tree?	Returns, defects, and delivery questions are where loyalty is won or lost. A scripted response in a high-emotion moment is worse than no response at all.
Tool calling reliability	Is the correct backend tool being selected with the right parameters on every invocation?	This is the deterministic spine of the system. A wrong SKU, a missed inventory check, or a malformed cart update breaks the entire experience downstream.
Language quality	Is the conversational English natural, correct, and consistent enough to serve as a foundation for multilingual expansion?	English-first quality is the baseline that determines whether multilingual fine-tunes inherit a strong policy or amplify a weak one.

Each dimension received its own task taxonomy, scoring methodology, baseline measurement, and threshold for "ship-ready." The deliverable was not a model. It was a measurement system.

Why Generic LLM Evals Don't Work in E-commerce

There is a comfortable myth in enterprise AI that you can take an off-the-shelf evaluation harness, point it at your assistant, and get a defensible production decision. We have not seen that hold up, least of all in commerce, where the domain assumptions buried in generic evals actively mislead.

A standard LLM-as-judge rubric will tell you whether the assistant's English is grammatical. It will not tell you whether asking, "What's your budget?" before "What's the occasion?" costs you a conversion. A generic tool-use benchmark will tell you whether the model can format a JSON payload. It will not tell you whether the assistant called add_to_cart with the right variant SKU when the shopper said "the blue one in medium." And almost no public benchmark distinguishes between empathy that resolves a return and empathy that performs concern while routing the user in circles.

Conversational commerce is a domain where the right metric is almost never the obvious one. That is why we built the rubric from the inside out, capability dimension first, automated scoring rubric second, human spot-checks third, and acceptance thresholds tied to the client's commercial KPIs last. The result is a benchmark the client can re-run against every model iteration, every prompt revision, every new tool integration, indefinitely. It compounds.

From Measurement to Method: Choosing the Fine-Tuning Path

Once you can measure all five dimensions independently, the fine-tuning question stops being "which method should we use?" and becomes "which method should we use for which dimension?" That distinction matters because the techniques are not interchangeable.

• Supervised fine-tuning (SFT) is the right tool for teaching the assistant the canonical patterns of good clarification — concrete examples of when to ask, what to ask, and when to stop asking.
• Direct Preference Optimization (DPO) and its single-stage cousin ORPO are the right tools for sales closure and empathy, because both are fundamentally preference problems: a human evaluator can reliably say "this response would have closed the sale, that one wouldn't" without being able to write the closing turn from scratch.
• RL-style methods become relevant when tool-calling reliability needs to be optimized against a verifiable reward: the tool call either succeeded with the right parameters or it didn't, and that signal is dense enough to learn from.

We also assessed candidate base models against the client's license, infrastructure, and size constraints — the operational realities that determine whether a fine-tuning plan is feasible, not just whether it is theoretically optimal. The deliverable from this phase was not a model. It was a prioritized optimization roadmap: the fastest path from current performance to production-grade quality across each of the five dimensions, sequenced so that improvements in one dimension would not silently regress another.

The Result: a Shopping Assistant with Systematic Evals, Benchmarks, and Roadmap

By the end of the first engagement phase, the client had four things they had never had before, in the order that mattered:

• A benchmark framework. Automated evaluation pipelines plus calibrated human spot-checks, giving them a quantitative view of model performance across all five dimensions for the first time. Repeatable for every subsequent model iteration.
• A comprehensive data strategy. Annotated conversation logs structured around the five-dimensional rubric, so every example contributed to the dimension it was designed to teach, not to a generic "make it better" pile.
• A base model evaluation. A clear-eyed assessment of candidate base models against the client's license, infrastructure, and size constraints, with recommendations on which fine-tuning approach (SFT, DPO/ORPO, RL-style) to apply to which dimension.
• A prioritized optimization roadmap. The fastest sequenced path from current baseline to production-grade quality, with the dependencies between dimensions made explicit so the team could plan around them.

How aion's Nexus Industrializes the Framework

The case study above is what one engagement looks like when our forward-deployed engineers embed with a client and stand up this framework by hand. The harder question is how you do this across dozens of use cases, dozens of model iterations, and the multilingual rollouts that come next. That is the role of Nexus, aion's proprietary platform for building, deploying, and continuously improving production-grade AI agents.

Each Nexus capability maps directly to a piece of the five-dimensional discipline:

• Systematic evaluation pipelines that score every model iteration against custom rubrics across every capability dimension you define, surfacing regressions in one dimension caused by improvements in another, before they reach users.
• Nexus Models for full lifecycle control: dataset curation, SFT/DPO/ORPO fine-tuning, A/B deployment, and a versioned model registry, so the path from a benchmark insight to a production deployment is one continuous workflow rather than three handoffs.
• Intelligent model routing that sends each task to the right model at the right cost, with evaluation built in so routing drift gets caught instead of compounding.
• Human-in-the-loop governance and full audit trails, so every clarifying question, every tool call, and every empathy turn is logged, reviewable, and explainable.
• Forward-deployed engineers embedded with your team, using Nexus to architect, implement, and ship the framework, not just advise on it.

The reason this matters for commerce specifically is that the next eighteen months will be defined by retailers' ability to make agentic systems convert reliably. Bain estimates AI agents could account for up to one-quarter of US e-commerce by 2030. McKinsey QuantumBlack notes that "by 2030, the US B2C retail market alone could see up to $1 trillion in orchestrated revenue from agentic commerce." The retailers that capture that revenue will not be the ones with the most impressive demos. They will be the ones whose five-dimensional scorecards stay green release over release.

You cannot fine-tune your way to a great shopping assistant on vibes. You measure first. You measure across every dimension that matters. And you build the infrastructure that lets you keep measuring as the model, the catalog, and the customer all change underneath you.

If that is the system you are trying to build, let's chat.