ProductBridge vs Gleap: Support Agent Benchmark on Smartlead's Documentation (4.28 vs 3.76)

TL;DR. We pointed the ProductBridge support agent at Smartlead's public documentation and asked it the same 50 questions Smartlead's live Gleap-powered support bot gets. A blind judge (GPT-6 Astra) scored both on six criteria without knowing which product wrote each reply. ProductBridge scored 4.28 / 5, Gleap 3.76 / 5. The biggest gap was not knowledge. It was calibration: knowing when to answer, when to ask, and when to say "I don't know". We also publish the one category Gleap won, and a retrieval defect we found afterwards that makes these numbers a floor rather than a ceiling.

Why we benchmarked the ProductBridge support agent against Gleap

When a SaaS team evaluates an AI support agent, the first objection is always the same: will it hallucinate to our customers? Every vendor answers with a number. Almost none of them show the method. A score without a method is marketing.

So we did it the other way round: publish the method, measure the method's own error, and then run it against a competitor's production bot on that competitor's documentation. Smartlead was a natural test. Its help center runs on Gleap, one of the platforms we are compared against, and its docs are public: 585 pages on a technical product, with plenty of confusable feature names, rate limits and edge cases.

How ProductBridge and Gleap were tested: blind judge, six criteria, measured error bar

Design. Both engines answer the same questions over the same documentation, so a score gap reflects the engine, not the corpus. We indexed Smartlead's docs into a ProductBridge workspace and captured 110 runs through our public API. For Gleap, we drove the live widget on Smartlead's help center, 50 cases in all.

One asymmetry, and it favours Gleap. The Gleap bot on Smartlead's help center is configured and maintained by Smartlead's own team. It has whatever context they chose to give it beyond the public articles, such as the questions their users actually ask most and how they want them answered. ProductBridge had none of that. We indexed the 585 public help pages and nothing else: no product access, no conversation history, no tuning on Smartlead's real tickets. That is the setup any prospect would have on day one, but it does mean the comparison is between a bot with home-ground context and an agent working from the public docs alone. With the full context of the product we would expect a wider margin, not a narrower one.

Blind grading. A single judge model, GPT-6 Astra, scores every reply on six criteria from 1 to 5 and is never told which product produced it. We deliberately avoided pairwise judging. It is more sensitive, but it leaks identity through formatting and position, and when the result informs a decision about our own product that is not a risk worth taking.

Deterministic facts are not left to opinion. Whether a reply repeated a known-wrong answer, handed off to a human, or timed out is computed in code and passed to the judge as context.

Isolation. Our agent can read a visitor's recent conversations. Running all 137 turns as one visitor would let case 100 answer partly from cases 1 to 99, inflating exactly the categories the suite exists to measure. So every case ran as a fresh anonymous visitor. Gleap got the same treatment, with a new conversation per case.

The grader has a published error bar. Before trusting any number, we had the judge grade the same 22 runs twice under identical settings to see how much it disagrees with itself.

So the hallucination count is solid, category averages average the noise down, and an individual case score is not solid. A gap under about 0.2 is not a finding. The 0.52 headline gap is 2.6 times the noise floor.

Result: ProductBridge 4.28 vs Gleap 3.76, and calibration is the real gap

Head-to-head on the 50 shared cases:

The line we care most about is calibration, at +0.78. It is the largest margin, and it measures whether the agent knows what it doesn't know: answering when the docs cover the question, asking when the question is ambiguous, declining when the docs are silent. That is the hardest habit for a support agent to learn and the most expensive to get wrong, because a confident wrong answer costs a customer's trust and then a ticket to unwind it.

One measure that is not in the table. The suite plants specific wrong answers that a pattern-matching bot tends to produce instead of retrieving, such as a fabricated API key or an invented limit. ProductBridge never took the bait in 110 runs. That is a narrower claim than "zero hallucinations", which we do not make: on the shared cases, 8% of our replies were flagged as hallucinated, against 18% of Gleap's.

By category

Seven categories won, two tied, one lost. Gleap was better on rate-limit questions. Only two cases sit in that category, so it is a thin sample, but a loss is a loss and it stays in the table.

The conclusion survives every averaging choice

Someone will ask, so here is the answer. Gleap was captured once per case, so the headline compares it against ProductBridge's first repetition only, one sample against one sample. We also computed it the other two ways.

The published figure is the middle, principled choice, not the flattering one. The gap holds under all three.

Where Gleap beat ProductBridge, and what we still get wrong

1. The same question could get materially different answers. Of 25 repeated cases, 11 varied by a full point or more between runs. No temperature or seed is pinned anywhere in the agent path, so it ran at the provider default. That is fixable, and it was not fixed when these runs happened.

2. Gleap beat us on false-premise handling. Asked how to configure a feature that does not exist, Gleap corrected the premise. We accepted it and invented setup steps around it.

3. Gleap won the rate-limits category. Two cases, 2.92 to our 2.33. We refused one question and answered the other wrongly, where Gleap gave usable answers to both. Two cases is too few to generalise from, but that is the result.

What ProductBridge found and fixed: half the index was navigation boilerplate

After these runs we traced the "not documented" failures to their root cause, and it was ours. Roughly half the knowledge index was not documentation at all. It was the docs site's navigation, captured again on every page.

On one workspace, 8,927 chunks contained only 3,893 distinct texts. The largest group was 231 chunks with a bit-identical embedding, all of them the sidebar. A cluster of identical vectors traps the search, so the right page becomes unreachable and the agent honestly reports that it cannot find something sitting in the index.

The Smartlead evaluation workspace measured 57% duplicate chunks. Every score above was therefore produced with a significantly degraded retriever, which we have since fixed. Add the fact that Gleap was working with Smartlead's own context and ProductBridge with the public docs alone, and these numbers are a floor, not a ceiling. That is a better story than a clean win, and it has the advantage of being true.

What this means if you are evaluating a support agent

• Ask for the method, not the score. Blind or pairwise? Which criteria? Was the judge's own error measured?

• Ask which categories they lost. A benchmark with no losses is a benchmark with no traps.

• Ask what "zero hallucinations" means. Zero planted wrong strings and zero hallucinations are different claims, and a vendor quoting the first as the second is hiding a number.

• Ask when the numbers were captured and what changed since. Ours are from 1 and 2 September 2026, before the retrieval fix above.

Try it on your own documentation

ProductBridge trains a support agent on your help center in minutes and runs it inside the same customer record as your feedback, roadmap and changelog. Start on the free plan at app.productbridge.io/signup, or book a demo and we will run a subset of this suite against your docs live.

Every number in this case study reproduces from the committed score files. If you want the raw transcripts, including every miss, email help@productbridge.io.