ResearchHuman Perception of AI
Conventional wisdom says speed matters in software and that fast is always better. But in testing our AI agent, we found that slowing down might actually make it feel smarter.

“Slow is smooth, smooth is fast.” -Navy SEALs saying
Nobody likes slow software. Despite Moore’s law and improving hardware, software doesn’t seem to be getting faster over time. Sometimes it feels like things are even slowing down due to increasing bloat. Software latency isn’t just a user annoyance, there’s also documented evidence of real business cost:
Does the same apply to AI agents for customer support? Is latency just annoying, or also harmful?
As LLMs inherently do so much computation – which takes time – and as there are so many different LLMs available, with different latency and quality tradeoffs, we needed to know.
The only way to answer that important question is through an AB test, as the natural latency variation is confounded by many factors (e.g. peak time has higher latency then early morning, but the user types and queries in those periods are different too).
At Intercom’s AI group, we run hundreds of AB tests per year. Almost all of them are to evaluate improvements to our AI Agent Fin, but we made an exception here: To understand the impact of latency to Fin, we ran an experiment where we increased latency in a small amount of conversations.
We faced two difficulties:
We managed to implement an experiment design where the changes were indeed imperceptible. The experiment also uncovered a non-obvious truth:
Those results were so surprising that we had to run confirmatory tests. Here is a summary of the 2 more relevant experiments:

Despite those results, since then Fin has become faster, not slower: Despite our findings, we still want to speed Fin up! Even if the latency doesn’t seem to negatively effect the business outcomes, we want to make the experience fast. Nonetheless, these results are valuable – not least because they have changed the way we run AB tests at Intercom, to account for latency as an important confounder.
Before exploring the results and take aways, let’s dive into the methodology first.
Due to the nature of the experiment and to make it blind, we had to design it in a way that it wouldn’t be noticeable to both users and customers:
We first measured the current latency percentiles and made increases that would be consistent with the already-existing latency. For example, at the time of the experiment, the P99 latency used to be 20 seconds over the median, so the variant that increased latency by 20 seconds was just 1% of the conversational volume.
In other words, as a user, you wouldn’t notice an increase in latency, unless you were unlucky enough to be in the natural 99 percentile and fall in the 20 second delay experiment arm, a tiny 1 in 10k chance!
During the time of the experiment, we didn’t receive any latency complaints that could be attributed to the experiment, indicating that the methodology was a success in terms of hiding the experiment effects from users and customers.
We randomized the experiment at the conversation-level[4] with 1 control and 4 treatment arms roughly mimicking the natural latency variation:
As the results were so surprising, we ran two more confirmatory experiments:
All 3 experiments found similar results.
When running AB tests, we monitor it through a dashboard that covers dozens of metrics. Here are the most important ones:
Those metrics help understand the impact of each experiment under many different dimensions. They allow us to judge whether the experiment was well designed and implemented, has enough sample size, and its impacts on business, users, and engineering metrics.
We will expand on our experimental methodology in future posts.
“Any figure that looks interesting or different is usually wrong” -Twyman’s law
Before we get to the actual results, here is what we expected to happen from first-principles:
Here is what we actually saw:
The overall resolution rate increases by up to 2 percentage points (pp):

To understand what is going on, we need to break resolution down into its two components:
As expected, assumed resolution is the main driver of the increase: we expect users to drop off more, leading to more such resolutions when AI answers are delivered[5].

Surprisingly, confirmed resolutions, which are resolutions with positive feedback, seem to actually increase by 0.4pp for higher latency increases. This was the most puzzling result – a Confirmed Resolution is the result of an end user explicitly clicking a button saying “that helped 👍”, and we use this as our north star. As such, this should only go up if the user was in fact more satisfied with the AI answers. It was this puzzling result that led to further analysis and experiments.

Also surprisingly, negative feedback is not increased and seems reduced for higher latencies. Note that negative feedback does not mean anger or dissatisfaction, only replying “No” after “did that help?”.

Finally, we don’t see a clear impact to CSAT rating, but we do see an increase in response rate in higher latencies, another puzzling result:

We saw expected results for deflections and assumed resolutions but not for the feedback associated metrics. Are users actually more happy with increased latency?
We brainstormed potential explanations:
We ruled the first hypothesis by double checking the data, running confirmatory experiments, and conferring with our excellent AI infrastructure engineering team, owners of the AB test assignment logic.
“When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth”. -Sherlock Holmes
Our best hypothesis is a psychological explanation: The longer users wait, the more human or effortful the reply becomes in their minds, leading to improved answer perception. We have some corroborating evidence for that hypothesis:
If true, a question we naturally might ask is, how else can we make an AI agent be more humanlike whilst remaining within legal[6] and moral limits? That has led to other interesting experiment ideas:
While some of those experiments are still ongoing (more on them in future posts), all have promising results, so we are even more confident that the psychological hypothesis is the correct one.
Another takeaway is that latency is a confounder for any other experiment that accidentally increases or decreases latency.
In other words, if an experiment testing something else happens to increase latency by 10 seconds, it might show positive benefits solely due to latency. Conversely, an experiment that reduces latency might decrease resolution rate solely due to latency reduction.
That means latency needs to be corrected for. For experiments with increased latency, we should remove the resolution rate gains as shown above. For experiments with decreased latency, we need to add another variant with latency corrections, resulting in an ABC test:
This effect also explains some historical results. For example, the biggest ever Fin latency reduction required multiple iterations not to harm Fin metrics. At first, we thought it was due to the quality of the original prompt, but now we realise some of it was just due to latency. Meaning, we underestimated how good the change was as the only reason the resolution rate didn’t go up was due to the latency reduction.
Despite those results, since we ran that experiment in late 2024, we have made good progress in reducing Fin’s latency, from changing the way we serve Claude Sonnet models to a new more streamlined architecture, where we rely on eager execution to save precious seconds.
As of March 14, 2025, Fin is at its fastest ever: The median time to first token is now 7 seconds, a 60% reduction from its peak.
Why are we making Fin faster, given the experiment findings? While we’re confident in the findings, we know latency effects go beyond just end user psychology. We must also serve our customers, who are actually paying for Fin’s resolutions. While they might be glad of a marginal resolution rate increase, they’d much prefer for their users to have the best possible experience.
Fin is now the first-line of support for many of our customers, which range from Anthropic to Clay to Culture Amp, and is responsible for over half of all their resolutions on average, meaning Fin is an integral part of their customer support operations.
From that prism, it’s unacceptable to have a subpar experience even if it doesn’t harm end user metrics. We go back to the original opener of this post: nobody likes slow software. We want Fin to be fast and to be perceived as such, as we want Fin to be the best AI agent in all dimensions.
You should expect even more latency improvements to Fin in 2025.
3. Walmart slides ↩
4. We typically randomize AB tests at the conversation instead of the more typical user-level. There are pros and cons to each alternative, but here the decision was clear: We didn’t want to harm the end user experience, so randomizing by conversation gives the end user opportunity to get support at natural latencies most of the times. ↩
5. AI answer delivered does not mean it was read or seen by the end user! ↩
6. California’s SB-1001, the Bolstering Online Transparency Act, prohibits bots from communicating online to mislead about their artificial identity for commercial transactions or influencing votes, requiring clear disclosure to California consumers, with enforcement by the state attorney general and fines up to $2,500 per violation. ↩