David vs Goliath: are small LLMs any good?

Are smaller fine-tuned LLMs competent for Intercom scale tasks?

Large Language Models (LLMs) are a powerful tech that have turned reasoning in natural language, into a service. They’ve had a huge impact on customer support, powering agents like Fin. Fin is already delivering real value, with many customers routinely experiencing resolution rates in the high 70s, and an overall average resolution rate across all customers of upwards of 60%.

Now that Fin is a mature product, we can start testing more ambitious ideas. One key hypothesis is that for narrow, well-scoped tasks, we might match the performance of much larger models by training smaller, more efficient ones, on enough high-quality data.

The goal here is to first focus on a well scoped, narrow task, that we could train a smaller LLM for: detect and extract the user’s issue.

Detect and extract issue summary

Issue detection and extraction is an important component of Fin’s RAG pipeline, where a series of messages between the end user and Fin are transformed into a single answerable summary issue, which is then used for the downstream retrieval pipeline.

The problem? Not all conversations have addressable issues. The old baseline setup used just one “issue detection and extraction” prompt: if there was no outstanding issue, it returned None.

But in reality, issue detection has lots of tricky edge cases, like:

• If a user gave negative feedback at the end, we want to catch it as non-informational, even if there’s still an outstanding issue.
• Users often mix feedback, greetings, or random noise with updates to their previous request, making intent hard to spot.

To deal with this, our prompt kept growing in order to cover for the nuances of non-issues – over 80 few-shot examples, >5k tokens of instructions, and 17 defined non-informational categories. A few categories of non-issue examples can be found in Table 1 below.

These nuances of issue detection made this prompt a prime candidate for experimenting with custom modelling. We can split the problem into two independent models, one that classifies an interaction as one with or without an issue, and another that extracts an issue if the first model thinks there is one.

This split strategy now makes the issue extraction task on its own a narrow task, allowing us to experiment with fine-tuned LLMs.

How do we measure success?

Before we talk about the model training effort, we need to have a clear definition of what success looks like. Following metrics are the key indicators of Fin’s health and performance:

• Offline metrics: These metrics are measured by locally replaying a sample of Fin’s production requests via the new feature:
  • Answer rates: This rate measures the fraction of times Fin was able to provide an answer for a real production query, when the new models were injected in the RAG process. Any large statistically significant drop in this number is an indicator of performance deterioration. However, a small change might not directly imply an actual degradation in production, which has been observed time and again.
  • Semantic alignment: The fine‑tuned model should generate issues whose meaning closely matches the production issues extracted by the large LLMs. We quantify this by computing the distance (e.g., cosine similarity) between the embedding of the production issue and the embedding of the corresponding issue produced by the fine‑tuned model.
• Online Metrics: These metrics are measured via an A/B test in production:
  • Resolution rates: This is the foundational metric that directly impacts Fin’s bottom line. No matter how good the model behaves offline, if it significantly deteriorates this metric, it is not a success. This metric can be split into
    • Hard resolutions : Resolutions where the end user acknowledges that the answer actually solved the problem
    • Soft resolutions: Resolutions where there is no explicit acknowledgement or positive feedback from the user.
  • CSAT: Customer satisfaction (CSAT) score indicates the quality of what Fin provides.
  • Latency: Latency has been an important metric to track for our product experience. We are constantly trying to bring this metric down, allowing end users to experience a seamless low latency interaction [5]. We want to make sure at the very least, this number remains the same.
• Cost: Often talked about in terms of amortised cost per token generated, this metric is an important one to track, especially for custom fine-tuned models. A comparable online performance, but at 2x the cost is not a success.

ModernBERT is a newer flavor of encoder-only transformer based models, and it outperforms almost all BERT-like models on retrieval and classification tasks [4]. As you can see in our blogs on retrieval, reranker, parsing feedback and escalation detection. ModernBERT works really well for routing and classification tasks, once you fine tune it on the right data.
For the issue classification task, we trained the model on 1M examples and ModernBERT achieved a remarkable 0.995 AUC score. When ModernBERT’s results didn’t match the ground truth, it was mostly the teacher’s mistakes, not the student model’s.

The blue blocks in the figure above visually describe what we actually train instead of the model weights (W). For each layer of the model, we inject two low rank matrices A and B, whose product can be added to the W once the training is complete. At train time, we optimize the weights of these two matrices instead of W itself.

A good introduction to LoRA can be found here. These LoRA adapters are like lego bricks which can be added or augmented to the original un-tuned model, to achieve an optimised performance for a specific task.

Data

We curate data from customers who have agreed with using their data for training.

The data is curated using the following conditions:

• Conversation must have happened within the past 2 months
• Customers with an account in the US, with conversation locale set as “English”[1]
• The conversation must have had an issue according to the older issue detection and extraction prompt.

The data is cleaned for obvious hygiene issues, and then anonymised by redacting any mention of emails, addresses, account numbers, phone numbers, names, places, organisations etc.

The resulting data contains 60 thousand training samples and 10 thousand validation samples.

Experiments

Before arriving at a final A/B testable model, we tested several variants of open source models, starting from a lightweight Gemma 8b, Qwen3 8b, finally getting a respectable result from Qwen3 14b variant. Some offline testing results are seen below:

The fine-tuned Qwen3 14b model seemed the most competent candidate out of all, performing at par with our baseline on answer rates. It is worth noting that we trained another variant of Qwen3 14b on just hard resolutions. The results were interesting to say the least, since despite learning to produce highly semantically aligned issues, the end to end answer rate performance was very poor. Upon closer examination, it seemed like the model only learned to produce an issue summary if the issue was extremely clear from the conversation, and refrained from producing any tokens when the conversation was ambiguous. This odd behaviour shows the importance of data curation in such projects.

The results of the issue detector model were promising, with a slight decrease in answering rates, but almost no impact on other online metrics. However, since we are not using an LLM to do the more nuanced task of detecting whether there is an addressable issue or not, this also indirectly results in a 5% reduction in cost.

Issue Extractor

Once the issue detector was shipped to production we ran an A/B test with the winning candidate model as per the offline tests, for a week. This was enough to get enough data for statistically significant results.

The overall answer rate dropped by 0.1pp. However, we saw no evidence of negative impact on other online metrics, except a slight 100 ms increase in P50 end-to-end latency. The biggest win here was the relative reduction in cost per transaction of 12.5%.

We see that both average semantic alignment and answer rates take a massive hit while using off the shelf base model. Particularly the answer rate drop shows that the off-the-shelf model lacks the competency to extract usable queries for the Fin’s RAG pipeline.

Average Perplexity computed on the generated tokens for the issue extraction task. Lower perplexity implies that the model is more sure of the tokens it generates for the same input context.

Another definitive sign of learning is a metric which is directly linked to the model’s loss that is optimized at training time. This metric is called perplexity.

All LLMs are trying to predict the next token, given all the tokens they have seen till that point in time. The way this is done is by predicting a vector of probabilities over all the tokens in their vocabulary, and then choosing the next token based on those probabilities. The way these models learn is by optimising these probabilities using a specific loss function called the cross entropy loss. Perplexity just averaged exponentiated cross entropy, progressively computed over all the generated tokens, given the model has seen the input context (in this case the anonymised chat history).

$$\text{Perplexity}(T)=\exp\left(-\frac{1}{N} \sum_{i=1}^{N}\log p \bigl(t_i \mid t_{<i}\bigr)\right)$$

The perplexity metric quantifies how much the model is surprised by seeing the next token ti, given it has seen all the previous tokens t<i. When we evaluate this metric on the generated tokens for off-the-shelf Qwen 3 14B base model and for the LoRA fine-tuned models, we see that the fine-tuned model shows substantially lower perplexity on the generated tokens. Both these results indeed confirm that fine-tuning models helps them acquire competency in specific tasks.

1. English is the most represented language with Fin, comprising about 80% of our traffic. Limiting to English language further narrows the problem, and controls for performance issues due to imbalanced language representation. We plan to explore multi-lingual in the future ↩