Unsupervised Learning Meets Generative AI: Topic Modelling for Real-World Dialogue

We designed a topic modelling system to improve the Fin AI agent by detecting underperforming areas, identifying root causes, and enabling continuous optimisation.

We wanted to get the quality benefits of modern generative AI, and the scalability and reliability of machine learning based approaches.

This blog describes how we achieved the following progress:

• Topic modelling is good for extracting themes from large text corpora. However, support conversations are short, noisy, and informal, which poses a lot of unique challenges
• To address this, we adopted an embedding-based clustering approach with HDBSCAN (which avoids predefined cluster counts and adapts to each customer’s vocabulary, volume, and complexity)
• Using a bottom-up hierarchy, the system organically discovers emerging topics – such as new feature friction or early bugs – without requiring prior labels
• Finally, we layered generative AI on top of the unsupervised structure to name topics, summarise intent, and surface examples
• This hybrid approach transforms messy support conversations into actionable insights

After exploring approaches mentioned above, we ultimately chose embedding-based clustering because it struck the right balance between flexibility, scalability, and performance. Unlike rule-based methods or supervised classification, it allowed for fully unsupervised exploration – critical for surfacing unknown or evolving topics without relying on predefined labels.

In our initial tests, traditional models like LDA or NMF struggled with short, noisy text, while embedding-based methods handled conversational nuance far better. Just as importantly, they scaled easily to millions of conversations. When we tested it, it just worked – producing coherent, interpretable topics out of the box.

So we made a bet on this method, and it paid off!

How It Works: A High-Level View

Each customer gets a tailored topic model that breaks their conversations into distinct themes. Topics and subtopics are linked to key metrics like CSAT and resolution rate, and teams can drill down to review individual conversations or apply filters to explore trends. Here is how it looks in the UI:

Here’s a simplified view of how it works under the hood:

The upper part of the picture is subtopic discovery. It is based on BERTopic framework, and here’s what it does:

1. Extract key questions – A lightweight LLM summarises the main questions from CS chats
2. Embed questions – A sentence transformer converts key questions into vector embeddings (we settled on sentence-transformers/all-MiniLM-L6-v2, based on speed and quality)
3. Reduce dimensions – UMAP (particularly effective for short-text data) projects the embeddings into a lower-dimensional space
4. Cluster into subtopics – HDBSCAN groups the reduced embeddings into coherent clusters/subtopics

The rest is topic aggregation:

1. From subtopics to topics – Clustering on subtopic embeddings surfaces higher-level, tentative topics
2. Refine with LLM – An LLM polishes tentative topics into a definitive set of final topics
3. Name topics & subtopics – Another LLM generates clear, human-readable labels, based on the final topics
4. Compute centroids – Average embeddings define centroids for each cluster (topic/subtopic)

Why HDBSCAN Beats K-Means for Conversational Data

A pretty natural question is why didn’t we use well-known K-Means for clustering?

Overall, we found embeddings and dimensionality reduction methods had only marginal impact, while clustering algorithms and hyperparameters made the biggest difference in output quality. The main advantage of HDBSCAN for our purposes is that it doesn’t require a predefined number of clusters. That flexibility was essential – each customer needs a unique topic model, and support volume, vocabulary, and conversation style vary widely (which results in very different number of topics for different customers).

The key parameter we tuned was min_cluster_size, which sets the smallest number of messages that can form a cluster. Lower values create more granular clusters; higher values produce fewer, broader ones. We settled on 15 as the minimum, which for our own Intercom data surfaced more than 700 micro-topics – a reflection of how diverse support conversations can be. Cluster counts also scale naturally with company size: more traffic means more clusters. This reinforced our intuition that fixing the number of clusters upfront wasn’t an option.

Here’s a relationship between company size (measured by 3-month conversation volume) and number of discovered subtopics. Larger companies with more diverse customers naturally yield more granular clusters – validating our choice of HDBSCAN’s adaptive approach over fixed cluster counts:

Another advantage of HDBSCAN over K-Means is how it handles noise. K-Means forces every data point into a cluster – even outliers – and offers no confidence scores. HDBSCAN, by contrast, assumes noise exists and can explicitly flag it. In customer support, many one-off or unusual questions don’t fit neatly into a topic. HDBSCAN’s ability to mark these as outliers, with associated scores, is invaluable for managing such edge cases.

From Subtopics to Topics: Building a Bottom-Up Hierarchy

Now let’s get to the second part of the pipeline: topic aggregation.

Many systems define topic hierarchies top-down: decide the categories first, then slot everything in. That works if you already know what to expect – but support data is full of surprises. We wanted the opposite: a bottom-up approach where the data speaks first, surfacing themes customers may not anticipate. The process starts with fine-grained subtopics, discovered via embedding-based clustering with HDBSCAN (as described above). Each subtopic captures a narrow slice of intent:

– Message: “What is the price for resolutions?”
→ Subtopic: Fin resolution rate

– Message: “What model does Fin use under the hood?”
→ Subtopic: Fin LLM usage

While different in detail, these subtopics both belong to the broader Fin topic. The challenge was: how do we automatically group subtopics into meaningful parent topics?

We first tried HDBSCAN’s hierarchy features, but the results were too coarse and inconsistent. So we reapplied our pipeline – this time clustering subtopic embeddings (mean sentence vectors). To our surprise, it worked: fine-grained Fin-related subtopics clustered under “Fin,” and so on. This recursive clustering produced flexible, two-layer hierarchies that adapt to each customer’s data – without predefined categories.

Still, the second clustering wasn’t perfect. To refine it, we added an LLM step: cleaning up inconsistencies, correcting edge cases, and generating clear, human-readable topic names. We layered generative models on top of the unsupervised structure so each cluster becomes understandable and actionable. The LLM helps by:

• Flagging spam or irrelevant data
• Naming subtopics and topics
• Summarising clusters in plain language
• Highlighting representative examples

This hybrid approach gives us the best of both worlds: the scale of ML with the fluency of generative AI.

PCA Aspect Ratio & Top Variance Ratio mostly fell in the expected range for spherical clusters (aspect ratio 1–2.5; variance ratio >0.35). Spectral Entropy was more mixed: most clusters scored 0.6–0.8 – egg-shaped rather than round – with some above 0.8, indicating near-spherical shapes.

In practice, this moderate anisotropy didn’t prevent centroids from being effective, especially when clusters were well-separated.

We also visually inspected the clusters to verify their shape, density, and cohesion. While not all clusters are perfectly spherical, both the metrics and visual inspections of samples confirmed that most are compact and coherent enough for centroids to act as reliable representatives:

Does it hurt performance? In short – no. To compare inference methods, we used LLM-as-judge: asking an LLM to evaluate whether each subtopic–topic pair accurately captured the theme of a conversation. It’s not a strict system metric, but a useful proxy for human judgment. The results were clear: centroid-based inference performed about 2 points better on average than the full pipeline, with both approaches exceeding 85% accuracy. In other words, simplifying didn’t just hold up – it slightly improved results.

Outlier reduction: switching to centroids cut outliers by 50%, boosting topic coverage. This mattered because support data mixes dense clusters with diffuse, low-density groups. HDBSCAN favors well-defined clusters and often discards the rest as noise – even when those “noisy” points are meaningful. Centroids let us reassign many of those borderline cases to valid topics, improving recall without sacrificing quality.

To balance performance and flexibility, we split the pipeline into two stages:

• Discovery with HDBSCAN – unsupervised exploration to uncover new topics without preset limits.
• Inference with centroids – a fast, similarity-based method optimised for scale

Evaluating Topic Quality

Evaluating topic modelling is inherently difficult. As well as being an unsupervised task, it is also inherently ambiguous. There are many valid ways to split conversations into clusters, and the “best” result often depends on human judgment and downstream usability rather than strict metrics. Still, every ML system needs some form of evaluation. We approached it from multiple angles:

• Clustering quality – Are topics well-separated and coherent?
• Prediction accuracy – Are new conversations assigned correctly?
• Human validation – Do these topics make sense to real users?

Next, we’ll walk you through how we assessed clustering quality, prediction accuracy, and human validation. If you’d rather skip the technical deep dive, feel free to skip – the key takeaway is that our evaluation confirmed the system’s clusters are both coherent and practically useful.

Clustering quality

To evaluate cluster geometry in embedding space, we focused on two cosine-distance metrics:

• Mean Inter-Centroid Distance (Separation). Measures the average distance between cluster centroids. This is critical for centroid-based inference, where close centroids can cause unstable topic assignments
  • Higher = better separation → topics are distinct, with less semantic overlap
  • Lower = weaker separation → clusters risk blending, making assignments less reliable
• Mean Intra-Cluster Distance (Cohesion). Measures the average distance between messages and their cluster centroid. This shows whether a centroid is truly representative of its cluster
  • Lower = stronger cohesion → messages are compact and consistent
  • Higher = weaker cohesion → clusters may contain noise or multiple subthemes

Together, these metrics capture the classic trade-off: clusters should be internally cohesive yet externally distinct – a must for our centroid-based approach.

Here’s a plot of these two metrics, with each data point being an Intercom customer:

Mean inter-centroid distance falls mostly between 0.6 and 0.8, averaging around 0.75, suggesting strong topic separation. Mean intra-cluster distance ranges from 0.0 to 0.35, with a typical value around 0.2, indicating good cohesion in most cases. We did observe a few outliers – models with lower separation and/or weaker cohesion. These may reflect noisier datasets or edge cases with very small or highly variable support volumes. However, the majority of models fall within an acceptable trade-off zone, showing that our unsupervised clustering approach generalises well across different customers with diverse conversation sets.

Beyond metrics, we ran sanity checks to confirm clusters behaved as expected across datasets:

• Noise Points (Outliers): HDBSCAN flagged ~10–15% of conversations as outliers – reasonable for noisy support data. Too many (50–70%) would suggest the model is overly strict, while too few might mean clusters are too broad
• Uneven Cluster Sizes: another failure mode is one giant cluster swallowing most data, leaving only tiny ones behind – usually a sign of overly permissive parameters
• Cluster Size Distribution: in practice, we saw a J-shaped distribution: many small, focused clusters and a few large, high-volume ones. This matches expectations – most issues are niche, while a handful dominate support traffic:

Prediction accuracy

It’s worth emphasising the distinction between cluster discovery and cluster assignment:

• Discovery is about in-sample structure – how well the model can group historical conversations into meaningful topics using unsupervised learning
• Prediction is about out-of-sample generalization – how accurately we can assign new, unseen conversations to the right existing topic.

This framing also highlights one of the most important challenges: deciding when a new conversation doesn’t fit any existing topic. That’s where the similarity threshold plays a key role. If enough conversations consistently fall below that threshold and form a dense region of their own it might signal the need to form a new topic cluster entirely.

To optimize the decision boundary, we inspected the distribution of cosine similarity scores. A threshold of 0.5 provided a good trade-off between precision and recall: above 0.5 = assign to closest cluster; below = flag as unassigned.

To validate the quality of topic assignments, we used LLM-as-a-judge – a lightweight evaluation method where an LLM was asked: “Does this topic match the conversation theme?”

The model returned a True/False answer, serving as a semantic proxy for accuracy in the absence of ground truth labels. This approach simulates human evaluation and helps assess how well the assigned topic captures the user’s actual intent. We also used this method to cross-validate our cosine similarity threshold. Based on these judgments, we found that a threshold of 0.5 struck the right balance between precision and recall. Overall, assignment accuracy exceeded 85%.

Unassigned: outliers or a new topic? When a new conversation receives a cosine similarity score below 0.5, we assume it doesn’t belong to any existing cluster – and it’s flagged as unassigned. However, that doesn’t always mean the conversation is noise or a rare edge case. Sometimes, it reflects something more important: a new topic that has recently emerged and wasn’t present in the data when the original clusters were created. These unassigned conversations can be early signals – new issues, product questions, or behavioural patterns that haven’t yet been captured by the system. More on how we detect and promote new topics in the next section (Model Updates).

Multi-topic aspect and borderline assignment. We also explored the multi-label nature of conversational data – since it’s well understood that a single conversation can touch on multiple topics. There are two main reasons for this:

1. In longer conversations, the topic can naturally change over time. To handle this, we extract multiple key messages per conversation, allowing us to capture more than one topic when necessary. For example, if a chat starts with Fin pricing and then moves on to a bug in conversation assignment, our system will spot and tag both topics separately
2. One message can genuinely fit more than one topic. For instance, “What’s the difference between Fin AI Agent and Fin AI Copilot?” belongs to both categories in the absence of a dedicated “comparison” topic. Such semantic overlap is common in real-world language, especially in support data where questions blend multiple intents. Occasionally, overlap instead signals redundancy – two clusters describing the same issue – which can be spotted through small inter-centroid distances.

To quantify ambiguity, we analysed secondary and tertiary topic matches via cosine similarity. Only about 5% of messages showed less than a 0.05 gap between their top two or three topic candidates – meaning true borderline cases are rare. While clearer separation between clusters reduces edge cases, genuine multi-intent conversations will always resist perfectly clean boundaries.

Human validation

No matter how good the metrics look, topics must make sense to people. Cohesion and separation scores are helpful, but they’re meaningless if clusters don’t reflect how users actually interpret their data.

Quantitative metrics can’t fully capture subjective structure – especially hierarchies. There’s no single “right” way to group subtopics: we cluster by semantic similarity (e.g., all “Fin”-related items under one topic), while some users prefer grouping by issue type, like bugs or pricing, across products.

To stay grounded in real-world needs, we interviewed customers and gathered qualitative feedback on topic clarity and usefulness. This feedback loop remains central to our process – because in the end, quality is measured by how well the system helps people do their jobs.