Using LLMs as a Reranker for RAG: A Practical Guide

The Core Idea

Vector search returns the top-$K$ passages from the index based on the user’s query. The LLM reranker then scores these passages to decide which are most relevant.

There are three ways to prompt the LLM for reranking [arxiv]:

• Pointwise reranking: ask the LLM to rate how relevant each passage is on a scale from 1 to 10. Example output with passage ids: [("id0", 6), ("id1", 10), ("id2", 5), ("id3", 10), ("id4", 4)]
• Listwise reranking: ask the LLM to order the passages by relevance. Example output: "id1" > "id3" > "id0" > "id5" > "id2" > "id4"
• Pairwise reranking: build a ranking through pairwise comparisons, asking the model which of two passages ($p_i$ or $p_j$​) is more relevant to the query. If you implement the LLM as a comparator inside a sort, you typically need $O(K \log K)$ or $O(K^2)$ comparisons, making it the most expensive. Studies often find pairwise prompting best on quality, though at higher cost [arxiv].

We went with pointwise reranking since it gives clear scores that are easy to use and allows useful optimizations.

Reducing output tokens

When cutting latency, reducing output tokens is a good first step: latency gain is roughly proportional to how much you cut.

We landed on two optimizations:

• Removed spaces (surprisingly expensive tokens!) and switched to a Dict format instead of List[Tuple].
• Added thresholding: instructed the LLM to omit passage ids if their score is below 5.

The first change cut output tokens by ~28%, and the second brought another ~50% latency drop.

Parallel Reranking

To improve both speed and accuracy, we split the $K$ candidate passages into $N$ batches and score them in parallel. For example, with $K = 40$ and $N = 4$, each worker gets 10 passages with the same prompt. This keeps inputs and outputs small, which improves speed and quality.

Batching strategy. Vector search already imposes an ordering bias (higher semantic similarity first). If you naively split passages into consecutive chunks, for example the first $\frac{K}{N}$ for worker 1, the next $\frac{K}{N}$ for worker 2, etc., you’ll overweight the first shard with the “best” candidates. We instead assign round-robin by index so each batch sees a similar mix of high/medium/low similarity passages: $B_j = \{\, p_t \mid t \bmod N = j \,\}$. So with $N=4$, the batches are $B_0=\{p_0, p_4, \ldots\}$, $B_1=\{p_1, p_5, \ldots\}$, and so on.

Merge step. We pool all scored items, sort by the LLM score, and use the BGE cross-encoder to break ties or fill missing scores.

RAG in Copilot: Keeping Source Diversity

Unlike Fin, Copilot searches across more entity types, including internal content and past conversation history that aren’t user-facing. If treated as one pool, past conversation excerpts can dominate, making it harder to surface more authoritative content.

To fix this, Copilot retrieves by type, scores within type, then merges using heuristic rules that keep source diversity near the top of the RAG context. We use three parallel streams: internal content, public content that Copilot can cite, and conversation history.

Impact

Fin. Our first working setup of the LLM reranker added ~5 seconds of latency. After tightening the output format, enabling thresholding, adding prompt caching, and parallelizing calls, the added latency dropped to <1s, and costs fell ~8x.

• Latency P50: +0.9s
• In the A/B test against the open source BGE reranker, the LLM reranker showed a clear quality win, with a statistically significant uplift in resolution rate

Copilot. With entity-aware retrieval and the LLM reranker, we observed the following on A/B test against BGE reranker:

• Assistance rate: +3pp
• The answer rate: +2pp
• Cited conversation excerpts: –27%
• Citations of public + internal articles: +63%

Pointwise vs Listwise

We ran an A/B test comparing a listwise LLM reranker with our parallel pointwise setup. The listwise version scores all passages at once, so needs a stronger model.

We gave it a solid try, but it didn’t outperform the pointwise version: resolution rate was the same, but latency increased ~40% and cost ~15%, so there was no clear benefit.

Reflections

LLM-based reranking helped us significantly improve quality compared to open-source cross-encoders. It also gave us a simple way to confirm that reranking quality really does make a difference in practice.

That said, this approach comes with trade-offs. Even after optimizations, the added latency is still noticeable (+0.9s), and coordinating multiple LLM calls in parallel introduces complexity.

These challenges motivated us to train a custom reranker, with the LLM reranker acting as a teacher, so we could keep the quality while reducing latency.

The Prompt

We’re open-sourcing our prompt for the LLM reranker below: