Balancing Relevance & Diversity in RAG — guide for smarter doc selection

TL;DR: Combine embeddings for relevance, bert similarity score for diversity, and the knapsack algorithm to select the best documents within token limits — to achieve balanced and flexible RAG solution.

In this article, I’ll introduce a simple yet effective approach for RAG system that you can use when you need to pull in a wider variety of data sources without sacrificing relevance.

For example, having diverse sources may be very important when tackling those ambiguous or open-ended questions that don’t have just one right answer, like opinions, trends, or cultural insights. Having diverse documents brings in a bunch of different viewpoints, making the responses way richer.

Additionally, mixing in varied data sources helps keep bias in check, giving users more balanced and reliable info, which boosts the credibility of the RAG system.

And when it comes to summarising complex topics, pulling from diverse sources lets you capture multiple perspectives and subtopics, resulting in a more well-rounded overview.

I know that there are more equally important reasons behind diverse sources, but I hope you can now feel the idea of it based on the 3 reasons above.