Abstract:
Recent advancements in large language models (LLMs) have significantly impacted various domains. However, the delay in knowledge updates and the presence of hallucination issues limit their effectiveness in knowledge-intensive tasks. Retrieval augmented generation (RAG) offers a solution by incorporating external information. Traditional RAG methods primarily rely on similarity to connect queries with documents, following a simple retrieve-then-read process. This work introduces MetRag, a framework that enhances RAG by integrating multi-layered thoughts, moving beyond mere similarity-based approaches. MetRag employs a utility model supervised by an LLM to generate utility-oriented thoughts and combines them with similarity-oriented thoughts for improved performance. It also uses LLMs as task-adaptive summarizers to condense retrieved documents, fostering compactness-oriented thought. This multi-layered approach culminates in a knowledge-augmented generation process, proving superior in extensive experiments on knowledge-intensive tasks.
Key Contributions:
• Utility-Oriented Thought: Incorporates a small-scale utility model for better relevance.
• Compactness-Oriented Thought: Utilizes LLMs to summarize large sets of retrieved documents.
• Knowledge-Augmented Generation: Combines multiple thought layers for enhanced output.
Significance: MetRag demonstrates improved performance in knowledge-intensive tasks by addressing the limitations of traditional RAG methods through a multi-layered thought approach.
Applications: This framework can be applied to various domains requiring up-to-date and accurate knowledge, enhancing the reliability and efficiency of LLMs in real-world tasks.
Similarity is Not All You Need: Endowing Retrieval Augmented Generation with Multi Layered Thoughts
Add a Comment