Retrieval-Augmented Generation (RAG) represents a breakthrough in knowledge management, promising organisations a way to transform vast, fragmented information repositories into accessible, intelligent systems. Yet beneath the surface lies a critical challenge that is often overlooked.

AI-powered knowledge tools are not static repositories. They are dynamic ecosystems that require continuous cultivation and strategic oversight. The initial excitement of near-instantaneous, comprehensive answers can quickly give way to inconsistent, sometimes inaccurate responses that, over time, can erode user trust and organisational effectiveness.

Earlier this year Stack Overflow published results of their annual survey and it showed that developers were losing faith in AI due to a lack of quality in response. Just 43% said they feel good about AI accuracy, while 31% are sceptical. Notably, these numbers are much the same as last year’s, showing that AI tools have made little progress in the accuracy of their results, or at least that is how it is perceived by end users.

The Quality Decay Dilemma

The core issue is quality decay—a subtle but pervasive problem that transforms promising AI solutions into frustrating, unreliable tools. As RAG systems continuously ingest new information, modify retrieval mechanisms, and adapt to changing organisational landscapes, their response quality naturally begins to deteriorate. Without a focus on quality management, this leads to poor responses, a negative user experience and declining trust and possible misinformation. Ultimately this means a reduction in productivity gains which the tools initially set out to achieve.

Building a Sustainable RAG Solution

We encountered this challenge directly while working with a leading oil and gas corporation struggling with fragmented knowledge management. Their engineers were losing countless hours searching through disparate documentation systems. Our solution, a RAG-powered knowledge portal, initially saved nearly more than 68,000 engineering hours in its first year.

However, this initial success doesn’t guarantee long-term effectiveness and a focus on quality of response was a metric that we always measured. Although we had initially rolled out a ‘thumbs up/thumbs down’ feedback system for users to indicate response quality, users simply didn’t use this feedback mechanism enough (as is often the case when users are busy focusing on their own work). As a result, the amount of feedback data generated wasn’t sufficient to make authoritative and accurate assessments about response quality.

Since response quality in LLM and RAG-based solutions tends to degrade over time as new data sources are added and removed, we needed to find a better way of ensuring response quality, even when feedback data was unavailable.

While there are plenty of industry and even government-backed frameworks out there that offer advice on overcoming these issues, these relatively new frameworks often lack real-world implementation examples, making it difficult to put them into action. So we decided to try a more innovative approach instead—we automated the collection and evaluation of feedback in the knowledge management system.

Firstly, we identified the most important metrics:

• Faithfulness: the accuracy of the information provided in the response
• Groundedness: the extent to which the response is based on the data available in the enterprise
• Relevance: the degree to which the answer is relevant to the user’s question
• Context relevance: consideration of the larger context when responding

Secondly, we used an LLM as a kind of judge, evaluating all user interactions in near real-time using these key metrics.

Finally, we visualised these metrics using MixPanel dashboards. This allowed us to track maximum, average, and minimum values over time, helping to identify trends and make any necessary alterations quickly. The narrative of these dashboards became a core point of discussion during our twice-weekly product meetings, and they’re now an essential factor in the knowledge management tool’s continuous improvement.

The most successful knowledge management strategies don’t just implement AI—they cultivate it. This means moving beyond the “set it and forget it” mentality and treating intelligent systems as living, evolving ecosystems that require ongoing attention and refinement.

The future of enterprise knowledge management isn’t about deploying revolutionary technology. It’s about creating adaptive, continuously improving information environments that grow more intelligent with each interaction.

To find out more about our Knowledge Management services click here.