When experienced employees leave a company, knowledge that isn’t documented anywhere is often lost. In small and medium-sized enterprises (SMEs) in particular, there’s rarely time for this amid day-to-day operations. Yet knowledge can be preserved without first documenting it in full: An AI agent starts with what’s already available and learns as it goes through daily work. In this guest post, Leon Rocca from AlgebraX explains how he and etg24 implemented such an agent in customer service.
For about five years, the same employees have been handling customer service at etg24, a provider of a cloud platform for digital real estate management. They know the product inside and out. However, only a portion of that knowledge is documented and readily accessible. Now the company wants to grow faster: It’s receiving more customer inquiries, and the two permanent employees must train new staff while also handling more inquiries. There isn’t enough time to fully document this knowledge first. That’s why, together with etg24 and the IDEV Institute at Karlsruhe University of Applied Sciences, we’ve taken a different approach.
The problem: Knowledge resides in people's minds, not in documents
Demographic change is making the loss of knowledge a growing risk. According to the medium-term forecast by the Federal Ministry of Labor and Social Affairs, the number of people of working age will decline by approximately 1.9 million by 2029. At the same time, experiential knowledge is rarely written down: According to findings from knowledge management research, only about 20 percent of relevant corporate knowledge is documented. Approximately 80 percent consists of the experiential knowledge of individual employees.
In many small and medium-sized enterprises (SMEs), this knowledge is tied to specific individuals who know exactly how the product works in detail or which solution is best suited to a particular customer problem. If one of them leaves the company or retires, there is usually not enough time for a thorough handover. At etg24, therefore, an AI agent was designed to support customer service employees in their daily work and, in the process, gradually make their experiential knowledge available.
The Approach: Start Incomplete, Learn Through Everyday Life
The usual approach would be to first fully document the empirical knowledge and then build an AI agent on top of it. We deliberately chose a different path: The agent started with the knowledge that was already available within the company and was designed to acquire new knowledge through day-to-day operations.
Among other things, there was an existing user manual and 412 completed customer communications, each of which contained the customer’s inquiry and the agent’s response. These formed the agent’s knowledge base. The goal was for the agent to be able to provide useful assistance for about 70 to 80 percent of inquiries from day one and to learn how to handle the remaining cases over time, without requiring any additional effort from the staff.
The agent itself functions like a digital employee in the customer service email inbox. An inquiry goes through four steps:
Technically, the agent consists of the low-code platform n8n, the Gemini language model, and a vector database in which it searches for text passages that match the content. With n8n, such workflows can be built without complex programming. The database containing the knowledge is stored on servers in Germany in compliance with the GDPR.
An AI agent never answers 100 percent of inquiries without error. However, if it is clear what problem it is supposed to solve, starting with 70 percent usable answers may initially be enough to noticeably lighten the workload for employees.
The Results: What the Tests Show
We tested the agent in two scenarios: a quality test using new customer inquiries and a live test with an employee.
For the quality test, we presented the agent with 33 new customer inquiries that were not yet included in its knowledge base. These inquiries had been received during the two to three weeks between the initial training and the test. A second AI language model served as a judge (“LLM-as-a-Judge”) to evaluate the responses using the RAGAS measurement method.
The evaluation was conducted on a scale from 0 to 1. The agent achieved scores ranging from 0.67 to 0.74. The evaluation assessed whether the answers were supported by the sources, whether the search found the relevant information, and whether the answer directly addressed the question. These are quality metrics, not hit rates: a score of 0.74 does not mean that 74 percent of the answers are correct. Given that we are starting with an incomplete knowledge base, we consider these scores to be acceptable.
In the second test, the live test, an etg24 employee evaluated the agent’s suggested responses to seven real customer inquiries, including standard cases, an escalated inquiry, and an inquiry regarding a temporary system malfunction. She considered the suggestions to be generally useful across the board. Where they deviated, she believed this was due to the knowledge base still being incomplete.
The learning effect was most evident during the disruption. A feature was temporarily unavailable, and several inquiries reported the same issue. The agent’s initial suggestion did not yet take this into account. The employee sent her own response. When the next similar inquiry came in, the AI agent noticeably adjusted its suggestion. This shows that it learns from just a single piece of feedback.
The Analysis: What the Results Indicate and What Remains Unclear
Both tests suggest that experiential knowledge can be captured without first documenting everything: The agent provides useful suggestions even with incomplete knowledge and learns from the responses it sends. However, these are merely indications and not yet conclusive evidence. They are the results of an initial, limited trial within a company. Long-term deployment has yet to take place.
It remains to be seen whether a 70 percent rate of useful suggestions is sufficient in day-to-day operations, how much feedback is needed to achieve a measurable improvement, and whether employee trust will hold up over the course of several months. This will only become clear with prolonged use. Nevertheless, the results can serve as a guide for other small and medium-sized businesses where customer service is primarily handled via email, knowledge resides in employees’ heads, and there is a lack of time and staff for complex IT projects. Preserving knowledge, therefore, does not have to begin with comprehensive documentation but can start with a tool that reduces employees’ workload immediately after implementation.
About the author
Leon Rocca is an AI Solution Consultant and COO/CFO at AlgebraX. He studied International Management (B.A.) and Technology Entrepreneurship (M.A.) at Karlsruhe University of Applied Sciences, with a focus on AI agent systems.
About the Project
The project on AI-agent-supported knowledge preservation in SME customer service was developed in collaboration with etg24, the IDEV Institute at Karlsruhe University of Applied Sciences, and AlgebraX. The results have been published as a short paper: “AI-Agent-Supported Knowledge Preservation in SME Customer Service: A RAG-Based Approach for Incomplete Knowledge Bases” (Leon Rocca, Johannes Penka, Prof. Dr. Carsten Hahn, Johannes Lauer).