Hello, this is 3Rabbitz.

The help section of the 3Rabbitz website has a ManualWorks chatbot. It answers based on product documents written with ManualWorks. We opened every single answer this chatbot gave over two months, August and September. There weren't a huge number of questions, so we summarized them by ratio rather than count. Some questions were test inputs from our own staff.

Let's start with the conclusion. In this analysis, most of the reasons the chatbot failed to answer properly were not in the chatbot but in the documents. The biggest improvement to make was filling in the documents, and ManualWorks already has a way to find and fix those documents.

Two months of chatbot answers

The ManualWorks chatbot records every answer and picks out answers with the following signals as "weak answers." It casts a wide net so that nothing slips through, so some properly answered questions are included too.

• The chatbot answered "It's not in the documents."

• It reached the search limit while looking through the documents.

• The reader clicked "Not helpful."

The ratios in the following table are the share of all answers in that month that had each signal. A single answer can have several signals, so the ratios for each signal add up to more than the ratio of weak answers.

In both months, nearly one in two answers (45%) was flagged as weak. The ratio was the same, but what was inside was different. In September, answers that reached the search limit fell from 45% to 31%, and "It's not in the documents" answers rose from 19% to 27%.

Apart from internal tests, there were no cases where the same person asked a similar question again within a short time.

Looking only at the number, nearly half of the answers being weak, your first thought might be "Should we change the chatbot model?" But when we opened them one by one, the story was different.

Opening the weak answers one by one

We read every question and answer flagged as weak and picked one main cause for each answer.

The 30% in the last row only reached the limit because the chatbot searched the documents several times; the answers themselves matched the documents. Leaving out this 30% that turned out to be fine, about 77% of the rest happened because the content wasn't in the documents (54%) or the reader's words differed from the document's words (23%). The rest were unrelated or vague questions.

The content wasn't in the documents

Many of the questions the chatbot answered with "It's not in the documents" really weren't covered in the documents.

• How to see the owners of all documents at once

• How to set the context path when serving behind a reverse proxy

• Which permission lets you create chapters

• The steps and installation files for updating from version 3.0 to the latest version

These are things customers actually want to know. The ManualWorks chatbot answers only from the documents you provide, so instead of making up content that isn't there, it honestly answered "It's not in the documents." Even a better model won't produce an answer that isn't in the documents. Nor should it.

Write down missing features as "not supported" too

There was an interesting point as well. The following questions ask about features ManualWorks doesn't support.

• Combining several short documents into one PDF file

• Comparing two different documents

• Going back from MySQL to the default database (H2)

Because the documents don't mention them at all, the chatbot answers "It's not in the documents" rather than "It's not supported." The reader can't tell whether the feature is missing or the documentation is.

On the other hand, to the question "Can I download it as HWP?", it clearly answered "It's not supported." That's because the documents clearly listed the file formats you can download (PDF, EPUB, HTML, MS Word). When you clearly write down what is supported, the chatbot can also answer what isn't.

The reader's words differed from the document's words

In some cases, the content was in the documents, but the chatbot couldn't find it.

• The reader asks about "SSO," and the documents explain "OAuth 2.0" and "LDAP."

• The reader asks about "folders," and the documents say "directories."

• The reader asks about "the icon with three lines and a pen," and the documents describe it only by its name, "Create Book."

The chatbot finds elements with similar meanings, but when the reader's words are far from the document's words, it misses them. The fix isn't hard. Just write the reader's words in the documents as well.

Now the administrator guide has the sentence "OAuth 2.0 is also used when integrating your company's authentication system with ManualWorks through SSO (Single Sign-On)." After updating the document, we reran only the chatbot's search step with the same "SSO" question, and the two elements that mention SSO came up first and second.

Few cases could be blamed on the chatbot

There were also cut-off questions like "Making a document," and questions like "How to order pizza with ManualWorks." Answering "It's not in the documents" to these is the correct behavior. For questions whose content was clearly in the documents, the chatbot mostly answered in line with the documents, even when it reached the search limit.

In 6.0.26, we refined the chatbot to handle these questions better. When a short question could point to different features, it asks which one you mean instead of guessing. When turning a question into search queries, it refers to the product description written in the AI assistant, which reduces searches that go astray into other products' content, such as reading "index" as a database index.

So documents matter more than the chatbot

When building a chatbot, the first concerns are usually which AI model to use and which embedding model to search with. Those matter, of course. But after looking at two months of logs, our thinking became clear. Changing the model may slightly improve the wording and search ranking. But it can't fill in what's missing from the documents.

We believe a chatbot's answer quality comes from the documents more than from chatbot features or AI models. And chatbot logs are the most honest record of what readers couldn't find in your documents. They tell you, in the readers' own words, exactly where readers got stuck.

That's why the tool for writing documents and the chatbot should be in one place. In ManualWorks, writing documents, having the chatbot answer from them, and managing the fixes as tasks all flow within one product. You can take problems found in chatbot logs straight into improving your documents, without moving them to another system.

From logs to tasks, from tasks to documents

If you stop at reading the logs, nothing changes. ManualWorks lets you carry on with an AI tool through the whole process: reading chatbot logs, turning fixes into tasks, fixing documents, and checking again. People decide what to fix and check the results. The rest is left to AI. It's a semi-automated approach.

Step 1. Collect weak answers

User: Sort the weak chatbot answers from August and September by cause.

AI: I've read all the weak answers from the two months.

- The content isn't in the documents: owners of all documents, reverse proxy context path ...

- The reader's words differ from the document's terms: SSO, folder, icon shape ...

- Answered properly but reached the search limit: ...

The AI tool uses the chatbot history tools of the ManualWorks MCP to find weak answers (get_chatbot_weak_answers) and questions the same person asked again (get_chatbot_reasks). Because it reads only existing logs, it doesn't call an AI model again, and it can look at many days at once. It also returns the original questions, so you can see exactly what words readers used.

Step 2. Turn fixes into tasks

User: For each question sorted as "The content isn't in the documents" or "Different terms," find the document and chapter to fix, and create a task in the "Document Improvement" channel.

AI: I found the related chapters and created tasks.

- "Administrator Guide - User Management: Add how to check the owners of all documents at once"

- "Installation Guide - Explain that subpaths aren't supported behind a reverse proxy, and how to set up a subdomain"

- "User Guide - Creating Files: State that several documents can't be combined into one file"

- ...

Each task includes the reader's original question and the chatbot's answer.

Which chapter of which document to fix, and what words the reader used, stay in the task as they are. Assign an owner and add a milestone, and you can manage document improvements just like development work.

Step 3. Fix the documents

User: Read the reverse proxy task and draft an element for the installation guide. Tell me where to put it, too.

AI: I suggest adding a new section, "Serving behind a reverse proxy," at the end of the "Linux Installation Guide" chapter. Here's the draft.

...

User: Good. Add it there.

AI: I've added the element. I'll leave the result as an answer on the task.

AI writes the draft, and a person checks it. If you're hesitant to edit a document directly, you can work in a branch, review it, and then merge. Often it's enough to add one sentence saying "not supported" for a missing feature, or to add the reader's words when they differ from yours.

Step 4. Recheck with the same question

User: Search the "SSO" question again with the fixed documents.

AI: I reran only the search step.

1st: Managing Users with OAuth 2.0 - "OAuth 2.0 is also used when integrating your company's authentication system with ManualWorks through SSO."

2nd: Handling User Login with JSP - "With your company's authentication system, such as a commercial SSO solution ..."

Now the elements that mention SSO come up first.

After fixing the documents, sync the vector store and enter the reader's question again as it was. You can see the answer and its source elements (test_chatbot), or only the search ranking without an answer (test_chatbot_retrieval). Test runs are recorded separately from customer questions and are left out by default when you look up the chatbot history.

Get started now

ManualWorks 6.0.26 brings the following changes to the chatbot.

• Find weak answers and repeated questions from the same person with MCP (get_chatbot_weak_answers, get_chatbot_reasks).

• Test the answer and source elements for a single question, or check only the search ranking (test_chatbot, test_chatbot_retrieval).

• For vague short questions, the chatbot asks back instead of guessing.

• When turning a question into search queries, it knows which product's documents it is searching.

• Elements that only point elsewhere, like "See ...," are excluded from search.

The MCP tools that read chatbot history are available to users with the "Admin" permission. For how to set up the chatbot, see the 3Rabbitz Chatbot Guide, and for how to connect an AI tool, see Using ManualWorks in Claude Code/Codex/Gemini CLI.

Hand this month's chatbot logs to your AI tool. "Collect the weak answers and create tasks for the documents to fix." This one sentence is where fixing your documents begins.