ManualWorks needs to tell whether a real user is accessing the web viewer with a web browser or whether a web crawler or bot is accessing it. It uses the User-Agent value in the HTTP request header to decide.

Whether a request comes from a web crawler affects the following.

• Whether a visit is a bot in the <Statistics | Visit Statistics> menu

• How search engines are identified when the “Disable the access of Web Crawler.” option is selected for a document, blog, library, or video

• Whether search-engine-optimized web viewer HTML is served

AI Fetchers

When someone pastes a document address into an AI tool, a fetcher retrieves the page on the spot. These fetchers do not have bot in their names, but ManualWorks recognizes them by default, so you do not need to configure anything.

• ChatGPT-User, Claude-User, Perplexity-User

• Google-Agent, GoogleAgent-URLContext, Google-GeminiNotebook, GoogleOther

• meta-externalagent, meta-externalfetcher

• MistralAI-User, MistralAI-Index, MistralAI-Training

• Headless Chrome (HeadlessChrome), used by Grok

ManualWorks also recognizes Google-InspectionTool, Google-Read-Aloud, Google-Pinpoint, GoogleMessages, and FeedFetcher-Google, which are not AI but fetch pages at a user's request.

If you select the “Disable the access of Web Crawler.” option for a document, these fetchers cannot read its content either. To let AI read your documents, do not select this option.

Updating Web Crawler Patterns

ManualWorks includes the User-Agent values of well-known bots. If the User-Agent value contains a pattern string, the request is treated as a web crawler. Add patterns as follows if needed. Patterns are converted to lowercase before comparison, so they are case-insensitive.

1. Go to the <Admin | Preferences | Web Crawler> menu.

2. Click the <Update Web Crawler Pattern> link.

3. Enter one pattern per line in "Web Crawler Pattern(User)".

4. Click the Save button.

Testing Whether a Request Is a Web Crawler

You can test whether a specific User-Agent value is treated as a web crawler.

1. Go to the <Admin | Preferences | Web Crawler> menu.

2. Enter the 1User-Agent value.

3. Click the 2run icon.