Chat Modes in AnythingLLM

AnythingLLM offers multiple ways to chat with your documents. Let's understand what each does and how to get the best results for your use case and expectations.

Available Chat Modes

Automatic Mode (recommended & default):

• Will automatically use all available agent-skills, tools, and MCPs to answer your questions
• This feature is fully dependent on the capabilities of your LLM provider and model to call tools natively.
• Most providers do not expose tool calling capabilities to the public API, so you may need to set the PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING environment variable have a always-on agent experience.

if you see the "@" symbol in your prompt input you will need to use @agent to start an agentic chat session. If it is not there, you are will be using agentic chat mode.

Chat Mode:

• Uses both your documents and the AI's general knowledge
• More conversational and flexible
• Good for brainstorming and exploring topics

Query Mode:

• Only uses information from your uploaded documents
• Will tell you if it can't find relevant information
• Best for when you need accurate, document-based answers and nothing else

Common Questions

"It keeps saying 'No relevant information found' in Query mode"

This usually means one of three things:

1. The information might be in your document but worded differently
2. The similarity settings might be too strict
3. The document might be too large and split in a way that makes finding information difficult

Quick fixes to try:

1. Go to workspace settings → Vector Database Settings
2. Change "Document similarity threshold" to "No restriction"
3. Try asking your question using words that match how it's written in your document

"When should I use Query mode vs Chat mode?"

Use Query mode when:

• You need factual answers from your documents
• You're working with technical documentation
• You want to prevent made-up information

Use Chat mode when:

• You want more conversational responses
• You need additional context or examples
• You're brainstorming ideas

"Why does it work better with some documents than others?"

Documents are processed in chunks, and each chunk is analyzed separately. This means:

• Large documents might need more specific questions
• Technical documents work better with technical questions

Tips for Better Results

1. Start with Query mode and "No restriction" similarity if you're not finding information
2. Use specific terms from your documents in your questions
3. Switch to Chat mode if you need more context or explanation
4. Try rephrasing your question if you're not getting good results