About Automatic document sync

The Automatic Document Sync feature for AnythingLLM allows you to "watch" a document for active changes. When changes are detected the file will be re-embed and all workspaces using this file will automatically be updated.

This enables you to reference a document and have it's content consistently updates so answers are always accurate to the original source.

Scope of documents

Docker

• Any website link
• Any file collected via a Data connector (eg: Confluence, Github, and YouTube)
• Manually uploaded files are not synced since the browser cannot read from your computer

Desktop

• Any website link
• Any file collected via a Data connector (eg: Confluence, Github, and YouTube)
• Manually uploaded files still in development

Enable the feature

First, you need to enable the feature from the feature preview management page.

How to watch a file for changes

Once enabled, you will see an "eye" icon on an currently embedded file. You currently cannot watch an entire directory. If this option on the row is not available - this file is not available for watching.

If you add the same file in any other workspace you will notice the file is automatically watched. If you delete the document totally from the system, it will automatically be unwatched.

Manage and observe watched files easily

Any watched file is checked hourly if it is stale. A stale file is any file that has not had its content refreshed in the last 7 days.

In the future, you will be able to force-refresh a document or change its default stale time.

Summary and notes

Watching a file with AnythingLLM's Automatic Document Sync will periodically fetch and replace all embeddings of that document across all of your active workspace.

This requires use of the connected embedder and therefore you may want to only watch a few files for resource reasons or cost concerns.

Currently, if you close the application or docker container, the watched files will not be synced as the background worker does not run if the process is killed.

Troubleshooting

If you are having issue with the document sync feature simply disable the toggle for the feature and it will not run any background workers while using AnythingLLM or on reboots.

Please ping the core team with a GitHub issue or Discord message for any questions or bug reports.