Desktop Only Feature (v1.8.3+)
The Authenticated Scraping tool is exclusively available in the AnythingLLM Desktop application!
Authenticated Scraping
Security Note All credentials and session data are stored locally on your machine. AnythingLLM never transmits or stores your login information outside of your local machine.
The Authenticated Scraping tool enables you to access and scrape gated online content from websites or services that require authentication but hold critical contexual content you might want to use in your workflows, such as your personal LinkedIn feed or internal company portals that you have access to.
Your LLM can now access these websites and scrape and view content just like you would in a regular browser!
Key Features
- Secure Session Storage: Credentials are stored locally using isolated browser sessions
- Session Persistence: Login sessions persist between app restarts until explicitly cleared or the authentication expires for the associated service
- Isolated Environment: Separate from your actual web browser
- Full User Control: Clear stored data or sessions at any time with a single click
Using the Authenticated Scraping Tool
Accessing the Tool
- Open AnythingLLM Desktop
- Navigate to Settings > Tools > Browser Tool
- Click "Open Private Browser" to launch the isolated browser window
Authentication Process
- Log into your desired service (e.g., LinkedIn, Gmail) through the Authenticated Scraping tool.
- Your session will persist until you explicitly clear the browser data or the authentication expires for the associated service.
- AnythingLLM can now access authenticated content from these services when scraping or via agentic workflow execution.
- The returned content will be text only. No images, videos, or other media will be returned.
Heads up! The Authenticated Scraping tool is not a magic bullet. It is a tool that allows you to access authenticated content from websites that require authentication. It cannot currently interact with the content of the page you are accessing (eg: browser automation, RPA, etc).
Managing Browser Data
- Clearing Data: Use the "Clear Browser Data" button to remove all stored credentials and sessions
- When should you clear the browser data?:
- When switching between different service accounts
- If you encounter authentication issues when the LLM tries to access the site you want to scrape
Common Use Cases
Warning! Some web services may not you doing this. While the private browser tool is by any proxy, just a regular browser, some services may not like it. So use this tool at your discretion to prevent whatever service you are trying to access from blocking your account!
- Scraping your personal linkedin profile or feed.
- Accessing internal company documentation that is behind a login or SSO portal.
- Collecting or accessing data from paid or authenticated web service you have access to normally.
Troubleshooting
If you encounter issues:
- Clear the browser data and try again
- Ensure you're fully logged into the service by opening the private browser and navigating to the site you want the LLM to access.
Some services have very short lived sessions, those services may require you to log in again after a certain amount of time or might be a bad use-case for this tool. You can always re-authenticate with the service by opening the private browser and navigating to the site you want the LLM to access and logging in again to refresh the session.