Overview
When you add a website as a source, Talkative spins up a headless Chrome instance in AWS to load the page exactly as a real browser would, then extracts all of the visible text content from it. This approach means the scraper can handle modern, JavaScript-rendered websites, not just static HTML.
Because this runs on AWS Lambda, each scraping request can come from a different, non-fixed IP address. There is no single IP (or IP range) that can be whitelisted on your side.
Authentication
To identify legitimate Talkative scraping requests, every request includes a custom header:
x-talkative-auth: <your-company-uuid>
- The value is a UUID unique to your company.
You will need to contact Talkative Support to confirm the UUID assigned to your company.
What You Need to Do: Whitelisting
Since there's no fixed IP to allow, you'll need to configure your website/hosting infrastructure (firewall, WAF, CDN, etc.) to whitelist any request that carries a valid x-talkative-auth header with your company's UUID, so it can bypass any protective measures that would otherwise block it.
Common Causes of Blocked Scrapes
If a scrape is failing or returning incomplete content, it's usually because one of the following is blocking the request before it reaches your site's content:
- Rate limiters – treating repeated requests as abusive traffic.
- DDoS protection – flagging the Lambda's changing IPs as suspicious.
- Bot/scraping protection (e.g. Cloudflare, Akamai, etc.) – blocking automated, non-browser-like traffic by default.
Whitelisting the x-talkative-auth header should be added as an exception rule in each of these systems.
Customising What Content the Bot Receives
By default, the scraper captures the website exactly as it's served — meaning menu text, footer text, navigation links, and other boilerplate get pulled in alongside the actual page content.
This isn't essential to fix: the bot's RAG search simply ignores irrelevant content when answering queries, so it won't affect answer quality. However, if you have the development capacity, you can reduce the size of your Knowledge Base by serving a trimmed-down version of each page specifically to the scraper.
Since every scraping request carries the x-talkative-auth header, your website/server can detect it and respond with just the relevant text content (stripping out menus, footers, and other repeated boilerplate) instead of the full page.
- This is entirely optional — the bot works fine without it.
- It's most useful for larger websites, where boilerplate repeated across many pages can otherwise bloat the stored Knowledge Base significantly.