Which crawlers get Base44's pre-rendered pages?

Base44 pre-renders for the crawlers that matter most for search and AI answers: Googlebot, Bingbot, Applebot, DuckDuckBot, and the crawlers from OpenAI, Anthropic and Perplexity, plus the main link-preview bots. It doesn’t pre-render for Google’s inspection tool, GoogleOther, Bravebot, Meta’s AI crawler, Amazonbot, Common Crawl, Bytespider, Mistral, or SEO audit tools. Those always get Base44’s generated summary.

We found this by requesting pages that already had a render with 33 different user-agent strings. The split held on every page but one, which served its render to every visitor.

The full list#

“Yes” means the crawler got the rendered page when one existed. When no render was ready, even recognized crawlers got the summary. See how Base44 pre-rendering works.

The ones that matter#

Google-InspectionTool powers Search Console’s Test live URL and the Rich Results Test. Because it gets the summary, those tools can show a very different page from the one Googlebot stored. The Search Console guide explains how to read them.

Bravebot builds Brave Search’s own index, which Brave licenses to other products for web results. On Base44 it indexes summaries.

Meta-ExternalAgent crawls for Meta’s AI products. Meta’s link-preview fetcher gets renders, so shared links look right, but Meta’s AI crawler reads the summary.

CCBot feeds Common Crawl, a public archive widely used for AI training data. Base44 pages enter it as summaries.

SEO tools explain a lot of confusing audit reports. If Screaming Frog, Ahrefs or Semrush shows every Base44 page with a near-identical heading and body, it’s reading the summary. Set the tool’s user agent to Googlebot to see what Googlebot gets.

Serving every crawler#

If the “No” rows matter to you, Base44’s settings can’t change them. A pre-rendering service in front of your domain decides for itself which user agents get rendered HTML, so you can include Bravebot, Meta’s crawler, Common Crawl and anything else. See the options guide.

Test it yourself#

for ua in 'Googlebot/2.1' 'bingbot/2.0' 'GPTBot/1.2' 'ClaudeBot/1.0' 'PerplexityBot/1.0' 'Bravebot/1.0'; do
  printf '%-20s ' "$ua"
  curl -s -A "Mozilla/5.0 (compatible; $ua)" https://yourdomain.com/ \
    | grep -q 'id="seo-snapshot"' && echo summary || echo rendered
done

Run it on a page that comes back rendered for Googlebot, so there’s a cached render to compare against.

What to do next#

1. Run the loop on your homepage and one inner page.
2. Decide whether any “No” crawler matters to your traffic.
3. If one does, read the pre-rendering options.