How to test what Googlebot and AI crawlers get from a Base44 app

To check a Base44 page, request it with a crawler’s user agent and look for Base44’s fallback block, id="seo-snapshot". If the block is in the response, the crawler got an auto-written summary. If it’s missing and your page’s real text is there, it got the pre-rendered page. Base44 decides by user agent, and every recognized crawler gets the same cached copy, so this test shows what that crawler is served right now. Run it again after each publish, since renders come and go.

For the background, see how Base44 pre-rendering works.

Test 1: one page, one crawler#

UA='Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
curl -s -A "$UA" https://yourdomain.com/pricing -o page.html
grep -c 'id="seo-snapshot"' page.html        # 1 = summary, 0 = rendered
grep -ci 'a phrase from your real page' page.html

Use a phrase that only exists in the page’s real content, like a product name or a sentence from the body. The summary never contains it.

Test 2: every page in your sitemap#

UA='Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
for u in $(curl -s https://yourdomain.com/sitemap.xml | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g'); do
  curl -s -A "$UA" "$u" | grep -q 'id="seo-snapshot"' && echo "SUMMARY   $u" || echo "rendered  $u"
  sleep 1
done

Every SUMMARY line is a page crawlers currently get as Base44’s generated summary. Requesting it again won’t change that: in our tests, pages requested every 10 seconds for four minutes stayed on the summary. In a test of 47 live Base44 sites, 31% of inner pages were summary-only after two passes, and a day later 13 of 44 rendered pages had gone back to the summary. The monitoring guide turns this into a script you can schedule.

Test 3: several crawlers#

for ua in 'Googlebot/2.1' 'bingbot/2.0' 'GPTBot/1.2' 'ClaudeBot/1.0' 'PerplexityBot/1.0' 'Bravebot/1.0' 'meta-externalagent/1.1'; do
  printf '%-24s ' "$ua"
  curl -s -A "Mozilla/5.0 (compatible; $ua)" https://yourdomain.com/ | grep -q 'id="seo-snapshot"' && echo summary || echo rendered
done

On a page with a render, the first five should say rendered, and Bravebot and Meta’s crawler summary. That’s Base44’s recognition list; see the crawler list.

Test 4: a record page#

Pick a page that shows one record, such as a product or listing:

curl -s -A "$UA" 'https://yourdomain.com/ProductDetail?id=abc' \
  | grep -oE '<title>[^<]*</title>|<meta name="description"[^>]*>|id="seo-snapshot"'

If the title is the generic page name and the same on every record, see the record pages guide.

Test 5: the sitemap#

curl -s https://yourdomain.com/sitemap.xml | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g'

Look for two things: utility pages that shouldn’t be there (login, reset-password, admin), and record pages that should be but aren’t. Base44’s sitemap lists page files only.

Test 6: link previews#

curl -s -A 'Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)' https://yourdomain.com/pricing \
  | grep -oE '<meta (property|name)="(og|twitter):(title|description|image)"[^>]*>'

Each shared page should have its own title, description and image.

What to do next#

1. Run tests 2 and 4 on your own site.
2. Note which pages stay on summary and whether record titles are generic.
3. Pick a fix from the options guide.