Can AI Assistants Read Your Website? A Check of 18 Cleaning-Company Sites
A tools vendor ran 18 cleaning-company websites through its free Site Check tools. 9 returned their home page; of those, 1 blocked 12 of 14 AI crawlers in robots.txt, 7 had no share image, and only 2 had all 8 basic search and share tags.
Who this is about: Local cleaning companies (an audit of 18 sites by a tools vendor)
What it takes
- Cost
- FreeFree within rate and daily limits, with no sign-up or API key.Price checked Oct 15, 2026 on tools.openkrill.app. Prices change; check before you buy.
- Setup time
- Under an hourOur estimate from the source, for one business.
- Skill needed
- Some tech skillsComfortable installing an app, pasting an API key or running a command.
Explain it simply
AI assistants like ChatGPT and Claude send programs called crawlers to read websites, and a site's robots.txt file can tell them to stay out, sometimes by mistake. This vendor checked 18 cleaning company websites to see whether AI crawlers were blocked and whether basic page labels were in place, such as the description and the picture that show up when someone shares a link. Most sites weren't blocking AI, but many were missing those labels, which are quick to add.
What was done
OpenKrill took every domain with "cleaning" in the name from the Tranco top 1 million list, removed registries, shops, blogs and other non-service sites, and kept 18 cleaning firms. On 2026-10-10 it ran two tools on each:
check_ai_crawler_access reads robots.txt, headers and redirects for 14 named AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot.
check_page_tags looks for 8 tags: meta description, canonical link, mobile viewport, og:title, og:description, og:image, og:url and twitter:card. The og tags control what people see when a link is shared.
How it went
9 of 18 sites sent back their home page; 4 refused the vendor's server (403) and 5 gave a CDN error or no answer, so those weren't counted as blocked. Of the 9, 1 blocked 12 of 14 AI crawlers in robots.txt, 7 had no og:image, 5 had no share title and description, 3 had no canonical link, 2 had no meta description and 2 had no mobile viewport tag. Only 2 had all 8 tags.
- The vendor calls it "a small snapshot, not a survey." A robots.txt check can't see firewall or CDN rules that turn bots away, and fixing these gaps doesn't make an AI assistant recommend you. It only makes sure the door isn't shut by mistake. The post also promotes the vendor's paid audit.
Prompt
1. Open https://[yoursite]/robots.txt. Look for "Disallow: /" under a bot you didn't mean to block (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended...).
2. View your home page source and confirm: <title>, meta description, canonical link, mobile viewport, og:title, og:description, og:image, og:url, twitter:card.
3. Or, with the Site Check tool added to Claude/ChatGPT/Cursor, ask:
"Can AI crawlers read example.com?"The do-it-yourself check from the post, in our words.
Business fit
Who this helps, and how:
Cleaning, plumbing and other local service companies: check that AI assistants and search engines can read your site.
Local & serviceSmall business owners: add the share tags so your link looks right when customers send it to each other.
Local & serviceWeb freelancers: run this quick check on client sites as a starting audit.
Agencies
Related
- Evidence grade BClaude in Chrome Set Up Four Google Ads Campaigns From a Keyword ListStrong evidence, using Claude, Google Ads and Semrush
- Evidence grade DClaude Picked a Slow Product and Segment From Omnisend Data, and the Email Brought In $26,619Light evidence, using Claude, Omnisend and Shopify
- Evidence grade CRun SEO with Claude doing the work and Search Console data doing the thinkingVendor or secondhand, using Claude Code and Claude
- AI for email campaigns and adsDepartment hub: AI for marketing, 26 examples
Similar tools
Other tools for this job, and how they differ. Chosen by our editors, never paid for.
- Screaming Frog SEO Spider
Crawls a site and reports meta tags, share images and robots.txt rules at scale.
- Ahrefs Webmaster Tools
Free site audits for sites you own.
- Cloudflare
If the site uses Cloudflare, its dashboard can show and control which AI crawlers are allowed.
“Vendor or secondhand” (grade C) means: A tool maker's case study, a promotional demo, or someone else's write-up. How we grade. Spotted a mistake? Suggest a correction.