Google published a technical guide on 17 November on how to let Google-Agent, the agent that browses and acts on websites for Google users, past a firewall or CDN. It lists 6 reasons verified requests still get blocked, and the IP file it points to held 20 address blocks on 16 November.
This is a job for whoever runs your firewall or CDN, not for your SEO settings, and it is worth an email to them this week. A blocked agent leaves no enquiry and no error in your reports, so if a customer asks Google to check your availability and the agent hits a CAPTCHA, the only sign is the business you never hear about.
What happened
Google added a page called Allow Google-Agent traffic to its crawler documentation on 17 November. Google-Agent is the name for agents running on Google's own infrastructure that navigate websites and carry out actions when a user asks them to. It has existed since March, but this is the first guidance on letting it in. Google says to verify 3 things: a user agent string containing Google-Agent, the IP address against Google's user-triggered-agents.json file, which listed 20 address blocks on 16 November, and a cryptographic signature where present, which Google is still testing on only some requests. Requests that pass should skip bot challenges. Google also lists 6 reasons verified requests still fail, among them allow rules that run after broad bot blocking policies, CAPTCHAs on search and forms, and proxies that strip the original IP address. Robots.txt does not apply, because the agent acts for a named person rather than crawling on its own.
What it means for you
Access is decided at your firewall, CDN or bot detection service, so your host or developer is the person who can fix it. Google has not said which of its products use this agent or how much traffic it sends, so this is cheap insurance rather than a new channel. The cost of getting it wrong is quiet: the agent is challenged, and the customer's answer comes from a site that let it through.
What to do
- Ask your host or CDN provider whether they verify Google-Agent already, and if so, switch the allow rule on.
- Check the rule covers search, filters, product pages, booking and cart steps and the API calls behind them, not only the homepage.
- Check your load balancer passes the original IP address and user agent through unchanged, and exempt verified ranges from rate limits.
Google crawler documentation
Worth doing something about this week.
Everyone's selling AEO. Here's what it actually is.
More signals
Get a focused review of how your business is found, where competitors win attention, and what to change to generate more of the right enquiries. You leave with a plan you can act on, whether we do the work that follows or not.
