How Google Sees Your Site
With Its Many Little Artificial Googly Eyes
I keep stumbling over different Google bots in my logs. So today I'm looking at each and every Googly eye that takes a look at my site. Because when you stare at me, I'll stare back.
You have the right and the means to define what Google is allowed to do on your site, whether the Google traffic you face is extensive or you'd like to opt-out of AI training. Sadly, using robots.txt is not enough, Google admits that some of their bots will not respect the rules.
Previous articles were about TLS fingerprinting JA3/JA4 and how to deploy it on HAProxy. In this article/marimo notebook (full source) I'll use the HAProxy logs containing IP addresses (plus ASNs), user agents and JA4 hashes to examine Google traffic. We'll see a couple of Google bots that are not mentioned in their docs.
Let's get into it.
This article doubles as a marimo notebook (of course, it's not modifiable and executable, that's a security nightmare to setup). Full source here. So to kick things off, I'll do some arbitrary global imports. There are a couple dependencies like my own asn-check (source, article) and requests.
import csv
from ipaddress import ip_address, IPv4Network, IPv6Network, IPv4Address, IPv6Address
from asn_check import ASNChecker
from asn_check.ip_binary_tree import IPTree
from typing import Union
import requests
import dataclasses
from enum import Enum
import re
from collections import defaultdictGetting Data: HAProxy Logs of This Blog
We need logs. First, I set up HAProxy (check out the previous part). Then I parsed the logs into something more sane, e.g. like this. That "format" optimizes lookups, so I've forced it to CSV for easier processing. Here's the CSV file. Here's a sample of what we're working with:
with open('haproxy_logs/stats.csv') as f:
cr = csv.DictReader(f)
data = [row for row in cr]
data[0]{
"ja4":
"t13d591000_a33745022dd6_1f22a2ca17c4"
"user_agent":
"Uptime-Kuma/1.23.15"
"ip":
"161.97.71.61"
"count":
"157907"
}Generally, the signals to look for are IP addresses, User-Agents and the overall traffic. My new signal here is JA4, which I hope will give us some clarity on classifying these bots.
ASN Checks - IP Trees
Another useful signal is ASN (Autonomous System Number). If IP is an address, then ASN is a city or a ZIP code. I have my own lib to get these numbers, so let's fetch them. Initialization is done in a separate cell, it takes some time (~4min) to build the trees.
asn_checker = ASNChecker()The benefit is that consequent searches are fast1, so we can replay them as we'd like. And it's already paying off, since there's trash even in the IP fields!
for r in data:
try:
asn = asn_checker.search(ip_address(r["ip"]))
r["asn"] = asn["asn"]
r["as_name"] = asn["name"]
r["country_code"] = asn["country_code"]
except ValueError as e:
print(r)
print(e){'ja4': 't13d250900_b78ed14e2fd0_e7c285222651', 'user_agent': '', 'ip': '} "SSTP_DUPLEX_POST /sra_{BA195980-CD49-458b-9E23-C84EE0ADCD75', 'count': '11'}
'} "SSTP_DUPLEX_POST /sra_{BA195980-CD49-458b-9E23-C84EE0ADCD75' does not appear to be an IPv4 or IPv6 address
{'ja4': '', 'user_agent': '', 'ip': '} "GET /graphql?query=+{customerDownloadableProducts+{+items+{+date+download_url}}+', 'count': '1'}
'} "GET /graphql?query=+{customerDownloadableProducts+{+items+{+date+download_url}}+' does not appear to be an IPv4 or IPv6 address
{'ja4': 't13d1412h2_e33ad33b3d25_6b314db333b6', 'user_agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.7 Safari/537.36', 'ip': '2a06:98c0:3600:', 'count': '1'}
'2a06:98c0:3600:' does not appear to be an IPv4 or IPv6 address
{'ja4': 't13d1412h2_e33ad33b3d25_6b314db333b6', 'user_agent': "Don't Hype Me RSS Reader/1.0", 'ip': '2a06:98c0:3600:', 'count': '1330'}
'2a06:98c0:3600:' does not appear to be an IPv4 or IPv6 address
{'ja4': 't13d1412h2_e33ad33b3d25_6b314db333b6', 'user_agent': 'minifeed_net', 'ip': '2a06:98c0:3600:', 'count': '491'}
'2a06:98c0:3600:' does not appear to be an IPv4 or IPv6 addressIn the log collection, I'm preferring the X-Forwarded-For header for the IP address, but that creates the opportunity to feed trash into this header in an attempt to break/exploit the parser of these data. Well, here I am, writing my parser around that.
We see the 2a06:98c0:3600: entry a couple of times. That's an IPv6 prefix, not a full address, so the lib is complaining rightfully. It belongs to Cloudflare and it should be a reverse proxy. Seems that some Cloudflare customers are abusing this. I imagine you can order a reverse proxy to point it to a target you don't own. Then if you generate traffic to that proxy, the target gets hit. The key difference is that this IP is usually white-listed and therefore these attacks are hard to block. Internet is fun!
Log Entry Dataclass
Let's finish this stage of parsing by defining a log entry:
@dataclasses.dataclass(frozen=True)
class LogEntry:
ja4: str
user_agent: str
ip: Union[IPv4Address, IPv6Address]
count: int
asn: str # not all ASNs might be numbers and we are not using any number operations on them.
as_name: str
country_code: strAnd let's also finish up the data clean-up by wrangling our data into this format. I employ some dirty tricks for brevity. The kwargs trick is especially useful with a dictionary -> dataclass transformation, but you have to conform to the format exactly.
log_entries = list()
for entry in data:
try:
new_entry = dict(entry)
new_entry["ip"] = IPv4Address(entry["ip"]) if ":" not in entry["ip"] else IPv6Address(entry["ip"])
new_entry["count"] = int(entry["count"])
log_entries.append(LogEntry(**new_entry))
except Exception as e:
print(f"Malformed entry {entry}, exception {e}")
log_entries[0]LogEntry(
ja4='t13d591000_a33745022dd6_1f22a2ca17c4',
user_agent='Uptime-Kuma/1.23.15',
ip=IPv4Address('161.97.71.61'),
count=157907,
asn='51167',
as_name='CONTABO Contabo GmbH',
country_code='DE'
)Google Assets
In this section, let's map what we know about Google assets to identify traffic from Google.
Declared Google Bots
Google has a lot of bots. Let me manually extract the "robots.txt" versions of user-agents. That should be an immutable string that appears in all versions and flavors of that particular bot. Here:
google_bot_ids = [
# https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
"Googlebot",
"Googlebot-Image",
"Googlebot-Video",
"Googlebot-News",
"Storebot-Google",
"Google-InspectionTool",
"GoogleOther",
"GoogleOther-Image",
"GoogleOther-Video",
"Google-CloudVertexBot",
"Google-Extended",
# https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers
"APIs-Google",
"AdsBot-Google",
"AdsBot-Google-Mobile",
"Mediapartners-Google",
"Google-Safety",
"DuplexWeb-Google",
"Google Favicon",
"AdsBot-Google-Mobile-Apps",
"googleweblight",
# https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
"Google-CWS",
"FeedFetcher-Google",
"Google-GeminiNotebook",
"Google-NotebookLM",
"Google-Agent",
"GoogleMessages",
"Google-Pinpoint",
"GoogleProducer",
"Google-Read-Aloud",
"google-speakr",
"Google-Site-Verification"
]
assert all(["google" in x.lower() for x in google_bot_ids])Noticed something? Literally all of them have "google" in them in one way shape or form, so we can filter them out based on user-agents like so:
google_ua_entries = [x for x in log_entries if 'google' in x.user_agent.lower()]
google_ja4s = {x.ja4 for x in google_ua_entries}
google_ja4_uas = {x:{y.user_agent for y in google_ua_entries if y.ja4==x} for x in google_ja4s}And if you ask why didn't I use that nice list to check for the Google bots, I'll show you in a later section. Spoilers: there are more bots.
Google ASNs
Another view we can try is to take a look at ASNs owned by Google and the traffic coming from those IPs:
google_asn_entries = [x for x in log_entries if 'goog' in x.as_name.lower() or 'alphabet' in x.as_name.lower()]
google_asns = {(x.asn, x.as_name) for x in google_asn_entries}
google_asns{
('396982', 'GOOGLE-CLOUD-PLATFORM - Google LLC'),
('16591', 'GOOGLE-FIBER - Google Fiber Inc.'),
('394089', 'GCP-ENTERPRISE-USER-TRAFFIC - Google LLC'),
('15169', 'GOOGLE - Google LLC')
}Not much info is available online about the purpose of these ASNs. But let's try anyway.
ASN 16591: Google Fiber (IPInfo details)
This one is clear - it's google's internet offering. If you ask me, Google has already way too much control over browsing, this is then the ultimate final step: your internet connection is fully owned by Google. Of course, they need an ASN for their fleet of routers. What's nice that this is an ISP type ASN that is treated differently than hosting type ASNs.
ASN 396982: Google Cloud Platform (IPInfo details)
You can rent a piece of Google in their Google Cloud Platform offering. It's a cloud hyperscaler, but you know this. I expect a lot of trash coming from this ASN as people might abuse the free tiers.
ASN 394089: GCP Enterprise User Traffic (IPInfo details)
This one I suspect is what Google can give you if you have assets in GCP plus a better contract. Otherwise I expect the same as in Google Cloud Platform ASN396982. As they say, if you have to ask, you can't afford it.
ASN 15169: Google (IPInfo details)
That leaves us with this general ASN. I can only presume that this is the traffic from all the rest of Google Internal services. It's explicitly not traffic from GCP workloads and from users hooked up to Google Fiber ISP. I make the conclusion that this is "official Google traffic." I've found this list of prefixes by searching in their docs.
To take a break from python, here's the breakdown after I ran a sophisticated bash analytics pipeline:
$ wget -O- https://www.gstatic.com/ipranges/cloud.json 2>/dev/null | \
jq | grep 'ipv4Prefix' | cut -f2 -d':' | grep -o '"[^"]*"' | tr -d '"' |\
sed 's?./.*$?9?' | asn-check |\
cut -f3 -d',' | sort | uniq -c | grep -v 'name'
909 GOOGLE-CLOUD-PLATFORM - Google LLC
81 GOOGLE - Google LLC
7 LEVEL3 - Level 3 ParentIt's at this point when we can count the percentage of traffic coming from Google to see how it stacks against the rest of the internet. Note, that I have counts of requests in the LogEntry, now's the time to use them.
I'll also exclude traffic from my monitoring tool (Uptime Kuma) as it would probably dominate the conversation.
no_monitor = [x for x in log_entries if x.user_agent != "Uptime-Kuma/1.23.15"]
total_sum = sum([x.count for x in no_monitor])
google_sums = {f"{x[0]}: {x[1]}":sum([y.count for y in log_entries if y.asn==x[0]]) for x in google_asns}
percentages = {x:f"{google_sums[x]} -> {round(google_sums[x]*100/total_sum,2)}%" for x in google_sums}
percentages{
"396982: GOOGLE-CLOUD-PLATFORM - Google LLC": "9617 -> 3.09%",
"16591: GOOGLE-FIBER - Google Fiber Inc.": "116 -> 0.04%",
"394089: GCP-ENTERPRISE-USER-TRAFFIC - Google LLC": "88 -> 0.03%",
"15169: GOOGLE - Google LLC": "1239 -> 0.4%"
}Not gonna lie, I was hoping for a bigger number here so that I could have a great clickbait headline. But alas, seems that Google has its bots under control. The 3% at GCP ASN might very well be Google customers, not Google and still it isn't that much.
Now let's take a look at who is trying to impersonate google:
google_as_numbers = [x[0] for x in google_asns]
google_ua_non_google_asn = [x for x in log_entries if x.asn not in google_as_numbers and "google" in x.user_agent.lower()]
imp_count = sum([x.count for x in google_ua_non_google_asn])
f"{imp_count} -> {round(imp_count*100/total_sum,2)}%"608 -> 0.2%Also a nothing burger. I have attempted science today. Sometimes the dramatic story just isn't there and now I have to think of another clickbaity headline.
Declared Google Traffic
Now let's take a look at user agents with "google" in them that also come from Google ASNs. That's the strongest signal we have for official Google traffic.
google_ua_set = {x for x in google_ua_entries}
google_asn_set = {x for x in google_asn_entries}
assert len(google_ua_set) - len(google_ua_entries) == 0 # based on log-collecting methods, this should be true
assert len(google_asn_set) - len(google_asn_entries) == 0
google_entries = google_ua_set & google_asn_set
f"{len(google_ua_set)} entries have 'google' in them, {len(google_asn_set)} entries come from google ASNs, {len(google_entries)} have both.""286 entries have 'google' in them, 4879 entries come from google ASNs, 148 have both."Let's break it down by the user-agent:
google_uas = {x.user_agent for x in google_entries}
google_uas_map = {x:[y for y in google_entries if y.user_agent==x] for x in google_uas}
google_uas_mapHere's the full list attached - I'll be analyzing it shortly and it'd break the flow of the article if I included it directly. The order is, of course, arbitrary. Also note, that some of these bots come from GCP rather than pure Google LLC. While Google might use GCP ranges for official traffic, it's more likely one of GCP customers impersonating Google.
GoogleBot, GoogleBot Image, GoogleBot Video
By far the most common bot, this is the one that brings your pages into the google search results (docs). Comes in various different flavors (desktop and mobile), I guess it wants to see if there's the same content for various user agents.
The Image and Video variants are similar, but they feed the multi-media results into the respective sections of Google search.
So if you want your pages to be searchable by THE search engine, you'll have to allow these bots.
BEWARE! These are also the most commonly impersonated bots on the internet. Since everyone wants to be searchable by Google, admins generally allow these user agents and the attackers will happily impersonate them. Take a look at these two entries:
LogEntry(ja4='t13d190900_9dc949149365_97f8aa674fd9', user_agent='Googlebot-Image/1.0', ip=IPv4Address('34.181.217.51'), count=1, asn='396982', as_name='GOOGLE-CLOUD-PLATFORM - Google LLC', country_code='US')
LogEntry(ja4='t13d181300_e8a523a41297_43ade6aba3df', user_agent='Googlebot-Image/1.0', ip=IPv4Address('66.249.74.4'), count=23, asn='15169', as_name='GOOGLE - Google LLC', country_code='US')One comes from GCP, so it might be anyone. The other comes from Google LLC so it seems to be directly from Google. The JA4 hashes are also different, which is a hint that this is weird. We'll meet the t13d190900_9dc949149365_97f8aa674fd9 hash again and I suspect this one is a spammer. So let's take a look at it:
Google Toolbar? Feed Fetcher? GoogleBot News? AdsBot? MediaPartners? IMPOSTORS!
For completeness, here's a quick breakdown of the listed bots:
- Google Toolbar: discontinued (check the Google graveyard fully, it's.. really something if you never seen it).
- FeedFetcher: legit google bot designed to fetch feeds using RSS.
- GoogleBot News: legit google bot similar to googlebot but providing info into the "news" segment of google search.
- AdsBot: legit google bot. Used for ads, specifically checking google ads quality. That's how I found these impostors - I'm not running Google Ads at all.
- MediaPartners: legit google bot. Used for google AdSense to provide relevant ads (or something IDK, IDC).
- Google Web Preview: Not mentioned in any google documentation I could find, the best I got is this archived link.
Also in this category there's a user agent BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0). I haven't found any WAP proxy service by google, but I found one reference for this user-agent.
Well actually these all come from Google GCP ASN, so they might be impostors. And if you'd look at the JA4 hashes, this one t13d190900_9dc949149365_97f8aa674fd9 is different from the usual t13d181300_e8a523a41297_43ade6aba3df. If I take even closer look, I see it in this batch:
... 35.245.141.163:47184 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Linux; Android 7.0; LGMS428) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.111 Mobile Safari/537.36|} "GET /backups/db.sql HTTP/1.1"
... 35.245.141.163:47196 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36|} "GET /tmp/backup.sql HTTP/1.1"
... 35.245.141.163:47176 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Mobile/14G60 MicroMessenger/7.0.5(0x17000523) NetType/4G Language/zh_CN|} "GET /backup/dump.sql HTTP/1.1"
... 35.245.141.163:47210 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0|} "GET /tmp/dump.sql HTTP/1.1"
... 35.245.141.163:47222 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13|} "GET /exports/db.sql HTTP/1.1"
... 35.245.141.163:47188 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/4.0 (compatible; GoogleToolbar 4.0.1019.5266-big; Windows XP 5.1; MSIE 6.0.2900.2180)|} "GET /backups/dump.sql HTTP/1.1"
... 35.245.141.163:47248 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Linux; Android 7.0; EVA-L09) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.111 Mobile Safari/537.36|} "GET /config.js HTTP/1.1"
... 35.245.141.163:47232 ... {t13d190900_9dc949149365_97f8aa674fd9|Peach/1.01 (Ubuntu 8.04 LTS; U; en)|} "GET /config.php HTTP/1.1"
... 35.245.141.163:47264 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (X11; Linux x86_64; en-US; rv:2.0b2pre) Gecko/20100712 Minefield/4.0b2pre|} "GET /config.yml HTTP/1.1"
... 35.245.141.163:47256 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36|} "GET /config.json HTTP/1.1"
... 35.245.141.163:47276 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 YaBrowser/17.3.0.1785 Yowser/2.5 Safari/537.36|} "GET /config.yaml HTTP/1.1"This is a textbook example of a random spray and pray vulnerability scan. So I conclude this is in fact not Google, but someone sending trash from GCP while rotating the user agents randomly.
Google AppsScript
There are a couple of Google AppsScript user agents, e.g. Mozilla/5.0 (compatible; Google-Apps-Script; beanserver; +https://script.google.com; id: UAEmdDd_1zdKXtniWEDkKiDKLGH6k2AjErCM)that are related to the Google Apps Script offering.
That means these are most likely not directly Google initiated, but Google-user initiated. I appreciate the inclusion of the script ID, but I didn't find a way to report a misbehaving script, so that's a minus from me.
Google NotebookLM, Google Agent, Gemini Deep Research
Google has a concept of "user initiated fetch" (see docs). So in a sense these are user-initiated but coming from the Google infra. I've observed these:
- Google NotebookLM: renamed to GeminiNotebook in August 2026, it's about fetching URLs provided as sources in those notebooks.
- GoogleAgent-URLContext: Not mentioned explicitly, but related to GoogleAgent. It's one of the tools available for the agents in the Gemini Enterprise Agent Platform.
- Gemini-Deep-Research: Seems to me like a user initiated action for deep research. Then there are the agent docs.
In a sense, these are AI bots. But the stated intent is to gather context to execute agents and not to train new AI.
Googlebot AMPHTML
So there's this user agent Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.96 Mobile Safari/537.36 (compatible; Google-AMPHTML) which can be identified by the Google-AMPHTML string. Apparently, this is a bot used to identify Accelerated Mobile Pages (AMP).
The issue is, I didn't find this one in the Google official documentation, nor did I find it in any google source when googling. Using reverse DNS on the 64.233.172.233 IP I get google-proxy-64-233-172-233.google.com which also gives me nothing.
Not cool google, not cool.
Google Safety
While it is recognized in the docs, there is only a general description of
The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties.
And I also love to see that:
The Google-Safety user agent ignores robots.txt rules.
So... if a link to my site somehow gets onto any Google property, they reserve the right to ignore my wishes for external crawlers to do what they need to do to ensure "safety" (whatever that means). A "Magic The Gathering" player in me recognizes the killer combo with the GoogleBot search spider. Yay.
Google (Yes, Just Google)
By far my favourite user agent on this list: Google. It's even better that it comes from the official Google IP range and not GCP. Like what do you want to know you silly admin? It's Google, that'll have to do.
So I guess hello John Google, enjoy your stay.
Looking at the reverse DNS for the 108.177.64.74 address I get rate-limited-proxy-108-177-64-74.google.com. Surely, Google wouldn't re-route the traffic and anonymize the User Agent when it detects rate-limiting (which I have) to ignore the website wishes. No, Google would never.
Google Other
Well of course, the Other bot traffic. The bot traffic that is other. This other traffic is unlike any traffic that we've seen. There are apparently other use cases that are not covered by the previous bots and product portfolio. There are other things Google wants to do. Don't worry, every other thing ends well.
The thing is, this one is actually recognized in the official documentation. By far my least favourite because it's so opaque. And keep in mind, most of these come from Google LLC ASN and not the GCP ASN, so it's "official". I think if you're such a big company like Google and you're setting multiple random bots my way, you should have the courtesy to explain your reasons. And not just "yea, we're looking at your site because we can, what are you going to do about it"?
Somehow, Cloudflare assigned the AI CRAWLER category to this bot. Cloudflare claims that this bot gathers data for AI training purposes. I don't know why, but I also don't know how to disprove this so... bear that in mind.
Undeclared Google Traffic
Recall there are these entries that come from Google ASNs and yet don't have "google" mentioned in their user agent. Let's break it down by ASN:
non_google_google_asn_set = [x for x in google_asn_entries if x not in google_entries]
sorted_non_google_google_asn = {x:[y for y in non_google_google_asn_set if y.asn==x[0]] for x in google_asns}
sorted_non_google_google_asnThe full list adds little to the article, but it can be found here for those interested. Once again, let's break it down into parts.
GCP Traffic (Both AS396982 and AS394089)
As expected, most of the trash comes from GCP where anybody can run anything. By trash I mean things like:
- Automated WordPress vulnerability Scanners (
xmlrpc.phpis my top 404), - Forced browsing attempts (trying things like
/.env,/backup.sqland others), - Completely random and obtuse bots (probably a project for scraping that went nowhere)
- etc..
Maybe the most fun info here is seeing Claude Code in the Enterprise GCP tier, confirming their claims of partnership. But ask yourself again, why would Claude Code, primarily a desktop application live in a cloud server?
I plan to do a proper analysis of that traffic, but as I was digging around this I've hit Google so many times that I'm starting here and excluding all of the weird Google quirks from the main article. Sorry for now and TBD.
Google Fiber
Not much to see here, it's a small provider and it provides somewhat sane traffic.
There is one fun user-agent though:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0As per this question, this is a service of Apple that pre-fetches links for previews. I'd appreciate if Apple docs could confirm this, but it is what it is.
Google LLC Traffic
There is some weird traffic coming also from Google LLC. Recall, these shouldn't be Google customers, but Google itself as we've covered the other cases previously. Let's group it by user-agents:
uas_15169 = {x.user_agent for x in sorted_non_google_google_asn[("15169", "GOOGLE - Google LLC")]}
official_google_traffic_lol = {x:[y for y in sorted_non_google_google_asn[("15169", "GOOGLE - Google LLC")] if y.user_agent==x] for x in uas_15169}
{x: len(official_google_traffic_lol[x]) for x in official_google_traffic_lol}{
"Chrome Privacy Preserving Prefetch Proxy": 26,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36": 84,
"": 3,
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36": 12,
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Mobile Safari/537.36": 20,
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36": 1,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36": 1,
"Mozilla/5.0": 2,
"Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36": 1,
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36": 1,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36": 39,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36": 44,
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Mobile Safari/537.36": 5,
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_3_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3.1 Mobile/15E148 Safari/604.1": 1,
"Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36": 1,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36": 1,
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36": 16,
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36": 17,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.4; rb.gy/xprgqj) Chrome/124.0.0.0 Safari/537.36": 6,
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36": 3,
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36": 23,
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36": 1,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36": 1,
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36": 7,
"abuse.xmco.fr": 2,
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0 abuse.xmco.fr": 2
}For the first time in forever, I can call upon JA4 to group the rest of the entries:
ja4s_15169 = {x.ja4 for x in sorted_non_google_google_asn[("15169", "GOOGLE - Google LLC")]}
official_google_traffic_ja4 = {x:[y for y in sorted_non_google_google_asn[("15169", "GOOGLE - Google LLC")] if y.ja4==x] for x in ja4s_15169}
uas_ja4s = {x:list({y.user_agent for y in official_google_traffic_ja4[x]}) for x in official_google_traffic_ja4}
uas_ja4s_keys_sorted = sorted(list(uas_ja4s.keys()))
for _k in uas_ja4s_keys_sorted:
print(_k)
for _ua in uas_ja4s[_k]:
print(f" {_ua}")t13d1514h2_8daaf6152771_827b515c4f52
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36
t13d1516h2_8daaf6152771_02713d6af862
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36
t13d1516h2_8daaf6152771_806a8c22fdea
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36
Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Mobile Safari/537.36
t13d1516h2_8daaf6152771_d8a2da3f94cd
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36
Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36
Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Mobile Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36
Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36
t13d171000_5b57614c22b0_78e6aca7449b
Mozilla/5.0 (iPhone; CPU iPhone OS 18_3_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3.1 Mobile/15E148 Safari/604.1
t13d181300_e8a523a41297_43ade6aba3df
Chrome Privacy Preserving Prefetch Proxy
t13d190900_9dc949149365_e7c285222651
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0 abuse.xmco.fr
abuse.xmco.fr
t13d1909h2_9dc949149365_97f8aa674fd9
Mozilla/5.0
t13d5212h1_b262b3658495_8e6e362c5eac
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.4; rb.gy/xprgqj) Chrome/124.0.0.0 Safari/537.36
t13d5213h1_b262b3658495_66863fb0a24c
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.4; rb.gy/xprgqj) Chrome/124.0.0.0 Safari/537.36We can clearly see a couple of groups here. Even though the JA4 hashes are not matching exactly, the similarities are strong enough to predict some trends.
First is the Chrome privacy preserving prefetch proxy (CPPPP) that has a surprisingly unique JA4 hash of t13d181300_e8a523a41297_43ade6aba3df.
Then we have the t13d190900_9dc949149365_.* group that seems to be linked to abuse.xmco.fr, a French security scanner. While it's slightly surprising that this comes from the Google LLC ASN and not GCP ASN, the addresses are quite similar and I can imagine that this IP range was originally in the GCP. Or they did have some agreement with Google, IDK. It seems highly unlikely to me that Google would decide to impersonate this random French company in particular. Or it's just an impostor, like we saw above.
We also have the t13d5212h1_b262b3658495_.* group which has BuiltWith/1.4; rb.gy/xprgqj in the User-Agent. They have a site describing their crawler and similarly, I think this is associated with Google LLC only because of some shuffle of IP addresses between Google LLC ASN and GCP ASN.
The last group seems the most interesting to me. All user agents are covered by the t13d1514h2_8daaf6152771_.* fingerprint but you can see quite the variety of browsers and operating systems. While it is possible that all this hardware and software versions produce very similar fingerprints, my bet would be that Google is trying to see how the page behaves under different versions of Chrome for.... unknown reasons. UX? Legibility? General interest? I don't know.
While I didn't consent to this level of scrutiny, I have to admit that it has literally no impact on the site operations, so it's all good in my book.
Concluding: All Googly Eyes on Me
There are way too many bots looking at my page from Google. I really don't know why they need a bag of eyes staring at me, at this point it would be easier to just copy my page fully and then dissect it internally. They behave quite well though and it's not a problem for my site.
While there's only one bot specifically marked for AI training (Google Other), there's nothing really preventing Google from grabbing the other bot outputs for these purposes. And as you can see, fully blocking Google is hard, even if you accept not being searchable by the most prominent search engine.
As we saw, there are several good marks you can focus on if you want to block Google. These are:
- Google ASNs: 15169 (Google), 16591 (Fiber ISP), 396982 (GCP), 394089 (GCP Enterprise),
- User-agents with "google" in them,
t13d181300_e8a523a41297_43ade6aba3dfJA4 Fingerprint (YMMV),t13d1514h2_8daaf6152771_.*JA4 Fingerprint (YMMV).
With that out of the way, I'll take a look at the rest of the traffic next time.
The trick is organizing IP addresses to binary trees, where the prefix is known and suffix is variable. This plays really well with the CIDR notation, read more in my ASN Check article. Here's a quick "illustration":
192.168.50.102 = 11000000.10101000.00110010.01100110
192.168.50.102/24 = 11000000.10101000.00110010.????????