Ask HN: Non-Malicious Bots
When I ask an AI to load a link, sometimes it can load it, and sometimes it can't (and I have to copy paste the text myself). Sometimes it can load it even when it has the "checking if you're a malicious bot". I'm wondering how that works. Isn't AI scraper traffic literally the thing that page is designed to stop? Do they let Google, OpenAI etc. user agents through? Or do those companies just have a bunch of guys in a warehouse somewhere with Chrome open, forwarding the AI's HTTP reque
Read original source
Source record
Recorded metrics
points · 1 votes
2026-11-07 13:45:32.660Z
cumulative since publication
Research analysis
Reported problem
Source account: When I ask an AI to load a link, sometimes it can load it, and sometimes it can't (and I have to copy paste the text myself). Sometimes it can load it even when it has the "checking if you're a malicious bot". I'm wondering how that works. Isn't AI scraper traffic literally the thing that page is designed to stop? Do they let Google, OpenAI etc. user agents through? Or do those companies just have a bunch of guys in a warehouse somewhere with Chrome open, forwarding the AI's HTTP reque
Frequency & alternatives
Recorded author: andai. Source votes: 1 votes (Nov 7, 2026 UTC; cumulative since publication) Recent records to compare: “Ask HN: Is anyone running multi/always-on agents that don't cost a fortune?” (Hacker News, Nov 6, 2026); “Ask HN: Engineers Focus Using AI” (Hacker News, Nov 6, 2026). Shared themes are navigation hints; they have not been verified as the same need or independent corroboration.
Demand hypothesis
Treat the reported task or obstacle as an interview lead. A feature request, complaint or search for alternatives does not by itself identify a budget owner or willingness to pay.
Commercial evidence gaps
Missing evidence: distinct affected users, repeated occasions, current workaround, measured time or money cost, purchasing authority and a response to a concrete offer.
Validation decision
Group original reports by the same task and deduplicate authors and reposts. Ask how the task is handled today and what it costs; test a defined offer before claiming paid demand.