今日已更新 200 条资讯 | 累计 27375 条内容
关于我们

Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt

pera 2026年08月02日 03:04 1 次阅读 来源:HackerNews

At the beginning of the year I decided to set up a scraping and LLM honeypot on one of my personal websites which included a fake git repo with code containing fake HTTP endpoints. The address to this repo was hidden in a public page inside a comment. About three weeks ago IP addresses from Amazon Searchbot attempted to make requests to the fake endpoints included inside a shell script. My robots.txt explicitly includes Amazonbot. I am honestly surprised that this is coming from Amazon. Is this

At the beginning of the year I decided to set up a scraping and LLM honeypot on one of my personal websites which included a fake git repo with code containing fake HTTP endpoints. The address to this repo was hidden in a public page inside a comment. About three weeks ago IP addresses from Amazon Searchbot attempted to make requests to the fake endpoints included inside a shell script. My robots.txt explicitly includes Amazonbot. I am honestly surprised that this is coming from Amazon. Is this kind of behavior legal?
本文内容来源于互联网,版权归原作者所有
查看原文