The proxy industry needs you to never open the network tab

I run 75 scrapers in production. Three of them do any fingerprint spoofing. Maybe five use residential proxies. The rest run on plain datacenter IPs or no proxy at all, and they have been running for months. If you learned scraping from blog posts, that number probably sounds wrong to you. Every tutorial you have read starts the same way: sign up for a residential pool, install a stealth browser, randomize your fingerprint, throttle like a human. Then, on step five, you finally get to look at the actual website. That order is backwards, and it is backwards on purpose. Proxy companies write most of the scraping content on the internet. They were never going to write "you probably do not need us." The scraper with the $80 a month costume Last month my guy sent me his Greenhouse job board scraper to fix. It had everything. Puppeteer with the stealth plugin. Rotating residential proxies. Randomized mouse movements between actions. Human-like typing delays. It still kept dying. So I did the thing nobody had done: opened the page in a normal browser with devtools up. The entire job list was sitting in one XHR request to a public JSON endpoint. No auth. No cookies. A rate limit so loose I never managed to hit it. I deleted basically his entire codebase and replaced it with a fetch call. It has not broken since. He had been paying for proxies for months to hit an endpoint that does not care who you are. This was not a rare lucky case. This is most cases. The 20 minute method What I do on every new target, before writing a single line of code: Open the network tab, filter to XHR/fetch. Reload the page. Click around. Paginate. Search. Find the request that returns the actual data. It is usually JSON and usually obvious. Right click, copy as cURL. Paste it in a terminal and start deleting headers one at a time. Rerun after each delete. Whatever survives step five is your scraper. Most of the time the answer is a user agent header and nothing else. Sometimes a referer. Occasion