标签：#webscraping

I run a small AI/automation consultancy in Brazil, and a recent lead-research project needed the full follower list of a public Instagram profile — about 153,000 followers — plus enrichment (bio, public email/phone) to find business accounts worth contacting. The problem Pulling a list that size is never one API call. Instagram reports ~153,628 followers; you get them page by page, and any long-running extraction WILL hit a failed request eventually. If your pipeline can't resume, you start over from zero — which is expensive and slow. What I built The pipeline runs on n8n with Supabase as the datastore: Batched extraction — followers are downloaded in batches of up to 10,000 per cycle, on a schedule, instead of one giant run. Resume on error — every page cursor and count is persisted. When a request fails mid-run (in one run it stopped at 4,782 followers after 96 pages read), the job logs the error, emails me a status report, and picks up from the same point on the next cycle instead of restarting. Enrichment pass — a second workflow walks the stored followers and pulls profile details, flagging commercial accounts and any public email/phone in the bio. Personal/private accounts return no contact data, which the report counts separately. Email reports — each cycle sends me a summary: profile, followers reported vs. downloaded, pages read, batch name, and the exact error if one occurred. For the Instagram data layer I used HikerAPI — I tested a few other options first, and it won on pricing and rate limits for this volume. It handled the pagination fine: the run above made 100+ requests without me managing sessions or proxies myself. Tradeoffs / what didn't go perfectly Long extractions still fail sometimes (timeouts); resume logic is not optional at this scale, whatever API you use. Early days for me on this stack: so far it has worked well, but I'm still collecting more data before I'd call the pipeline battle-tested. I'll know more after a few full 150k-follower

Most local business lead scrapers on Apify are Google Maps scrapers with a website-crawling step bolted on. lukaskrivka/google-maps-with-contact-details is the most used (87,957 users, 4.63 stars). flash_scraper/local-business-leads is the outlier: it discovers businesses on OpenStreetMap instead of Google Maps, and includes MX email verification in its $3 per 1,000. Every figure below was read from Apify's public Store API ( GET /v2/store ) on 2026-09-05 — including every user count, so they are all on the same footing. The per-actor endpoint ( GET /v2/acts/<id> ) can read one higher: it gives flash_scraper/local-business-leads 33 rather than 32, and code-node-tools 33 as well. Prices, users and ratings change; the Pricing tab on each actor page is authoritative. Disclosure: I publish flash_scraper/local-business-leads , one of the actors compared here. Its limits are listed in the same detail as everyone else's, including the one that will disqualify it for many buyers. How prices are normalised These actors bill per event, and the events differ in kind, which makes headline prices misleading. Some charge per place found. Some charge separately for the website crawl that actually produces the email. Some charge again to verify that the email is deliverable. The table lists the primary per-result event multiplied by 1,000 at the free-plan rate , then names the add-on events, because a $5 per 1,000 place price with a $100 per 1,000 email-verification add-on is not a $5 tool. Paid Apify plans get tiered discounts on several of these actors, ours included — and on the add-on events the discount can be enormous. lukaskrivka's three $100-per-1,000 add-ons fall to $4.00 (email verification), $7.50 (lead enrichment) and $10.00 (social-profile enrichment) per 1,000 on Bronze, and lower again above it (Store pricing record read 2026-09-05). Our own free-plan-to-Diamond spread is about 30 percent. So if you are on a paid plan, re-read every figure below off the Pricing tab:

A browser and a language model can look at the same URL and effectively see two different things. A browser sees a rendered interface: navigation, cookie banners, buttons, ads, sidebars, images, scripts, interactive components, and eventually the text a human came to read. A language model sees whatever representation we decide to give it. That distinction matters when the URL is going into a RAG pipeline. Open the developer tools on any major news or documentation site and look at the raw HTML. A typical article page runs between 300KB and 800KB of markup. The article text itself is usually between 2KB and 10KB. The ratio of markup to content is consistently between 10:1 and 40:1 depending on how heavily templated the site is. When you pass raw HTML to a language model, you are passing all of it, and most pipelines treat this as an acceptable default. Perceive is the endpoint we built to fix that. You give it a URL. It returns clean Markdown. This post is about what happens in between and why we made the engineering decisions we did. Why raw HTML is a poor RAG input The token waste is real but it is not the worst problem. Three failure modes compound each other. Token waste . A blog post with 800 words of real content can run to 6,000–12,000 tokens as raw HTML once you include navigation, scripts, inline styles, and layout markup. The same content in Markdown is often 900–1,200 tokens. That is not just a cost issue. It is context window space that cannot go to content. Embedding contamination . Embedding models are trained predominantly on natural language. When you embed a chunk containing <div class="sidebar-widget__title">Related Articles</div> alongside the article content, the vector is pulled toward the markup semantics rather than the content semantics . The embedding does not cleanly represent the article; it represents a mixture of the article and the site's component naming conventions. Retrieval degrades as a result: chunks that should be semantically si

Introduction When you're scaling infrastructure or managing network security, proxies become essential tools—but they solve fundamentally different problems. A reverse proxy sits between your users and your backend servers, while a forward proxy sits between your users and the internet. This distinction might sound academic, but it shapes your entire architecture: from load balancing and security posture to compliance requirements and cost structures. Choosing the wrong proxy type can lead to bottlenecks, security vulnerabilities, or unnecessary infrastructure complexity. This article walks you through real-world scenarios, pricing considerations, and decision frameworks to help you deploy the right solution. Forward Proxies: Controlling Outbound Traffic What Forward Proxies Do A forward proxy intercepts requests from your internal network and forwards them to external servers on the internet. From the external server's perspective, the proxy is the client—the real origin of the request is masked or modified. Common use cases include: Employee internet access control : A company deploys a forward proxy so IT can block malicious domains, filter content, and enforce acceptable use policies Data residency compliance : A financial services firm routes all outbound API calls through a forward proxy in a specific geographic region to meet regulatory requirements Web scraping at scale : When extracting data from multiple websites, forward proxies rotate request sources to avoid IP-based blocking DDoS mitigation for outbound traffic : Distributed request aggregation through a forward proxy can reduce fingerprinting risks Pricing and Infrastructure Costs Forward proxies typically charge per: Concurrent connections : Enterprise solutions like Zscaler or Palo Alto Networks start around $5–15 per user/month Data transferred : Cloud-based forward proxies charge $0.05–$0.30 per GB, depending on geography and provider IP rotation : Proxy services offering residential IPs (for non-

Most football data scrapers on the market only extract high-level final scores (e.g. 2-1 ). But quantitative sports analysts, data scientists, and predictive betting modelers need granular data: Expected Goals (xG) , Official Referee Assignments , Goal Scorers paired with Assist Providers , and Half-Time vs Full-Time (1H/2H) statistical breakdowns . When I set out to build a professional-grade Flashscore scraper on Apify, I ran into two major engineering challenges: The Memory Problem : Keeping Puppeteer running to scrape hundreds of historical matches consumes over 1.5GB of RAM per run. The Protocol Problem : Flashscore serves its deep statistical feeds using a proprietary pipe-delimited data format ( ~ , ¬ , ÷ ) over CDN endpoints, rather than standard REST APIs. In this tutorial, I'll explain how I engineered the Flashscore Elite Statistics Extractor , how the hybrid Browser + HTTP/2 streaming pipeline drops RAM footprint from 1.5GB to 70MB , how to parse Flashscore's custom feed protocol, and how to pipe the resulting datasets directly into Python and Pandas. 🏛️ The Hybrid Pipeline Architecture To achieve zero proxy reliance for standard runs and ultra-low compute costs, the Actor splits execution into a 2-Phase Hybrid Pipeline : [ League & Season Selection ] │ ▼ ┌───────────────────────────────────────────┐ │ Phase 1: Browser Handshake (Puppeteer) │ │ - Captures x-fsign security tokens │ │ - Extracts countryId & tourId │ └─────────────────────┬─────────────────────┘ │ [ Immediate Browser Shutdown ] (RAM drops from 1.2GB -> 70MB) │ ▼ ┌───────────────────────────────────────────┐ │ Phase 2: Parallel HTTP/2 Feed Workers │ │ - got-scraping with JA3 TLS matching │ │ - Decodes df_st_1_ (Stats) & df_sui_1_ │ └─────────────────────┬─────────────────────┘ │ ▼ ┌───────────────────────────────────────────┐ │ Self-Healing Recovery Pass │ │ - Auto-retries skipped/failed matches │ └─────────────────────┬─────────────────────┘ │ ▼ ┌───────────────────────────────────────────┐

I built an OLX scraper for 24 countries — the boring version that actually ships OLX runs classifieds in about two dozen countries. Same brand, different domains, different anti-bot setups. Everyone scraping it does one country at a time. I got tired of forking. So I put 24 countries behind one input. country: "id" or country: "pl" or country: "br" — same schema out. It's live on Apify as primesieve/olx-global-scraper . One file. No browser. Here is the boring part that matters. What it does Input: { "country" : "id" , "keywords" : [ "iphone 13" ], "maxResults" : 50 , "maxPages" : 3 , "proxyConfiguration" : { "useApifyProxy" : true , "apifyProxyGroups" : [ "RESIDENTIAL" ] } } country — two-letter code ( id , pl , in , br , ua , pt , ro , bg , kz , uz , pk , za , ng , ke , eg , lb , ph , co , ar , pe , ec , gt , az , ma ). Default id . keywords — one or more search terms. Each runs sequentially. maxResults / maxPages — caps. Defaults 50 / 3, max 1000 / 30. proxyConfiguration — optional for Indonesia, required for the other 23. Output — same shape every country: { "listingId" : "123456789" , "title" : "iPhone 13 128GB mulus" , "price" : 6500000 , "priceText" : "Rp 6.500.000" , "currency" : "IDR" , "city" : "Jakarta Selatan" , "location" : "Tebet, Jakarta Selatan, DKI Jakarta" , "images" : [ "https://...jpg" ], "thumbnailUrl" : "https://...jpg" , "listingUrl" : "https://www.olx.co.id/item/123456789" , "country" : "id" } Title, price (numeric plus display text), currency, location, images, URL. No seller PII beyond what the listing page shows. No tricks. Try: https://apify.com/primesieve/olx-global-scraper The boring stack // no playwright, no puppeteer // apify + fetch + cheerio. That's it. The scraper is one file. Apify SDK for input, dataset, and pay-per-event. Native fetch for HTTP. cheerio for the HTML path. Undici ProxyAgent when a proxy is configured. Node 20, 512 MB, 600s timeout. I check the endpoint before I write the scraper. Indonesia answered with clean JSO

Google Trends has no public API. What it has is the same internal JSON endpoints the trends.google.com single-page app calls — and those endpoints do something most REST clients aren't built to survive: they answer with HTTP 200 and an empty body when Google decides you look like a bot. Quick answer A 200 OK from Google Trends' widgetdata endpoints does not mean you got data. If the response body is empty, Google soft-blocked the request without bothering to send a 429. The fix is to stop trusting the status code alone: check resp.text.strip() on every call that's supposed to return a payload, and if it's empty, rotate the proxy session and retry exactly like you would on a 429 — because that's what it functionally is. if status == 200 : if require_body and not resp . text . strip (): # Soft block: Google returns 200 with empty body when it detects bots. # Treat the same as 429 — rotate session and retry. logger . warning ( " %s: HTTP 200 but empty body (soft block, attempt %d/%d) " , ...) if proxy_cfg is not None : new_sid = _fresh_session_id () current_proxy_url = await proxy_cfg . new_url ( session_id = new_sid ) await asyncio . sleep ( delay ) continue return resp Why does a working response start with )]}' ? Every Trends JSON endpoint prepends an XSSI-protection prefix before the actual JSON body — a defence against cross-site script inclusion attacks that predates fetch() . Naive json.loads(resp.text) throws a JSONDecodeError on a perfectly healthy response. Worse, the prefix isn't even consistent: /trends/api/explore sends )]}'\n (no comma), some widgetdata endpoints send )]}',\n (with a comma). We check the longer variant first so a response using the short prefix doesn't get mis-stripped: XSSI_PREFIX = " )]} ' , \n " XSSI_PREFIX_NO_COMMA = " )]} ' \n " def _strip_xssi_prefix ( body : str ) -> str : if body . startswith ( XSSI_PREFIX ): return body [ len ( XSSI_PREFIX ):] if body . startswith ( XSSI_PREFIX_NO_COMMA ): return body [ len ( XSSI_PREFIX_NO_COMMA

Job postings are one of the most underrated public data sources on the internet. Recruiters use them to spot placement opportunities, B2B teams read them as buying signals (a new Head of Data means data-tooling budget), and job seekers want to apply on day one — not when a posting finally reaches the aggregators. The usual instinct is to scrape career pages. Don't. Most tech companies host their careers page on one of a handful of Applicant Tracking Systems (ATS), and the four biggest ones — Greenhouse, Lever, Ashby and SmartRecruiters — all expose public, documented JSON APIs . No auth. No proxies. No brittle HTML selectors. The career page itself loads the same JSON you're about to fetch. In this tutorial we'll build a single-file Python tool that: fetches every open job for a company from any of the four ATS, auto-detects which ATS a company uses, normalizes everything into one clean schema, monitors changes — run it on a schedule and get only new / removed / changed postings. The four endpoints ATS Endpoint Greenhouse GET https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true Lever GET https://api.lever.co/v0/postings/{slug}?mode=json Ashby GET https://api.ashbyhq.com/posting-api/job-board/{slug} SmartRecruiters GET https://api.smartrecruiters.com/v1/companies/{slug}/postings (paginated) The {slug} is the company identifier you see in career-page URLs: boards.greenhouse.io/stripe → stripe , jobs.lever.co/spotify → spotify , jobs.ashbyhq.com/linear → linear , careers.smartrecruiters.com/Visa → Visa . Try one right now — no API key needed: curl -s "https://api.ashbyhq.com/posting-api/job-board/linear" | head -c 400 Step 1 — fetchers, one per ATS Each API returns a different shape, so we normalize as we fetch. Here are all four (Python 3, only requests ): import requests UA = { " User-Agent " : " ats-jobs-tutorial/1.0 " } def get_json ( url , params = None ): r = requests . get ( url , params = params , headers = UA , timeout = 30 ) r . raise_for_statu

Most write-ups on this subject are about technique. This one is about the three decisions you should make before you write any code, because in my experience every project that went badly went badly for a reason that was decided on day one and not noticed until much later. I have built this kind of collection twice, for competitive analysis and for a partner-vetting workflow. Neither of them needed to touch anything behind a login, and I want to explain why that turned out to be the useful constraint rather than the limiting one. Line one: the login wall is a boundary A login wall is a statement about who the content is for. Treating it as an engineering obstacle to be routed around is the decision that puts a project on the wrong side of everything: terms of service, the platform's own detection, and in several jurisdictions the law. So the first line is simply: if it requires an account to see, it is out of scope. Not "hard," not "for later." Out of scope. I am not going to discuss techniques for getting past one, and I would be sceptical of any article that does. The interesting engineering question here is not how to see more. It is how much you can actually do with what is openly published, and the honest answer is: considerably more than people assume before they check. This constraint also has a practical benefit that is easy to miss. A pipeline built only on openly available data does not break when authentication changes, does not require credential management, and does not put an account at risk. Mine has survived two platform changes that took down colleagues' authenticated collectors. Line two: public does not mean unrestricted The second line is the one developers get wrong most often, and it has nothing to do with access. Data being publicly visible says nothing about whether you may store it, for how long, or what you may do with it. In the EU and UK, information about an identifiable person is personal data whether or not they published it themselves

I build a web service that preserves evidence of harassment on social platforms. The core feature is a single thing: automatically capture a real screenshot of the offending post. There was no substitute for it. I built an alternative that pulled the text through an API and rendered a tidy "evidence card" image, and threw it away. An image you can author freely afterwards proves nothing. Here's the conclusion first. Third-party wrappers eventually die, and when they do, the failure comes back as a plausible-looking image rather than an error. The first approach was refused by the other side I started with Cloudflare Browser Rendering. The wiring worked. The capture didn't. X blocks headless browsers. The request times out YouTube refuses script injection under a Trusted Types CSP. There's no way to make it render the comment Neither is a bug in my implementation — that is how they are built. So I declared Cloudflare alone impossible for this and moved to a service with a real browser and bot avoidance behind it. Both captures started working. For X, open the post page and clip the tweet element. For YouTube, open the URL with &lc= and screenshot just that comment element. Element screenshots have one trap worth knowing: selector_algorithm=clip returns a blank image when the element sits below the fold. The selector matches, the capture "succeeds," and the file is empty. That took a while to see. ytd-comment-thread-renderer :has ( a [ href *= "lc=ID" ]) A parameter that had worked started returning 400 I wanted timestamps rendered in Japan time, so I passed time_zone: Asia/Tokyo . One day every request started coming back 400. Every capture failed. The provider had narrowed which timezones they accept. Nothing changed on my side. I could diagnose it immediately only because I was storing the raw error body in the database. The response went into rawPayload.screenshotError , so opening one row told me why. Without that, this starts as "captures stopped working, no ide

If you clip short-form video for money, you know Whop Content Rewards: hundreds of live campaigns paying $0.15–$20 per 1,000 views. The discover page lets you sort by budget. That sort is quietly costing you nights of work. Here's the number that changed how I pick campaigns: on the live board right now, 21% of active campaigns have never paid out a single cent. Big banner budget, $0 actually spent. A "$30,000 budget" campaign that has paid nobody in three weeks is not a $30,000 opportunity — it's a landing page. The problem: the board doesn't show you payout speed. You can see budget and budget left , but not how fast the money is actually moving — and that's the only number that separates a campaign that pays from a campaign that poses. The trick: the page already contains everything you need Every campaign card on Whop publishes three things: when it was funded, how much has been spent, and how many creators joined. From one snapshot — no monitoring, no state between runs — you can derive: dailyBurnUsd = budgetSpent / daysSinceFunded → is money moving? estimatedDaysLeft = budgetLeft / dailyBurnUsd → will it still be there? payoutPerCreatorUsd = budgetSpent / creators → what did the average clipper earn? budgetPace = "draining" | "healthy" | "slow" | "stalled" That last field is the shortcut. On today's board of 456 campaigns: pace meaning what to do draining <3 days of budget left skip — gone before your clip gains traction healthy 3–60 days this is where you clip slow 60–180 days fine, but budget may outlive the campaign stalled >180 days at current burn the "big budget" mirage — money posted, almost nobody paid null zero paid out so far unproven; could be brand new, could be dead Real example from today: two campaigns, both showing ~$30K budget. One burns $255/day and has paid the average creator $75 . The other burns $19/day — at that rate its budget lasts four years , which is a polite way of saying nobody is getting paid. On the default board they look ident

A few months back I was running Sales Navigator searches for a client project — filtering down to "VP Sales, fintech, based in Italy or Spain" type lists — and the results were genuinely good. 60, 80 leads that actually matched. Then I hit the part nobody warns you about: there's no button on that page that says "save this." So I did what everyone does. Opened a spreadsheet, alt-tabbed back and forth, typed names and job titles by hand. Around profile 40 I gave up and went looking for a better way. This is what I found, roughly in the order I found it, including the tool I ended up building because none of the existing options quite fit what I needed. First: the export LinkedIn actually gives you LinkedIn has a real, built-in data export, and most people don't realize how narrow it is. It's under your profile photo → Settings & Privacy → Data Privacy → Get a copy of your data . From there you either tick specific categories (that email usually lands within minutes) or request the full archive, which takes closer to a day and sometimes arrives in two batches. Either way you get a download link that expires after 72 hours — and it's desktop only, the mobile app won't let you request one. What you get back is genuinely thorough: connections, messages, your own profile history, activity, even the ad-targeting data LinkedIn holds on you. A couple of quirks worth knowing before you rely on it: some connections' email addresses will just be missing, because sharing an email on download is something each person opts into individually, and you won't get a list of who viewed your profile or any "People You May Know" data. If you're in the EU, EEA, or Switzerland, LinkedIn also runs a separate API for pulling your data on a schedule rather than as a one-off request. Here's what this export is not built for, though: it has no idea what you searched for yesterday. It's an archive of your own account, not a way to capture a live search. Run a Sales Navigator query and pull 80 lea

Cómo solucionar el error “Enable JavaScript and cookies to continue” Este error aparece cuando Cloudflare (u otro proxy inverso de seguridad) detecta que el navegador del usuario no cumple con los requisitos mínimos para acceder al sitio: JavaScript está deshabilitado o las cookies no están permitidas . Pero en entornos reales, el problema suele ser más sutil: el navegador sí tiene JS y cookies habilitados, pero la configuración del entorno de ejecución (como un headless browser, test automation, o un scraper) no emula correctamente el comportamiento del cliente . 🔍 Causa raíz técnica Cloudflare emite un desafío (CAPTCHA o JS challenge) para verificar que el cliente es un navegador real. Si la respuesta no cumple con el desafío (por ejemplo, porque: El navegador no ejecuta el JS del desafío (headless sin soporte), Las cookies no se persisten entre solicitudes, El User-Agent o Accept-Language no coinciden con navegadores reales, Falta el Referer o Origin en headers, Se bloquean cookies de terceros (como las de Cloudflare), … entonces el servidor devuelve este mensaje estático en lugar de redirigir a la página solicitada. ⚠️ Nota crítica : Si estás usando herramientas como curl , requests de Python, o navegadores headless sin configuración especial, no pasarás el desafío de Cloudflare . Es intencional: Cloudflare bloquea tráfico no humano por diseño. ✅ Solución definitiva (por escenario) 🛠️ Caso 1: Navegador real (usuario final) Verifica que JavaScript esté habilitado : Chrome: Configuración → Privacidad y seguridad → Configuración de sitios → JavaScript → Permitido . Firefox: Preferencias → Privacidad y seguridad → Cookies y datos de sitios → Deshabilitar “Bloquear cookies y datos de sitios” . Limpia cookies y caché (especialmente para *.cloudflare.com ). Reinicia el navegador y vuelve a cargar la página. 🛠️ Caso 2: Automatización / Scraping (Python + Playwright/Selenium) No uses requests o urllib : no ejecutan JS. Usa un navegador real con soporte para Cloudflare. ✅

This article was originally published on e-shikumi-labo . Hello, I'm Shin from e-Shikumi-Labo. This is Part 3 of "Systematized Thinking," where we use AI to build our own tools and independently maintain them. Last time, I talked about creating a system to automatically output Markdown (.md) files to Google Drive simultaneously with appending to a spreadsheet. With list management in a spreadsheet and a comfortable viewing environment in Obsidian established, it was getting very close to completion as a tool. However, as I continued to use it practically, new challenges emerged on the operational front. This time, I will share the risks I faced while transitioning from a "manual button" to "full automation," and the process of evolving into safe code. 1. I Want to Eliminate the "Hassle of Pressing a Button" During the prototype stage, the system was designed so that logs were saved by pressing a button placed on the screen. However, as long as a human operates it manually, there are inevitably limitations. If you are concentrating on the conversation, you might forget to press the save button and close the screen. If the conversation gets long, you might miss past utterances that are no longer displayed on the screen. "If I have the screen open and am conversing, I want it to automatically save in the background without bothering human hands." Thinking this, I asked the AI to write the code for full automation. 2. The Code the AI Produced: "Patrolling the Screen Every 1.5 Seconds" When I consulted the AI, it immediately presented code for full automation. The mechanism was, "Start a timer every 1.5 seconds, check the entire screen in the background, and send any new utterances." When I actually tried it, the logs accumulated automatically as soon as I conversed without pressing the button, and at first glance, it looked like exceptionally well-done full automation. However, I felt something was slightly off regarding this "monitoring on a 1.5-second cycle." 3. The B

Key takeaways A request is judged on many layers at once — IP reputation, TLS fingerprint, HTTP/2 shape, headers, and how the browser is driven — and failing any one is enough for a 403. Your laptop passes because every layer is consistent with a real home browser; a server changes one (usually the IP) and the inconsistency is the tell. A 403 with no challenge page almost always means you were blocked at the network layer (IP/ASN reputation or TLS/JA3 fingerprint) before any HTML was served — not a credentials or rate-limit bug, so 'add a User-Agent' or 'slow down' won't fix it. The fix order that actually works: get off datacenter IPs (residential/ISP proxies), match a real browser's TLS fingerprint, and spin up a real-browser stealth setup only for the pages that truly need JavaScript — escalate, don't lead with a browser. A proxy only changes your IP; a Linux VPS still leaks a Linux-shaped TLS/JS fingerprint, so 'residential IP + datacenter everything-else' is a contradiction a real home machine never makes — which is why a proxied server can get blocked harder than your laptop. Your scraper runs perfectly on your laptop. You deploy it to a VPS or a CI runner, change nothing in the code, and suddenly every request comes back 403 . It feels like a bug — the code is identical — but it usually isn't. Anti-bot systems judge a request on many signals at once, and moving from your home machine to a datacenter flips several of them at the same time. This post breaks down exactly which signals change, how to tell which one is blocking you, and how to fix it — for authorized access to public data (we'll keep that framing honest throughout; nothing here is about defeating a protection). A request is judged in two stages It helps to know that detection happens in two stages: Stage 1 — before any HTML is served. IP reputation, your TLS handshake, your HTTP/2 settings, and header order are all inspected on the connection itself, passively and cheaply, before your request is e

A scraping job usually fails in boring ways: the browser hangs, a selector starts returning empty strings, a login expires, or the target site returns a captcha halfway through the run. The awkward part is that many platforms still bill you for the work done before the failure. If you run enough jobs, that difference shows up both in your invoice and in the amount of defensive code you need around the scraper. Billing by compute time changes how you build A lot of scraping platforms charge for runtime. Apify, for example, uses compute units: memory multiplied by time. A browser-heavy actor running for ten minutes with 2 GB of RAM consumes roughly a third of a compute unit before any actor-specific result fees. That model is reasonable from the provider side. Chromium processes are expensive. Proxies cost money. Retries use resources. But as the caller, you care about a different unit: did I get the rows I needed? The hard part is that runtime billing makes cost hard to know before execution. A job that normally takes 30 seconds might take 8 minutes when a site slows down. A job that returns malformed data can still count as successful from the platform's point of view. A job that fails after rendering 200 pages still consumed browser time. If your pipeline runs once a day, that may be fine. If it runs continuously, you probably want a local cost model that tracks outcomes, not just requests. type ScrapeRun = { jobId : string ; target : string ; startedAt : string ; finishedAt ?: string ; status : " queued " | " running " | " succeeded " | " failed " ; rowsExpected ?: number ; rowsReceived ?: number ; billedUnits ?: number ; }; function isUsefulResult ( run : ScrapeRun ) { if ( run . status !== " succeeded " ) return false ; if ( run . rowsExpected && ( run . rowsReceived ?? 0 ) < run . rowsExpected * 0.9 ) { return false ; } return ( run . rowsReceived ?? 0 ) > 0 ; } function costPerUsefulRow ( run : ScrapeRun ) { if ( ! isUsefulResult ( run )) return Infinity ; ret

When you scrape at any real volume, the bottleneck is rarely your code — it's the target site's rate limiting and IP bans. Rotating residential proxies solve this by routing each request through a different real-user IP. Here's how to wire them into requests and Scrapy in Python, including the sticky-session trick most tutorials skip. The proxy URL format A residential proxy is just an authenticated HTTP/SOCKS endpoint. With a pool gateway, you target a country and control session behavior through the username , not separate endpoints: http://USERNAME_country-us_session-a1b2c3_lifetime-30:PASSWORD@proxy.gproxy.net:1000 country-us — exit country (ISO code) session-a1b2c3 — a sticky-session id; reuse it to keep the same IP , change it to rotate lifetime-30 — how many minutes that session's IP stays fixed Basic request through a rotating proxy import requests USER = " USERNAME " PWD = " PASSWORD " def proxy ( country = " us " , session = None , lifetime = 30 ): tag = f " _country- { country } " if session : tag += f " _session- { session } _lifetime- { lifetime } " url = f " http:// { USER }{ tag } : { PWD } @proxy.gproxy.net:1000 " return { " http " : url , " https " : url } # New IP on every call (no session id): r = requests . get ( " https://api.ipify.org?format=json " , proxies = proxy (), timeout = 30 ) print ( r . json ()[ " ip " ]) Run it in a loop and you'll see a different IP each time — the gateway rotates automatically when no session id is present. Sticky sessions: keep one IP across requests Some flows (login, multi-step checkouts, paginated results behind a cookie) break if your IP changes mid-session. Pin the IP by passing a stable session id: import uuid sess = uuid . uuid4 (). hex [: 8 ] # one id for the whole flow p = proxy ( country = " de " , session = sess , lifetime = 30 ) s = requests . Session () s . proxies . update ( p ) s . get ( " https://example.com/login " ) s . post ( " https://example.com/login " , data = {...}) # same exit IP When you

For a long time I ran product extraction off a database of custom selector configs. Hundreds of retailers, each with its own set of CSS selectors mapped field by field: price here, title there, image over there. It worked. It was also a treadmill. Every retailer that redesigned its frontend silently broke its config, and the maintenance tax grew with every retailer I added. Hundreds of configs is hundreds of things that can rot without telling you, usually right before someone downstream asks why a brand's prices went null. At some point I stopped feeding it. The scraping platform I ran at my last engagement fed a 20M+ product catalogue at millions of pages a day, and the thing that made that survivable wasn't better per-site config. It was leaning on almost no per-site config at all. The pattern Instead of one careful parser per site, run several cheap generic extractors on every page, in parallel: JSON-LD. A huge share of e-commerce pages ship a schema.org/Product block because Google rewards it. It has name, price, currency, availability, images. It's the closest thing to a public API hiding in the HTML. OpenGraph tags. og:title , og:image , product:price:amount . Lower quality than JSON-LD, but present on sites too lazy for structured data, because everyone wants pretty link previews. Microdata / RDFa. Older sites, still surprisingly common outside the US. Embedded state. __NEXT_DATA__ , window.__INITIAL_STATE__ and friends. A JSON blob the site's own frontend hydrates from. Heuristics. A price-shaped string near a currency symbol inside the main content region. The largest above-the-fold image. The h1 . Dumb, and dumb works more often than you'd think. None of these is reliable alone. OG price tags go stale, JSON-LD sometimes describes the wrong variant, heuristics grab the crossed-out "was" price. The trick is you don't pick an extractor. You pick per field, and you let agreement between sources carry the decision. Confidence merge Each extractor emits candida

Every founder I know has burned a week building an investor list: digging through Crunchbase profiles tab by tab, copying partner names into a spreadsheet that is stale before the seed round closes. The data you want is simple, firms plus focus plus contacts, and it is weirdly hard to get in bulk. The shortcut I use now is the Startup Investors Data Scraper on Apify, a queryable investor database of 10,469 firms that returns filtered JSON in one call. Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you. Is there a public API for investor data? Not really. The big commercial databases keep their APIs behind sales calls and paid plans sized for funds, not founders. Free sources are scattered lists and shared spreadsheets with no filters and no freshness guarantees. This Actor takes a different shape: a curated database of 10,469 investment firms (as of December 2025) that you query like an API, filtering by firm type, sector, stage, and country, and paying only for the records you pull. What the investor database API returns The investor database API returns one JSON record per firm: name, type, description, location, website, social links, assets under management, stages, and sector focus, with partner contacts when you ask for them. Field Example Notes firm_name Acme Ventures With firm_description alongside firm_type_name Venture Capital Investor One of 17 firm types firm_country Germany Plus firm_city and firm_state firm_website https://acme.vc Also firm_linkedin_url , crunchbase_url , twitter_url firm_aum $250M Assets under management when known investor_contacts [{ "job_title": "Partner", ... }] Names, titles, LinkedIn URLs, emails when available, and check sizes, with Include_Contacts on Who this is for Founders building a raise pipeline, sales teams selling into VC and PE back offices, and analysts mapping which firms fund a sector. If your CRM needs 200 seed funds with war

Heads-up: this post references a tool I built. It's a genuinely useful walkthrough either way — the technique applies to any Airbnb scraping project. If you've ever tried to scrape Airbnb, you already know the two walls you hit: the pages are rendered by JavaScript, and Airbnb aggressively blocks datacenter IPs. Below is the reliable way to get clean Airbnb data in 2026 — listing prices, ratings, coordinates, and discounts — without running a headless browser or babysitting proxies. The key insight: Airbnb ships its data in the HTML You don't need to render the page. Every Airbnb search response embeds the full result set as JSON inside a <script id="data-deferred-state-0"> tag. Parse that and you get structured data straight away — no DOM scraping, no selectors that break on the next redesign. The path to the results is: niobeClientData[*][1].data.presentation.staysSearch.results ├── searchResults[] // ~18 listings per page └── paginationInfo.pageCursors[] // all page cursors, upfront Each listing carries a base64-encoded ID in demandStayListing.id (decode it, take the segment after the last colon, and you have the numeric listing ID for airbnb.com/rooms/<id> ), a price line with discounts, avgRatingLocalized ("4.95 (123)"), and GPS coordinates. The two gotchas Datacenter IPs get blocked. You need residential proxies. If a response comes back without the data-deferred-state marker, you've been served a bot check — rotate to a fresh IP and retry. ~270 result cap per search. Airbnb won't paginate past ~15 pages. To cover a whole market, split into narrower searches (by price band or neighborhood) and dedupe by listing ID. The no-code way If you'd rather not maintain proxy pools and parsers, I published an Airbnb Scraper on Apify that does exactly the above. Paste a location or a full Airbnb search URL (every filter is honored), and get flat JSON/CSV back. curl -X POST "https://api.apify.com/v2/acts/ethanteague~airbnb-scraper/run-sync-get-dataset-items?token=YOUR_TOKE