标签：#rap

Earlier this year, the “Because I Got High” rapper went viral for winning a case against the cops. Now he’s crypto’s free-speech hero, even though he isn’t quite sure how the digital currency works.

The Anti-Bot Detection Checklist I Use Before Every Scraping Project Every scraping project I take on starts with this checklist. Not because I'm paranoid — but because I've learned the hard way that production scrapers fail silently. They return 200 OK with garbage data, or they get rate-limited so gradually you don't notice for days. This is the systematic approach I've refined over 50+ scraping projects. Pre-Scraping: Know Your Target 1. Identify the CDN and Protection Stack Before writing a single line of code, check what you're up against: # Check CDN and headers curl -I https://target-site.com # Look for these common protection headers: # X-Engine: akamai-html-protection # X-Served-By: DataDome # cf-ray: Cloudflare # X-Bot-Status: blocked Common protection platforms: Cloudflare → Look for cf-ray and __cfduid cookies DataDome → Look for datadome in headers or scripts PerimeterX → Look for _pxff cookies Akamai → Look for akamai-html-protection headers 2. Check Robots.txt Respectfully curl https://target-site.com/robots.txt | grep -v "^#" Don't take this as gospel — but it's a good signal. If they explicitly disallow your use case, that's a flag. 3. Map the Site's JavaScript Rendering Some sites are fully static (fast, easy). Others render everything with JavaScript (need Playwright/Puppeteer). Check: // Quick check - fetch raw HTML vs rendered content // If they differ significantly, you need JS rendering const https = require ( ' https ' ); const html = await fetch ( ' https://target.com ' ). then ( r => r . text ()); const hasAngularVueReact = /ng-app|vue|react|__NEXT_DATA__/i . test ( html ); console . log ( ' Needs JS rendering: ' , hasAngularVueReact ); Code-Time: Defensive Patterns 4. Rotate User Agents const USER_AGENTS = [ ' Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120 Safari ' , ' Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120 Edge/120 ' , ' Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chro

This is the story of how a mundane complaint — "the VPN is slow" — turned into a US patent application. Not a granted patent. An application . I want to be precise about that from the start, because the distance between the two is the whole point of this post. It started with a slow VPN The company I work for had an internal VPN that everyone routed through. It lived in the Tokyo office, it was old, and it was not something I built. Then the complaints started arriving — from a lot of people, all saying the same thing: it's slow. I work from Thailand most of the time. That detail matters. If that aging box in Tokyo had fallen over, I would have been the person furthest from the power button, in the worst position to fix it. A slow VPN is annoying. An unreachable VPN, when you're a few thousand kilometers away, is a real problem. So I started moving it to the cloud. I stood up a WireGuard VPN — modern, fast, and something I could actually reason about and operate remotely instead of inheriting a black box. Down the WireGuard rabbit hole Around that time I was deep into building my own iPhone apps. So the cloud migration turned into a personal project on the side: I built my own server and wired WireGuard into an iPhone app of my own. And to do that properly, I started studying how WireGuard actually works under the hood — the Noise protocol, the handshake, the key exchange. That study is where everything else came from. I wasn't trying to invent anything. I was just trying to understand the thing I was now responsible for. The SYN flood that primed my brain Not long before, the same company had been hit with a SYN flood attack. If you've dealt with one, you know it lodges the mechanics of connection handshakes firmly in your head — the back-and-forth, the round trips, the cost of every "hello" before any real data moves. So I had handshakes on the brain. And then, reading through how WireGuard establishes a session, a thought stopped me: Wait — does it really handsha

Sigma’s new entry is both a bold design experiment and a pretty decent camera.

A scraper can pass every check you wrote and still be wrong about the one thing you actually care about: how much it collected. No exception. No 500. No broken row. Exit code 0, logs green, every field valid. And the set on disk is a quarter of what the site actually has. I have run scrapers in production enough times to stop trusting a green run on its own, and this is the failure that taught me to count. TL;DR A paginated source can serve fewer rows than it claims and never throw — page caps, hidden offset limits, infinite scroll that "ends" early. Your status check (200), schema check (valid row), and byte check (you got data) all pass. None of them counts records. The tell: declared total vs unique ids collected. Or, when there's no declared total, the page that quietly repeats an earlier page. Below is a 40-line probe you can run right now. On a source that caps at 1,500 of a declared 4,000, it returned VERDICT: INCOMPLETE (missing 2500 rows) . This is a completeness check, not a correctness check. Different layer, different bug. What actually goes wrong You write the loop everyone writes. Walk ?page=1 , ?page=2 , keep going until a page comes back empty. Stop. Save. Done. The source has other plans. It says it has 4,000 records — the count is right there in the envelope, or in a "Showing 4,000 results" line in the HTML. But it only ever hands out real data for the first 30 pages. Page 31 doesn't error. It doesn't return empty either. It returns page 1 again. Still HTTP 200. Still 50 valid rows. Your loop has no reason to stop, so it grinds on until its own page budget runs out, collects a pile of rows, and exits clean. You now have 5,000 rows in hand and feel great about it. Looks like plenty. The catch: only 1,500 are unique. The page cap fed you the same first page over and over, and those duplicates hid the shortfall behind a big-looking row count. That is the exact shape of "50 rows passed every check while 4,000 existed" — the scraper saw a lot of rows an

Most web scraping projects are not unique snowflakes. Track competitor prices. Enrich a list of leads. Audit a site for SEO. Pull training data for a model. It is the same handful of recipes, over and over. A web scraping template is one of those recipes, pre-wired: a ready-to-use JSON config that chains the right tools in the right order, so you copy it, point it at your targets, and run. CrawlForge ships 24 of them in the templates gallery . This guide is about using them well — not just copy-paste, but read, adapt, and cost them out before you scale. TL;DR: A CrawlForge template is a copy-paste JSON config that chains multiple MCP tools into one workflow (price monitoring, lead enrichment, SEO audits, market research, AI training data). There are 24 across 9 categories, each costing 3–19 credits per run. Run them from Claude/Cursor, the crawlforge CLI, or the REST API. Free tier = 1,000 credits, no credit card. Table of Contents What Is a Web Scraping Template? Templates Gallery vs the scrape_template Tool How to Use a Template the Right Way 8 Templates Worth Copying First The Other 16 Templates Customizing or Building Your Own FAQ What Is a Web Scraping Template? A template is a saved configuration that orchestrates two or three CrawlForge tools into one workflow with a business outcome attached. Instead of wiring search_web then scrape_structured then analyze_content yourself — and guessing every parameter — you copy a config that already does it. Each template in the gallery carries: A category — E-commerce, Research, Data Collection, Monitoring, AI & LLM, Sales, SEO, Content, or Advanced Scraping (nine in total). A difficulty — beginner, intermediate, or advanced. The tool chain it runs and a fixed credit cost per run (3–19 credits). A copy-paste JSON config with sensible default parameters. You run that config from any MCP client (Claude, Cursor, Windsurf), the crawlforge CLI, or the REST API. Same config, same shape of result. Templates Gallery vs the scrap

The dangerous part of web extraction is not the error. The dangerous part is a clean JSON response that looks correct and is not. If an AI agent uses that output, the mistake does not stay inside a scraper. It moves into a report, a lead list, a price alert, a CRM update, or an automated decision. So before you trust any web extraction API in an agent workflow, test whether it fails honestly. Here is the checklist I use. 1. Test a real page, not a demo page Demo pages are usually polite little museum exhibits. Use pages that behave like the actual internet: a JavaScript-heavy product page a search results page a page with missing fields a login-gated page a blocked or rate-limited page a thin page that has layout but little useful content If the tool only looks good on clean static pages, you have not learned much. 2. Check whether the API separates fetch success from extraction success A page can return HTTP 200 and still be useless. Common examples: the final page is a login screen the visible content is a bot challenge the useful data loads after hydration and never appeared in the fetch the page contains a generic country selector instead of the product the source has navigation text but no actual item data The extractor should not treat these as success just because the request completed. A better response says something like: { "status" : "failed" , "failure_type" : "login_required" , "extracted" : null , "next_step" : "Use a public URL, authorised access, or sample content." } That is boring. Boring is good here. Boring means the agent can stop safely. 3. Ask for fields that are not on the page This is the easiest lie detector. Ask the extractor for something impossible: Extract the product name, price, stock status, warranty length, CEO name, office address, and refund policy. If the page only contains product name and price, the API should say the other fields are missing or low confidence. It should not politely invent them because the schema asked nicely.

You've been building REST APIs — one endpoint for users, another for posts, another for comments. The client makes three requests, stitches the data together, and half of it gets thrown away because it wasn't needed in the first place. GraphQL was built to fix exactly that. It gives the client full control over what data it receives. One request. Exactly what you asked for. Nothing more, nothing less. The Problem REST Couldn't Solve Before understanding GraphQL, you need to understand the two problems that drove its creation. Over-fetching The server returns more data than the client needs. GET /users/123 Response: { "id": 123, "name": "Anne", "email": "anne@example.com", "phone": "...", "address": "...", "createdAt": "...", ← you didn't need any of this "updatedAt": "..." ← but the server sent it anyway } The client only needed name and email — but it downloaded the whole object every time. Under-fetching One endpoint doesn't return enough, so the client has to make multiple requests. GET /users/123 → gets the user GET /users/123/posts → gets their posts GET /users/123/followers → gets their followers Three round trips to the server just to render one screen. On a mobile network, that cost is real. GraphQL's answer: Let the client write the query. The server returns exactly what was asked. What Is GraphQL? GraphQL is a query language for your API and a runtime for executing those queries. It was created by Facebook in 2012, open-sourced in 2015, and is now maintained by the GraphQL Foundation . Unlike REST, which exposes multiple URL endpoints, GraphQL exposes a single endpoint — typically POST /graphql . The client sends a query in the request body describing exactly what it wants, and the server responds with only that data. Key characteristics: Single endpoint — everything goes through POST /graphql Client-driven — the client defines the shape of the response Strongly typed — every field has a declared type in the schema Introspective — clients can query the API

Cloudflare Turnstile in Playwright: Why Your Tests Stall and How to Solve It in 8 Lines If you're running Playwright or Selenium against any site behind Cloudflare, you've already met Turnstile. It's the new "managed challenge" widget Cloudflare started shipping in 2023, and it now appears in front of login flows, contact forms, signup pages, and increasingly the entire site root. Here's the part most teams miss: Turnstile doesn't always show a checkbox. A lot of the time it just sits invisible, runs its scoring loop, and either issues a token silently or stalls forever. Your test doesn't crash. It just times out at the next page.click("button[type=submit]") . The CI log says "element not interactable." Nobody knows why. I work on CaptchaAI. I'm going to show you exactly what's happening, then drop in 8 lines that fix it. The real scenario You have a Playwright suite that runs every PR. One day a test starts failing on the signup flow. You re-run it. It fails again. Locally on your laptop it passes. On CI it doesn't. What's actually happening: Cloudflare flagged your CI runner's IP block (GitHub Actions, GitLab runners, Hetzner, OVH, DO — all of them are on Cloudflare's "elevated risk" list). Turnstile decides to switch from invisible mode to "managed challenge" mode. Now there's a widget in the DOM that needs a real token before the form submit will accept. Your test never interacted with the widget because last week it didn't exist. Why retries don't help The instinct is to add a retry: 2 and move on. Don't. Cloudflare's scoring is per-IP-per-fingerprint, and each retry from the same runner makes the next challenge harder, not easier. After ~3 attempts you'll get full block pages instead of the widget. The right move is to solve the widget once, inject the token, and submit normally — exactly what a human user does, just faster. How Turnstile actually issues a token The widget renders an iframe pointing at challenges.cloudflare.com . Inside the iframe it runs a fi

Researchers are using machine learning algorithms to decrypt historical pencil-and-paper ciphers.

pluckmd exists so an agent can pull blog posts into markdown, index them into a wiki, and generate interactive HTML to learn from. This post is about the first step, the part with no per-site code, because the design is the interesting bit. If you want the practical side, how I actually use it day to day, I wrote that up separately: https://dev.to/taisei_ide/how-i-use-pluckmd-to-read-blogs-with-an-ai-agent-1jpe It downloads articles from a blog without any per-site code. No handler for Medium, no handler for Substack, nothing keyed on a domain. Here's how that works. The core idea: treat extraction as data, not code. AdapterSpec Instead of branching on which site you're on, pluckmd resolves an AdapterSpec . It's a plain object that says which selector finds article links, what the URL pattern looks like, and how pagination behaves. interface AdapterSpec { listing : ListingExtractionSpec ; // how to find article links article : ArticleExtractionSpec ; // how to pull the body pagination : PaginationSpec ; // none | scroll | button-click | next-url | auto evidence : string ; } Because it's data, the same shape can come from a heuristic, an LLM, an agent, or a person typing it by hand. They all produce the same thing, and they all go through the same checks. Resolving it, cheapest path first cache -> heuristics (local, free) -> LLM (only if needed) Cache first, rechecked against today's DOM so a stale entry can't sneak through. Then local heuristics. The LLM only gets called when the heuristics aren't sure. Every result that works gets written back, so the second run on a site is basically instant. How the heuristics find an article list This part has no idea what site it's looking at. It takes every link, normalizes the path, and collapses the parts that vary into wildcards. / blog / my - first - post -> / blog /* / blog / another - article -> / blog /* / about -> / about Group by that shape. Any group with the same pattern repeated three or more times is a candidate f

As part of their 20th Anniversary celebration, Dark Reading asked five cybersecurity industry leaders who wrote blogs or columns for them over the years to select their favorite piece and share their reflections on the topic today. This is my section. Renowned technologist and author Bruce Schneier contributed a column on June 20, 2010, warning about cryptography’s inability to secure modern networks , a point he says he has been trying to argue since 2000. “For a while now, I’ve pointed out that cryptography is singularly ill-suited to solve the major network security problems of today: denial-of-service attacks, website defacement, theft of credit card numbers, identity theft, viruses and worms, DNS attacks, network penetration, and so on...

Most scraper demos lie by accident. They show the happy path: one URL, one clean page, one neat JSON object. Then the first real user tries a marketplace search page, a login wall, a JavaScript shell, a rate-limited product page, or a site that serves different HTML to every fetch path. The response still comes back as JSON, so everyone relaxes. That is the trap. A JSON response is not the same thing as a useful extraction. The failure mode agents hate AI agents do not just need scraped text. They need to know what happened. Bad extraction output looks like this: { "title" : "Example product" , "price" : "$29.99" , "availability" : "in stock" } That looks fine until you inspect the source and discover the page was a login prompt, a bot challenge, or a thin JavaScript shell. The extractor filled the schema because the schema was requested. Helpful. Like a smoke alarm that hums a little song while the kitchen burns. Better extraction output separates the data from the confidence and the failure class: { "status" : "failed" , "failure_type" : "login_required" , "confidence" : 0.94 , "extracted" : null , "evidence" : { "final_url_type" : "restricted_page" , "visible_content" : "login prompt" , "structured_data_found" : false }, "next_step" : "Use an authorised source, public item URL, feed, API, or sample HTML." } That is less flashy. It is also much more useful. The useful contract is not “scrape anything” “Scrape anything” is usually a warning label wearing lipstick. For agent workflows, the better contract is: Return structured data when the page provides enough evidence. Return a specific honest failure when it does not. Preserve enough metadata for the caller to decide what to do next. Never invent fields just because a prompt asked nicely. This matters for ecommerce, lead enrichment, price monitoring, competitor tracking, procurement, and internal research agents. If the agent cannot tell the difference between “product unavailable”, “page blocked”, “login require

Shopify introduced GraphQL Cardinal, a new execution engine replacing depth-first traversal with breadth-first execution. The redesign improves large-scale GraphQL performance with up to 15x faster field execution, 6x lower GC overhead, and +4s P50 latency gains. It focuses on execution-layer efficiency and batched resolver processing for high-cardinality commerce queries. By Leela Kumili

Strava will charge a flat monthly fee from developers to access its API

The row looked perfect. rating: 7 . Valid JSON, right type, no nulls, no missing keys. My schema check waved it through. The page had returned HTTP 200. The selectors hadn't moved. Everything green. A rating of 7 on a 5-star site is impossible. The model invented it, formatted it correctly, and handed it to me with total confidence. That's the failure I want to talk about. Not the scraper that breaks loudly. The one that hands you a clean-looking row that is quietly, plausibly false — and sails past every check you have, because your checks are all looking at the shape of the data, and the lie is in the value . TL;DR HTTP 200, intact selectors, and valid JSON tell you the form is fine. They say nothing about whether the value is true. When an LLM extracts from messy free-text, structured-output mode guarantees you get valid JSON. It does not guarantee the content is real. The model fills uncertain fields rather than leaving them empty — because the schema demands a complete row. A ~60-line value-level sanity gate (ranges, dates, cross-field, reference, language) catches the obvious lies before they hit your database. Real code and real output below. The honest catch: this gate catches rule violations , not plausible lies inside the allowed range . A rating: 4 where the truth is 2 slides right through. I'll be specific about where the gate stops. Two different ways a scraper lies to you I wrote about source drift last week — the case where the page changes underneath you and a 30-line schema check catches the structure shifting. That's an input problem. The source mutated; your agreement with the page broke; you detect it by watching the shape. This is the other end of the pipe. The source is fine. The page is intact, the selectors are correct, the structure is exactly what you expected. The thing that lied to you is the model , on the extraction step, when you asked it to pull structured fields out of a paragraph of human prose. Those two failures feel similar and t

I built a small web scraping framework in Rust, mostly with an AI doing the typing. It's called ferrous — a Colly-style collector: register CSS selector callbacks, queue URLs, write JSONL. About 700 lines. The pitch I kept hearing, and half-believed, was that Rust and LLMs are a good match now: the borrow checker is a correctness oracle the model can lean on, so the class of bugs that plagues AI-written Python just won't compile. That's true. It's also where the story gets uncomfortable, because the build was green and the code was still wrong. How I worked I'm not a Rust native. ferrous was partly an excuse to get fluent — build something real instead of reading about lifetimes — and partly a test of how far an LLM could carry the typing while I drove. The loop was plain: describe the next change in English, let the model write the Rust, read what came back, run cargo , move on. It kept observations.md as a running design journal, one entry per change, each with a short rationale for the decision it made. That setup has a soft spot, and it's the whole point of this post. When you drive a language you don't fully know, the only reviewer you've got with real authority is the compiler. Everything past that — is this idiomatic, is it the right abstraction, does it actually do what the journal claims — depends on already knowing what correct looks like. Which is exactly the knowledge a learner doesn't have yet. Keep that in mind through the next part, because it's the difference between the one bug I could have caught and the six I couldn't. The bug that the toolchain told me wasn't there I ship two fetch backends. The default goes through the Zyte API; an optional one, gated behind a wreq feature, makes direct requests with browser TLS emulation. Each has an example. After a refactor that added URL resolution — ctx.resolve_and_visit(href) so callbacks stop hand-building absolute URLs — I had the model update the examples to use it. It did, and it wrote up the change in

Quick answer: The Google Ads Transparency Center is a public registry of every ad Google runs — but it ships no API and no bulk export . To get the data programmatically you scrape it. A Google Ads Transparency scraper sends the same RPC call the website uses and returns every ad creative for an advertiser as structured JSON. The Apify Actor below does it for $0.0012 per ad (~$1.20 per 1,000), with the TLS fingerprinting, proxy rotation, and pagination handled for you. Google's Ads Transparency Center is one of the most underused datasets in marketing. Launched in 2023 under the EU Digital Services Act and parallel US pressure, it indexes every ad campaign currently running on Search, YouTube, Display, Shopping, Maps, and Play — keyed by advertiser. Google's own counter lists 300,000+ active creatives for a brand like Nike . For your nearest competitor, it's usually 50–500. The catch: there's no download button. Just an interactive UI that paginates 40 creatives at a time. If you want this as a CSV — for a competitor sweep, a trademark audit, or a RAG corpus — you have to extract it yourself. Here's what that actually takes, and how I shortened it to one API call. What is the Google Ads Transparency Center? 🔎 The Google Ads Transparency Center is a public, Google-operated registry that shows the ad creatives any verified advertiser is running, the date range each ad was shown, and roughly where. Google built it to comply with ad-disclosure regulation, so the data is public by design — you're reading the same registry a regulator would. What it gives you per advertiser: Every ad creative currently or recently live (text, image, video) The landing domain each ad clicks through to First-shown / last-shown timestamps and a rough impression count A deep link to each creative inside the Transparency Center What it does not give you: a search-by-keyword mode, region-filtered results from the server, or — crucially — an API. Does the Google Ads Transparency Center have an A

I came across Scrapling through a recommendation on X and decided to put it through its paces — not against a demo page, but against Lazada Singapore, a production site with Google reCAPTCHA and a custom slider verification. The setup: a single 4GB VPS, no residential proxies, no credits, just open-source tools. Here's the full journey: installation pitfalls, wiring it into an AI agent, choosing the right browser for the job, and the real-world benchmarks that followed. What Is Scrapling? Scrapling is an adaptive web scraping framework for Python (BSD-3, v0.4.8). It handles everything from single HTTP requests to full-scale concurrent crawls. What sets it apart from the BeautifulSoup/Scrapy world: Adaptive element tracking — saves fingerprints of targeted elements and relocates them after site redesigns using similarity scoring. Your scrapers survive CSS changes without maintenance. Three fetchers, one API — HTTP ( Fetcher , curl_cffi), browser ( DynamicFetcher , Playwright Chromium), and stealth ( StealthyFetcher , Chromium + anti-bot patches). Swap with one line. Spider framework — Scrapy-like API with async, concurrent crawling, Ctrl+C pause/resume via checkpoint persistence, multi-session support. MCP server — 14 tools exposed natively for AI coding agents. Your agent can call mcp_scrapling_get , mcp_scrapling_fetch , mcp_scrapling_stealthy_fetch directly. It's open source, pip-installable, and designed to be the backbone of a scraping stack — not just another tool in the toolbox. Installation on a 4GB VPS This is where the real story starts. The VPS has 4GB RAM, 2 vCPUs, 77GB disk, and runs an AI agent gateway (615MB baseline). Every browser installation decision matters. What we installed pip install scrapling[fetchers,ai] # HTTP + Chromium + MCP server scrapling install # Downloads Playwright browsers This pulls in Playwright Chromium, Firefox, and WebKit (~1.3GB disk), plus curl_cffi for HTTP requests and patchright (Playwright fork) for browser automation.

With sharper resolution and lightning-fast performance, Canon’s latest full-frame mirrorless punches well above its class.