AI 资讯
Your Scraper Collected 50 Rows. There Were 4,000.
A scraper can pass every check you wrote and still be wrong about the one thing you actually care about: how much it collected. No exception. No 500. No broken row. Exit code 0, logs green, every field valid. And the set on disk is a quarter of what the site actually has. I have run scrapers in production enough times to stop trusting a green run on its own, and this is the failure that taught me to count. TL;DR A paginated source can serve fewer rows than it claims and never throw — page caps, hidden offset limits, infinite scroll that "ends" early. Your status check (200), schema check (valid row), and byte check (you got data) all pass. None of them counts records. The tell: declared total vs unique ids collected. Or, when there's no declared total, the page that quietly repeats an earlier page. Below is a 40-line probe you can run right now. On a source that caps at 1,500 of a declared 4,000, it returned VERDICT: INCOMPLETE (missing 2500 rows) . This is a completeness check, not a correctness check. Different layer, different bug. What actually goes wrong You write the loop everyone writes. Walk ?page=1 , ?page=2 , keep going until a page comes back empty. Stop. Save. Done. The source has other plans. It says it has 4,000 records — the count is right there in the envelope, or in a "Showing 4,000 results" line in the HTML. But it only ever hands out real data for the first 30 pages. Page 31 doesn't error. It doesn't return empty either. It returns page 1 again. Still HTTP 200. Still 50 valid rows. Your loop has no reason to stop, so it grinds on until its own page budget runs out, collects a pile of rows, and exits clean. You now have 5,000 rows in hand and feel great about it. Looks like plenty. The catch: only 1,500 are unique. The page cap fed you the same first page over and over, and those duplicates hid the shortfall behind a big-looking row count. That is the exact shape of "50 rows passed every check while 4,000 existed" — the scraper saw a lot of rows an
AI 资讯
Introducing BulkSMSOnline: Global SMS API Built by a Small, Developer-First Team
We’re a tiny team of 2–9 engineers who believe business messaging should be simple, reliable, and accessible to everyone. Today we’re officially opening up BulkSMSOnline to the Dev.to community, and we’d love your feedback. What’s BulkSMSOnline? A global bulk SMS platform that lets you send campaigns, alerts, OTPs, and notifications via: A clean web portal A REST API An HTTP API It’s designed for developers who want reliable global delivery without fighting arcane telecom protocols or opaque pricing. Why We Built It We noticed a pattern: most SMS platforms either overcomplicate things with bloated SDKs or hide behind enterprise gatekeepers that don’t listen. We wanted something different a lean, transparent API backed by real people who actually care about your deliverability. So we built BulkSMSOnline around three principles: Reliability : Messages must arrive, every time. Radical simplicity : A clean API you can integrate in minutes. Transparency : Honest pricing, clear limits, no surprises. Quick Start: Send an SMS in Under 5 Minutes Here’s how simple it is with our REST API. For full docs, check out our developer portal . curl -X POST https://api.bulksmsonline.com/v1/sms \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "to": "+1234567890", "message": "Hello from BulkSMSOnline!", "sender": "MyApp" }' You’ll get a JSON response with a message ID and status. That’s it. No multi-page setup or carrier negotiations. What else can you do? Send bulk messages with a single API call Track delivery in real time via webhooks Pull reports programmatically Use our HTTP API for legacy systems Who’s Behind This? We’re a small, agile team (2–9 people). That means: No bureaucracy: Fixes and features ship fast. Direct access: When you email support, you reach the engineers who built the platform. Your feedback shapes our roadmap: Many of our recent features came from developer conversations. Tech stack we love: Python, Node.js, PostgreSQL, an