标签：#API

Quick answer Craigslist search pages ship two copies of every listing: a static HTML list, and a JSON-LD <script> block with images, currency, and geo-coordinates. The obvious move is to join them by ID. Don't — Craigslist's JSON-LD carries no shared identifier at all , not a bare post ID, not a URL, not a SKU. The only field both copies reliably share is the listing's title, and titles repeat. We joined by title through a per-title FIFO queue and measured it recovering 290 of 325 listings (89%) end-to-end on a captured 298-item page. That number is the ceiling of what a title-only join can do on this page shape — plan your field completeness around it, don't assume 100%. Why can't you just match the JSON-LD by ID? 🧩 When we built the Craigslist Multi-City Listings Scraper , the first design assumed what almost every JSON-LD block on almost every e-commerce-shaped site provides: a productID , a sku , a url , an @id — something that lines up a JSON entry with its DOM counterpart deterministically. Live inspection of a captured Craigslist search page found none of those. Each itemListElement entry has exactly name , image , offers , @type , and a position field that looks like it should solve the problem — until you check it against the static list past the first ~18 entries, where some static-list rows have no JSON-LD counterpart at all and the position numbering drifts out of alignment. So the join key that's actually usable, live, is title — and titles aren't unique. The fix is a FIFO queue per title: walk the static <li> list in document order, and for each title pop the next unconsumed JSON-LD entry with a matching name. # actors/craigslist-listings-scraper/src/search_parser.py def _parse_ld_json ( tree : HTMLParser ) -> dict [ str , deque [ _LdEntry ]]: by_title : dict [ str , deque [ _LdEntry ]] = defaultdict ( deque ) for list_item in data . get ( " itemListElement " , []): entry = _ld_entry_from_item ( list_item ) title = list_item . get ( " item " , {}). get

Quick answer BBB.org doesn't render business data into the HTML table it looks like it does. Both the search-results page and every business profile page embed the real data as one inline JSON blob — webDigitalData — sitting inside a <script> tag meant for analytics, not for you. The visible <dt> / <dd> list underneath it only carries the fields the analytics layer left out (accreditation date, years in business). If you scrape the DOM table and ignore the script tag, you'll get a name and maybe a phone number and nothing else. If you scrape the script tag and ignore the DOM, you'll get ratings and IDs but no address, no years-in-business, no website. You need both, merged, per business. Why does the DOM only have half the data? 🕵️ When we built the BBB Business Leads Scraper , the first pass assumed BBB profile pages worked like most directory sites: a template with labeled fields you css_first() your way through. That assumption survives for exactly four fields — address, accreditation date, years in business, and website — which live nowhere except a <dt> / <dd> definition list further down the page. Everything that actually matters for lead scoring — business name, BBB letter grade, accreditation status, phone number, the internal IDs BBB uses to build its own canonical URL — comes from a different place entirely: a webDigitalData object, wired into the page for BBB's own analytics vendor, that happens to be complete, well-formed JSON: # actors/bbb-business-leads-scraper/src/parsers/common.py WEB_DIGITAL_DATA_MARKER = " webDigitalData " def extract_web_digital_data ( html : str ) -> dict | None : marker_pos = html . find ( WEB_DIGITAL_DATA_MARKER ) if marker_pos == - 1 : return None brace_start = html . find ( " { " , marker_pos ) raw = _extract_balanced_json ( html , brace_start ) ... return json . loads ( raw ) That is not a regex grabbing {.*} between two markers — a naive non-greedy match snaps shut on the first stray } inside a nested object, which this blo

If you sell on a marketplace whose currency differs from your bank's, you have probably tried to compute, per order, how much you will actually be paid — and found that the numbers never tie out. Here is why, and the reconciliation approach that does tie out to the cent. There is no per-order exchange rate. By design. The SP-API does not expose a per-order FX rate anywhere, and it is not an oversight: Orders API (getOrders/getOrder): amounts in the marketplace currency only. No FX, no converted amount. Finances API (listFinancialEvents / listTransactions): also marketplace/transaction currency only, and no applied FX rate. Transfers API: gives you the amount actually disbursed — but at the disbursement level, aggregated, not per order. The reason is that Amazon does not convert per order. When Amazon Currency Converter for Sellers (ACCS) is used, the conversion is applied once to the net disbursement — many orders, fees, refunds and reserves aggregated — at ACCS's own rate and timing. What this breaks If you build your books by applying a spot FX rate to each order, they will never reconcile against your bank deposits. The residual sum(converted_orders) - actual_transfer becomes a moving error you cannot close, because your per-order spot rates were never the rate Amazon used. The approach that reconciles Stop reconciling per order. Reconcile per disbursement: Group every financial event that belongs to a settlement/disbursement, in the marketplace currency, and sum to the net. Compare against the transferred amount in your bank currency. net_bank / net_marketplace is the effective blended FX rate for that disbursement. That single rate ties out exactly, because it is derived from what actually happened, not from an external rate feed. If you need a per-order figure (for COGS or margin), allocate the blended rate proportionally to each order's net. It is an allocation, not Amazon's per-order rate (which does not exist) — but every order rolls up to the deposit you a

There are two real ways to do this: Cloudflare's own native Email Service binding, or calling an external email API like Notify over fetch() . I'll walk through both, but I want to flag something about the native option up front that's easy to miss until you're actually setting it up: it currently requires the Workers Paid plan, not just a Cloudflare account. If you're on the free Workers tier and just want to send a password reset email, that's worth knowing before you spend time on it. Option 1: Cloudflare's Native Email Service Binding Cloudflare's Email Service (which covers both sending and receiving) lets a Worker send email through a binding, with no external API key. As of now it's still in beta, and there's a real gate on it: sending to arbitrary recipients requires the Workers Paid plan, and before you've fully onboarded a domain, the binding can only send to destination addresses you've explicitly verified. Setup looks like this: In the Cloudflare dashboard, go to Compute > Email Service > Email Sending , click Onboard Domain , and pick the domain you want to send from. Cloudflare adds the DNS records it needs automatically — an SPF record, a DKIM record, a DMARC record, and MX records on a cf-bounce subdomain. This usually finishes in minutes, though Cloudflare says it can take up to 24 hours. Add the binding to your Wrangler config: { "send_email" : [{ "name" : "EMAIL" }] } Send from your Worker: export default { async fetch ( request , env , ctx ) { await env . EMAIL . send ({ from : " noreply@yourdomain.com " , to : " user@example.com " , subject : " Welcome! " , html : " <h1>Thanks for signing up.</h1> " , }); return new Response ( " Email sent! " , { status : 200 }); }, }; A couple of things worth knowing before you build on this: by default, wrangler dev simulates the binding locally — emails are logged to your console, not actually sent — unless you set remote: true on the binding to send real mail during local development. And you can restrict wh

Microsoft has enabled zone redundancy on the Standard v2 tier of Azure API Management, following its arrival on Premium v2 in December. Standard v2 starts at $700 per month against $2,801 for Premium v2, but carries a 99.95% SLA rather than 99.99%. Zone redundancy can only be configured when creating an instance. By Steef-Jan Wiggers

The idea I wanted a small side project that could actually run itself once shipped — no cron jobs to babysit, no upstream API to go down at 3am and take my uptime with it. That constraint led somewhere specific: an API that validates common business data formats — phone numbers, IBAN, VAT/tax IDs, BIC/SWIFT codes, credit card numbers, postal codes — using nothing but offline checksum and format rules. No third-party lookups. No API keys to rotate for an upstream provider. No rate limits inherited from someone else's infrastructure. If it's slow or wrong, it's my bug, not a dependency's outage. The stack Hono on Cloudflare Workers — TypeScript, no cold starts, runs on the free tier comfortably up to 100k requests/day libphonenumber-js , ibantools , jsvat , card-validator — all well-maintained, all pure computation, zero network calls Vitest for tests, run against real fixtures (not made-up test data — every "valid" example in my test suite is a real IBAN/VAT/card number pulled from each library's own published examples, verified against the actual library output before I trusted it) The whole thing is about 300 lines of TypeScript across the router and six validator modules. Small enough to actually reason about, which mattered more to me than feature breadth. app . post ( " /v1/iban/validate " , async ( c ) => { const body = await c . req . json < { iban ?: string } > (). catch (() => null ); if ( ! body ?. iban ) { return c . json ({ error : " missing required field: iban " }, 400 ); } return c . json ( validateIban ( body . iban )); }); The part that actually surprised me I expected the code to be the hard part. It wasn't. Deploying and listing it on RapidAPI was. Two things stood out: CORS mattered even though I "shouldn't" need it. Real production traffic through RapidAPI's gateway is server-to-server — CORS is a browser-enforced concept, so I assumed it was irrelevant. But RapidAPI's own in-dashboard request tester runs as a real browser fetch, and without an O

Frontier AI models are expensive to call directly. A single day of heavy Claude or GPT-5 usage in an agentic coding loop can rack up real money. But a small cluster of gateways and coding-agent products has figured out how to hand developers meaningful free access anyway. This post breaks down four of them — Freebuff, AgentRouter, OpenRouter, and Experiential Labs — and the actual tactics each one uses to keep the lights on while giving inference away. 1. OpenRouter — the "free router" and community-subsidized models OpenRouter is a unified, OpenAI-compatible API that sits in front of hundreds of models from dozens of providers. Its free tier isn't a special OpenRouter model — it's a curated set of models, mostly open-weight ones like DeepSeek R1, Llama variants, and Qwen releases, that carry a literal $0/M-token price tag because providers or OpenRouter itself are subsidizing the compute. The tactic: instead of making you pick a free model by hand, OpenRouter built openrouter/free , a router that automatically picks a working free model for each request, smart enough to filter for whatever the request needs — image understanding, tool calling, structured outputs, and so on. That's a neat trick: it turns "which free model works today" from a research chore into a solved problem, since free-model availability shifts constantly and the router absorbs that churn for you. To keep this sustainable, OpenRouter caps usage per key — community trackers put it at roughly 20 requests per minute and 200 requests per day on the free tier — and openly frames free access as ecosystem-building: it says free models help democratize access to AI and let large numbers of people experiment and learn, while it keeps expanding capacity by onboarding new providers and covering some costs directly. In plain terms, the free tier is marketing and community goodwill; paid usage across the rest of the catalog is the actual business. Using it is as simple as pointing any OpenAI-compatible SDK a

If you have ever tried to programmatically get the current RBA cash rate, you know the drill. You open the RBA F1 statistical table, download f01hist.xls, write a pandas.read_excel call, fight with the multi-row header (Series ID on row 11, units on row 6), filter, sort, take the last row. That is 30 lines of code to get a single number that changes 11 times a year. One line (MIT, no key) from rba_mcp import client print ( client . latest ( " F1_1 " , series = " cash_rate_target " ). records [ - 1 ]. value ) # live AU.CASHRATE as of 2026-09-03: 4.35 pip install rba-mcp No key. MIT-licensed. Attribution and source URL come back with the number. Hosted gateway Use GET /v1/series/AU.CASHRATE/latest on api.ausdata.io with a free key from ausdata.io (500 calls/mo). Live 2026-09-06: cash 4.35 percent, trimmed-mean CPI 3.6 percent (2026-Q2), real rate 0.75 percent. Why the hosted path exists The RBA publishes the nominal cash rate. The ABS publishes inflation. Neither publishes the real cash rate (nominal minus trimmed-mean CPI). Use /v1/real-rate-regime for that join. Same envelope across nine AU sources: source, source_url, attribution, retrieved_at. MCP Wire ausdata-mcp via npx in Claude Desktop or Cursor. Sisters on PyPI run fully local with no key. What this is not Suburb-level property prices Live KYC / company-officer lookup 5-minute wholesale electricity bid stacks AU macro public data, one envelope, citations done for you. R users readrba by Matt Cowgill is the R equivalent. This is the Python / JS / agent path. Links Canonical: https://ausdata.io/blog/rba-cash-rate-python-api/ Free key: https://ausdata.io Series: https://ausdata.io/series/AU.CASHRATE PyPI: rba-mcp

If you do local SEO work, you've run into this: a client's Google Business Profile says one phone number, their website footer says another, and nobody noticed until a customer called the wrong number. Or the postal code on the GBP listing is a leftover from an old office. This kind of drift is called a NAP (Name, Address, Phone) inconsistency, and it's widely cited as a local search ranking factor. But checking it by hand means opening every listing and every website side by side. I built Google Maps NAP Consistency Checker , an Apify Actor that takes Google Maps scraper output, fetches each business's own website (lightly: homepage plus one likely subpage), and checks whether the name, postal code, and phone number on the Google Business Profile actually show up on the site. What it does, and what it doesn't This Actor checks one thing: does a business's own website agree with its Google Business Profile on name, postal code, and phone number. It does not check third-party directories (Yelp, Facebook, etc.). That's a different problem with a different competitor landscape. It does not crawl an entire website; it fetches at most two pages per business (homepage, plus a subpage if one with a keyword like "contact" or "about" is linked from it). It does not use an LLM. It's regex and string matching against fetched text, which makes it fast, cheap, and predictable. There's no model that can hallucinate a match that isn't there. Businesses with no independent website (only a social profile, or nothing) are skipped entirely, because there's nothing to fetch and compare against. How it works For each place with a real website, the Actor: Checks robots.txt for that domain before fetching anything, and skips the business if the checker's user agent isn't allowed. Fetches the homepage HTML (up to 3 MB), strips <script> , <style> , and comments before converting to text, so JavaScript variables and tracking IDs don't get misread as phone numbers. Looks for an internal link

How to turn struct-tag validation failures into a standard "application/problem+json" response — the RFC 9457 format — with one method call, no hand-rolled error envelope. Most Go APIs invent their own validation error shape. One team returns {"errors": [...]} , another {"field_errors": {...}} , a third just a flat {"error": "message"} and hopes the client parses it. Every one of those is a private contract the client has to learn from your docs, because there's no shared shape for "here's what's wrong with your request." RFC 9457 — Problem Details for HTTP APIs — is the IETF standard that fixes this: a application/problem+json body with type , title , status , and room for problem-specific extensions. Checker now builds one of these directly from a failed struct validation, via CheckErrors.ProblemDetails() . The shape RFC 9457 defines four base members — type , title , status , detail , instance — and lets a specific problem type add its own. For validation errors, RFC 9457 §3.1 sketches exactly this extension: an invalid-params array listing which fields failed and why. That's what Checker produces. From a failed struct to a problem+json body Take a struct with a missing required field: type Person struct { Name string `checkers:"required"` } person := & Person {} errs , ok := checker . CheckStruct ( person ) if ! ok { data , _ := json . Marshal ( errs . ProblemDetails ()) fmt . Println ( string ( data )) } { "type" : "about:blank" , "title" : "Your request parameters failed validation." , "status" : 400 , "invalid-params" : [ { "name" : "Name" , "reason" : "Required value is missing." , "code" : "REQUIRED" } ] } One method call — errs.ProblemDetails() — turns the same CheckErrors you'd otherwise call .JSON() on into a *ProblemDetails value, ready to marshal. type defaults to "about:blank" (RFC 9457's own default for "no more specific problem type registered"), status defaults to 400 , and each invalid-params entry carries the field name , a localized human-readab

Since 2026 every Shopify storefront answers an agent-commerce endpoint at POST /api/ucp/mcp , advertised at GET /.well-known/ucp . Merchants did not turn it on and it is not in their admin. It speaks the Universal Commerce Protocol over JSON-RPC, and the tool that matters is search_catalog : an AI shopping agent asks a store for its catalogue and gets structured product data back - integer prices in minor units with a currency, variants, SKUs, canonical URLs, and a Shopify taxonomy category per product. Fetch the tool list from any store and search_catalog declares catalog.filters.categories , an array of strings documented as "category filters combined with OR logic", next to catalog.filters.price.{min,max} . So an agent should be able to ask for running shoes and get running shoes. We were about to write a paragraph about what it costs a merchant to leave the category field blank. Then we tried it. Method 200 stores, drawn deterministically from a corpus of 10,099 known Shopify storefronts: sort the hostnames, take every Nth. Reproducible, so nobody has to take "we picked 200 stores" on trust. Run on 2 September 2026. Every store got the same five calls, 10 products requested each time: # Call Filter sent A working filter would 1 Control none return products 2 Impossible category gid://shopify/TaxonomyCategory/zz-99-99-99 return nothing 3 The store's own category a category the control's products carry return at least that product 4 Same, unwrapped the bare id without gid://... the other form an agent would try 5 Price control price.max = 1 return nothing - nothing costs a cent Calls 2 and 3 only mean something together. Call 2 alone cannot distinguish "the filter is ignored" from "the filter rejects everything". Those are opposite findings, and both happen. The query matters more than it looks. Generic words ("gift", "set", "new") surface a catalogue's odd corners rather than its catalogue, and produce numbers that are measured honestly and still wrong. Every que

I run a small AI/automation consultancy in Brazil, and a recent lead-research project needed the full follower list of a public Instagram profile — about 153,000 followers — plus enrichment (bio, public email/phone) to find business accounts worth contacting. The problem Pulling a list that size is never one API call. Instagram reports ~153,628 followers; you get them page by page, and any long-running extraction WILL hit a failed request eventually. If your pipeline can't resume, you start over from zero — which is expensive and slow. What I built The pipeline runs on n8n with Supabase as the datastore: Batched extraction — followers are downloaded in batches of up to 10,000 per cycle, on a schedule, instead of one giant run. Resume on error — every page cursor and count is persisted. When a request fails mid-run (in one run it stopped at 4,782 followers after 96 pages read), the job logs the error, emails me a status report, and picks up from the same point on the next cycle instead of restarting. Enrichment pass — a second workflow walks the stored followers and pulls profile details, flagging commercial accounts and any public email/phone in the bio. Personal/private accounts return no contact data, which the report counts separately. Email reports — each cycle sends me a summary: profile, followers reported vs. downloaded, pages read, batch name, and the exact error if one occurred. For the Instagram data layer I used HikerAPI — I tested a few other options first, and it won on pricing and rate limits for this volume. It handled the pagination fine: the run above made 100+ requests without me managing sessions or proxies myself. Tradeoffs / what didn't go perfectly Long extractions still fail sometimes (timeouts); resume logic is not optional at this scale, whatever API you use. Early days for me on this stack: so far it has worked well, but I'm still collecting more data before I'd call the pipeline battle-tested. I'll know more after a few full 150k-follower

On August 1 I published a report whose headline finding was that frontier LLM API prices are structurally sticky . Across 40 daily readings of an equal-weight index of ten flagship models — one per lab — not one lab had ever changed the price of an existing model. Every move in the index had come from a new model replacing an old one. August made that sentence false in three weeks. Here's what moved, why the index still ended the month lower , and what happened 72 hours after the cutoff that dwarfs all of it. The month in one table The index is the equal-weight average of ten flagships' blended price per million tokens (3 parts input to 1 part output, list prices as printed on the vendor's own pricing page). Date What happened Index ($/Mtok) Aug 1 Opening level $4.39 Aug 4 Alibaba's slot: Qwen3.7-Max → Qwen3.8-Max ($3.75 → $3.00 blended) $4.32 Aug 16 DeepSeek V4 Pro repriced : flat $0.435/$0.87 → peak $1.32/$3.96 (+264% blended) $4.46 Aug 21 GPT-5.6 Sol repriced : $5/$30 → $4/$20, labelled promotional (−29%) $4.14 Sep 1 Closing level $4.14 Net for the month: −5.7% . Since the first reading on February 23: −9.4% . Three other flagship handovers happened in August (Muse Spark 1.1 → 1.2, Grok 4.5 → 4.6, GLM-5.2 → 5.3) and moved nothing, because each successor kept its predecessor's list price. That's the pattern I described in August. The two bolded rows are the pattern breaking. Move 1: DeepSeek turned "list price" into a schedule Until 16:00 UTC on August 16, DeepSeek V4 Pro billed a single flat rate: $0.435 in / $0.87 out. Then the pricing page split it in two: Peak (01:00–04:00 and 06:00–10:00 UTC): $1.32 / $3.96 Off-peak (every other hour): exactly half — $0.66 / $1.98 The index tracks the peak rate as the list price. Two reasons. DeepSeek defines off-peak as a discount from peak, not the other way round, so peak is the published number. And a caller who doesn't schedule around the clock needs a ceiling, not a floor. But note that even the off-peak rate ($0.99 ble

Most local business lead scrapers on Apify are Google Maps scrapers with a website-crawling step bolted on. lukaskrivka/google-maps-with-contact-details is the most used (87,957 users, 4.63 stars). flash_scraper/local-business-leads is the outlier: it discovers businesses on OpenStreetMap instead of Google Maps, and includes MX email verification in its $3 per 1,000. Every figure below was read from Apify's public Store API ( GET /v2/store ) on 2026-09-05 — including every user count, so they are all on the same footing. The per-actor endpoint ( GET /v2/acts/<id> ) can read one higher: it gives flash_scraper/local-business-leads 33 rather than 32, and code-node-tools 33 as well. Prices, users and ratings change; the Pricing tab on each actor page is authoritative. Disclosure: I publish flash_scraper/local-business-leads , one of the actors compared here. Its limits are listed in the same detail as everyone else's, including the one that will disqualify it for many buyers. How prices are normalised These actors bill per event, and the events differ in kind, which makes headline prices misleading. Some charge per place found. Some charge separately for the website crawl that actually produces the email. Some charge again to verify that the email is deliverable. The table lists the primary per-result event multiplied by 1,000 at the free-plan rate , then names the add-on events, because a $5 per 1,000 place price with a $100 per 1,000 email-verification add-on is not a $5 tool. Paid Apify plans get tiered discounts on several of these actors, ours included — and on the add-on events the discount can be enormous. lukaskrivka's three $100-per-1,000 add-ons fall to $4.00 (email verification), $7.50 (lead enrichment) and $10.00 (social-profile enrichment) per 1,000 on Bronze, and lower again above it (Store pricing record read 2026-09-05). Our own free-plan-to-Diamond spread is about 30 percent. So if you are on a paid plan, re-read every figure below off the Pricing tab:

Your team decides to add an AI agent to your Laravel application. The initial plan seems straightforward: give the LLM access to your existing REST API, let it figure out the endpoints, and watch it automate customer support. Then production happens. The agent calls GET /api/users and pulls 14,000 records into its context window, blowing past the token limit and costing $0.80 for a single turn. It tries to POST to a nested route, guesses the JSON payload wrong, and triggers a validation exception. Worse, it calls the refund endpoint without checking if the current user actually owns the order, because your API relies on middleware that the agent orchestrator bypassed. Building an API for human developers or frontend frameworks is fundamentally different from building an API for an AI agent. Humans read Swagger docs and write deterministic code. Agents read JSON schemas, reason probabilistically, and execute in a loop. If you just expose your Laravel routes to an LLM, you aren't building an agent. You're building a very expensive, highly unpredictable curl client. TL;DR: Turning a Laravel backend into an agent-ready system requires shifting from HTTP-centric controllers to action-centric tools. You must generate strict JSON schemas from PHP attributes, enforce authorization inside the tool boundary, curate outputs to protect the context window, handle failures without breaking the agentic loop, and offload execution to background queues. 📋 Table of Contents 1. Stop Exposing Routes, Start Exposing Actions 2. Generating Tool Schemas from PHP Attributes 3. The Authorization Gap: When Agents Bypass Policies 4. Taming the Context Window with Structured Tool Outputs 5. Surviving the "Infinite Retry" Loop on Flaky Tools 6. Building the Agentic Loop with Laravel Queues 7. Defending Against Tool-Output Prompt Injection 8. Observability: Tracing the Agent's Thought Process The Agent-Ready Backend Checklist 1. Stop Exposing Routes, Start Exposing Actions Scenario: You give an L

A browser and a language model can look at the same URL and effectively see two different things. A browser sees a rendered interface: navigation, cookie banners, buttons, ads, sidebars, images, scripts, interactive components, and eventually the text a human came to read. A language model sees whatever representation we decide to give it. That distinction matters when the URL is going into a RAG pipeline. Open the developer tools on any major news or documentation site and look at the raw HTML. A typical article page runs between 300KB and 800KB of markup. The article text itself is usually between 2KB and 10KB. The ratio of markup to content is consistently between 10:1 and 40:1 depending on how heavily templated the site is. When you pass raw HTML to a language model, you are passing all of it, and most pipelines treat this as an acceptable default. Perceive is the endpoint we built to fix that. You give it a URL. It returns clean Markdown. This post is about what happens in between and why we made the engineering decisions we did. Why raw HTML is a poor RAG input The token waste is real but it is not the worst problem. Three failure modes compound each other. Token waste . A blog post with 800 words of real content can run to 6,000–12,000 tokens as raw HTML once you include navigation, scripts, inline styles, and layout markup. The same content in Markdown is often 900–1,200 tokens. That is not just a cost issue. It is context window space that cannot go to content. Embedding contamination . Embedding models are trained predominantly on natural language. When you embed a chunk containing <div class="sidebar-widget__title">Related Articles</div> alongside the article content, the vector is pulled toward the markup semantics rather than the content semantics . The embedding does not cleanly represent the article; it represents a mixture of the article and the site's component naming conventions. Retrieval degrades as a result: chunks that should be semantically si

How to Get YouTube Transcripts as a Developer (4 Methods That Work in 2026) YouTube transcripts unlock a lot: AI video summarizers, searchable course databases, RAG over video libraries, dataset generation for fine-tuning, repurposing videos into articles. But getting transcripts programmatically is full of sharp edges: disabled captions, rate limits, datacenter IP blocks, and YouTube's ever-changing frontend. This guide walks through every practical method with working code. Method 4 is the managed service I run. Skip ahead if you just want the API call. The DIY methods below are real and will serve you well for small jobs. What you're actually fetching YouTube stores captions as timed tracks in two flavors: Manual captions : uploaded by creators, best accuracy Auto-generated captions : YouTube's speech recognition, most videos Each track is text plus timing ( text/start/duration ), servable as SRT, VTT, or YouTube's timedtext XML. Everything below ultimately resolves to that shape. Method 1: youtube-transcript-api (Python) The standard open-source library. Start here for scripts and prototypes. pip install youtube-transcript-api from youtube_transcript_api import YouTubeTranscriptApi video_id = " dQw4w9WgXcQ " # the ID from the watch URL transcript = YouTubeTranscriptApi . get_transcript ( video_id ) for entry in transcript : print ( f " [ { entry [ ' start ' ] : . 2 f } s] { entry [ ' text ' ] } " ) It returns a list of dicts, one {'text', 'start', 'duration'} per segment. For other languages, list what's available first, then fetch or translate: tl = YouTubeTranscriptApi . list_transcripts ( video_id ) for t in tl : print ( t . language_code , " generated: " , t . is_generated ) transcript = YouTubeTranscriptApi . get_transcript ( video_id , languages = [ " id " , " en " ]) track = tl . find_transcript ([ " en " ]) translated = track . translate ( " id " ). fetch () # free, server-side Handle the caption-less case explicitly instead of catching bare Exception .

"A URL shortener" sounds like a weekend project. Slug in, long URL out, 302 , done. That's what I thought too. Then real usage showed up: links opened inside Instagram's in-app browser and didn't convert, bot traffic wrecked the analytics, and one link needed to send a US visitor somewhere different from an EU visitor. Suddenly the "trivial" part was 5% of the work. I built the whole thing on FastAPI + Redis + MySQL + htmx , deliberately with no frontend framework . This post is about the parts that turned out to be interesting — the redirect hot path, geo/device routing, and escaping in-app browsers — and why htmx was the right call for a one-person team. Disclosure: I build tapurl.io , a link shortener for marketers. This is a write-up of the engineering behind it, not a pitch — everything below is patterns you can apply to any shortener. The redirect is a hot path, so treat it like one Every other page in the app can be a bit slow. The redirect cannot. It sits in front of someone's click, and it runs on every click, so it has to be a tight, predictable read. The naive version hits your database for every redirect: @app.get ( " /{slug} " ) async def redirect ( slug : str ): link = await db . fetch_link ( slug ) # DB round-trip on every click if not link : raise HTTPException ( 404 ) return RedirectResponse ( link . destination , status_code = 302 ) That's fine until you have traffic. The slug-to-link lookup is a near-perfect cache candidate — a slug maps to the same link record every time. So the real path reads from Redis first and only falls back to MySQL on a miss: async def resolve ( slug : str ) -> Link | None : cached = await redis . get ( f " link: { slug } " ) if cached : return Link . parse_raw ( cached ) link = await db . fetch_link ( slug ) if link : await redis . set ( f " link: { slug } " , link . json (), ex = 3600 ) return link Two things worth saying out loud: Cache the lookup, not the decision. You cache the link record, but the actual destination

GPT-6 Astra API: دليل عملي للتسعير والترحيل من GPT-5.6 Sol أصدرت OpenAI نموذج GPT-6 Astra في 3 سبتمبر 2026، أولًا لمجموعة محدودة من المؤسسات، ثم تدريجيًا لمستخدمي ChatGPT Plus وPro وBusiness وEnterprise، وواجهة برمجة تطبيقات OpenAI، وMicrosoft Azure، وAWS Bedrock. معرف النموذج هو gpt-6-astra ، مع نافذة سياق تبلغ 1,050,000 رمز، وتصفه OpenAI بأنه «أفضل نموذج لهندسة البرمجيات حتى الآن». وهو أول نموذج تصنفه OpenAI بأنه حرج لقدرته على الأمن السيبراني، ما يؤثر في سلوكه عبر واجهة برمجة التطبيقات. جرّب Apidog اليوم هذا الدليل يركز على واجهة برمجة التطبيقات: معرف النموذج ونقاط النهاية، أول طلب، مستويات جهد الاستدلال، التسعير، تغييرات الترحيل من GPT-5.6 Sol، وطريقة اختبار ما إذا كانت الزيادة في السعر تستحقها. الخلاصة السريعة معرف النموذج هو gpt-6-astra ، وهو متاح عبر إكمال الدردشة، والردود، والدفعات. لا يدعم الوقت الحقيقي أو المساعدين أو الضبط الدقيق. نافذة السياق: 1,050,000 رمز. الحد الأقصى للإخراج: 128,000 رمز. تاريخ قطع المعرفة: 30 أبريل 2026. يدعم إدخال النص والصورة، ويُخرج النص. التسعير القياسي لكل مليون رمز: 10 دولارات للإدخال، و1 دولار للإدخال المخزن مؤقتًا، و12.50 دولارًا لكتابة ذاكرة التخزين المؤقت، و50 دولارًا للإخراج. تتضاعف أسعار السياق الطويل عند تجاوز 272,000 رمز إدخال. الدفعات وFlex بنصف السعر، بينما الوضع السريع بضعف السعر. مستويات جهد الاستدلال: low و medium و high و xhigh و max . لم يعد none أو minimal متاحًا. أزالت OpenAI معاملات temperature و top_p و logprobs . كما استُبدل prompt_cache_retention بـ prompt_cache_options.ttl . يبقى GPT-5.6 Sol متاحًا بالسعر الترويجي 4 دولارات للإدخال و20 دولارًا للإخراج حتى 21 نوفمبر 2026 على الأقل؛ لذلك يكلف Astra 2.5 ضعف هذا السعر. GPT-6 Astra في لمحة العنصر القيمة معرف النموذج gpt-6-astra نافذة السياق 1,050,000 رمز أقصى إخراج 128,000 رمز تاريخ قطع المعرفة 30 أبريل 2026 الأنماط الإدخال: نص، صورة. الإخراج: نص نقاط النهاية إكمال الدردشة، الردود، الدفعات غير مدعوم الوقت الحقيقي، المساعدون، الضبط الدقيق الميزات البث، المخرجات المهيكلة، استدعاء الدوال، البحث في الملفات، البحث في الويب، التخزين المؤقت للمطالبات، إدخال الصور الأد

A headless CMS migration often sounds simple: Contentful → Strapi Move the content, update the API calls, fix a few components, and you're done. Except... you're usually not. The hardest part of a headless CMS migration isn't moving the content. It's managing the contract between the CMS and the frontend . And if your React or Next.js application is tightly coupled to the CMS response structure, changing the CMS can turn into a much bigger project than expected. The problem Imagine your frontend directly consumes Contentful responses: const ProductCard = ({ product }) => { return ( < article > < h2 > { product . fields . title } < /h2 > < p > { product . fields . description } < /p > < img src = { product . fields . image . fields . file . url } / > < /article > ); }; It works. Until you migrate to Strapi. Now the response might look completely different: product . title product . description product . image . url Suddenly, the frontend needs to understand both CMS structures. And this problem isn't limited to simple fields. Things become much more complicated with: Rich text Media and assets References Nested relations Localization Draft/preview content SEO metadata Dynamic components Pagination GraphQL vs REST Different content modeling approaches The architecture I prefer Instead of allowing React components to consume the CMS directly, introduce a layer between the CMS and the application. ┌───────────────┐ │ Strapi │ └───────┬───────┘ │ ▼ ┌───────────────┐ │ CMS Adapter │ └───────┬───────┘ │ ▼ ┌───────────────┐ │ Domain Model │ └───────┬───────┘ │ ▼ ┌───────────────┐ │ React / Next │ └───────────────┘ The frontend doesn't need to know whether the data came from Strapi, Contentful, Shopify, WordPress, or something else. It just receives the data it needs. For example: type Product = { id : string ; title : string ; description : string ; image : { url : string ; alt : string ; }; }; The CMS adapter is responsible for transforming the CMS response into this model