标签：#cloudflare

Your robots.txt may express an AI policy you did not write. We checked the homepage and robots.txt of 9,037 live AI tools listed on directree on 6 and 7 September 2026. Of those, 945 explicitly disallow OpenAI’s GPTBot in its own user-agent group: 10.5% of the sample. Treat AI crawler rules as deployment configuration. Review them when you change hosting, enable a CDN feature, adopt a starter template, or hand site operations to someone else. Read the full research and methodology . GPTBot, search, and user browsing are separate A common configuration blocks model training while keeping a site available in AI-assisted search and browsing: User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / These are separate crawlers with separate purposes. In our sample, 839 of the 945 sites that block GPTBot, or 88.8%, still allow OAI-SearchBot. That is a deliberate and useful distinction if your goal is to opt out of training while remaining eligible to be cited in ChatGPT search. The same pattern appears across AI labs. ClaudeBot is explicitly blocked by 10.1% of the 9,037 tools, while Claude-SearchBot is blocked by just 0.1%. Google-Extended is blocked by 9.9%, but its purpose is also distinct from ordinary Google Search crawling. Do not assume a broad-looking rule has the result you want. Check the actual crawler names and decide which capabilities you want to permit. A safe way to review your file Start by opening the public URL: https://your-domain.example/robots.txt Then look for three things: A named crawler group, such as User-agent: GPTBot . A Disallow: / directly inside that group. A wildcard group, User-agent: * , that could affect all crawlers. Our measurement only counts a site as blocking GPTBot when the named GPTBot group itself contains Disallow: / . This matters because ordinary technical exclusions are widespread. Only 31 sites in the 9,037-site sample, or 0.3%, block every crawler outright. Meanwhile, 44% have a path-level Disallow rule in a wildc

The idea I wanted a small side project that could actually run itself once shipped — no cron jobs to babysit, no upstream API to go down at 3am and take my uptime with it. That constraint led somewhere specific: an API that validates common business data formats — phone numbers, IBAN, VAT/tax IDs, BIC/SWIFT codes, credit card numbers, postal codes — using nothing but offline checksum and format rules. No third-party lookups. No API keys to rotate for an upstream provider. No rate limits inherited from someone else's infrastructure. If it's slow or wrong, it's my bug, not a dependency's outage. The stack Hono on Cloudflare Workers — TypeScript, no cold starts, runs on the free tier comfortably up to 100k requests/day libphonenumber-js , ibantools , jsvat , card-validator — all well-maintained, all pure computation, zero network calls Vitest for tests, run against real fixtures (not made-up test data — every "valid" example in my test suite is a real IBAN/VAT/card number pulled from each library's own published examples, verified against the actual library output before I trusted it) The whole thing is about 300 lines of TypeScript across the router and six validator modules. Small enough to actually reason about, which mattered more to me than feature breadth. app . post ( " /v1/iban/validate " , async ( c ) => { const body = await c . req . json < { iban ?: string } > (). catch (() => null ); if ( ! body ?. iban ) { return c . json ({ error : " missing required field: iban " }, 400 ); } return c . json ( validateIban ( body . iban )); }); The part that actually surprised me I expected the code to be the hard part. It wasn't. Deploying and listing it on RapidAPI was. Two things stood out: CORS mattered even though I "shouldn't" need it. Real production traffic through RapidAPI's gateway is server-to-server — CORS is a browser-enforced concept, so I assumed it was irrelevant. But RapidAPI's own in-dashboard request tester runs as a real browser fetch, and without an O

Nineteen OOM kills in nine days. Ten WordPress apps on a 32GB shared box. One of them a client site that took a CPU spike on 14 July during a paid campaign burst, and pushed the whole tenant into the wall. This post is the diagnosis before the rebuild. What we actually found when we stopped guessing. Two layers of Cloudflare, one page cache plugin, one preloader being silently challenged, and a language subpath that was cold every time it mattered. I'm writing it partly for anyone who runs multi-tenant WordPress on Cloudways or similar, and partly as a reminder for future-me. There's a checklist at the end. Steal it. The client is anonymized throughout. Every number is real. The stack Traffic hits two Cloudflare layers before it reaches origin. Both are Cloudflare, but they're different zones on different accounts, and they own different things. [ visitor ] ↓ [ Upstream Cloudflare zone (managed by a third party) ] ← DNS, SSL, HTML edge cache ↓ [ Cloudflare Enterprise add-on sold by Cloudways ] ← WAF, bot, rate limit, AI crawler block ↓ [ Cloudways origin: nginx + PHP-FPM ] ↓ [ WordPress + WPML + Elementor + FlyingPress ] Two Cloudflares isn't a mistake. The domain has been on Cloudflare via an upstream party since before the site moved to Cloudways. When Cloudways later offered a Cloudflare Enterprise add-on for its security stack, we kept both. We manage the Cloudways side. We don't own the upstream zone, which shapes what we can and can't do without a request going out. The trap is that both layers can cache HTML, and both can serve security challenges. If nobody writes down which layer does what, they fight. Our ownership split ended up like this: Layer Owns Upstream Cloudflare (third party) DNS, SSL, HTML edge cache, purge lifecycle Cloudways CF Enterprise add-on WAF, bot management, rate limiting, AI crawler blocking, ScrapeShield, Browser Integrity Check FlyingPress Origin page cache, Cloudflare integration pointed at the upstream zone, purge rules Cloudways "

Originally published on indiecore.net . I deployed, opened the console on the live site out of habit, and found this: Loading the script 'https://static.cloudflareinsights.com/beacon.min.js/v3d52…' violates the following Content Security Policy directive: "script-src 'self' 'unsafe-inline' 'inline-speculation-rules'". The action has been blocked. I had not added that script. It is in no template, no build output, and no dependency. grep -c cloudflareinsights dist/index.html returns 0. It is not in the page you build Cloudflare Web Analytics has an automatic mode, on by default when a site is added, that injects beacon.min.js into HTML responses at the edge. Your origin never sees it. Your repository never contains it. It also does not inject for everything. I fetched the same URL with curl, then again with a full desktop browser User-Agent, and neither response carried the script. Only a real browser navigation gets it, which is why Lighthouse saw it and my terminal did not. That combination is worth sitting with for a second. The artefact exists in production, is absent from your source, and cannot be reproduced with the tool most of us reach for first when we want to see what a server actually returned. Nothing was tracked The CSP did its job. From the Lighthouse network trace: url : https://static.cloudflareinsights.com/beacon.min.js/v3d52… resourceType : Script statusCode : -1 transferSize : 0 Status −1 with zero bytes transferred means the request never started. The browser matched the URL against script-src , found no permitted source, and refused before opening a connection. No data left anyone's browser. So the console error is the sound of a guard working. It still costs something: a logged error drops the Best Practices category from 100 to 92, and a red line in the console trains you to ignore red lines in the console. The fix everyone reaches for is the wrong one here Search the error and the common answer is to add https://static.cloudflareinsights.com

Cloudflare has added optional OAuth scopes, letting client owners mark which permissions users may deselect at consent. The company names MCP servers as the motivating case, since agents request the union of everything they might do. Partial consent exists elsewhere, but developer control over which scopes are droppable does not. By Steef-Jan Wiggers

Originally published on indiecore.net . I moved this site off a hosted blogging platform onto Cloudflare, with GitHub Actions doing the building and Cloudflare doing the serving. It costs nothing, deploys in about two and a half minutes, and refuses to publish anything that fails its checks. Getting there took longer than it should have. Here is the setup, and the five things that tripped me up — none of which are obvious from the documentation. The shape of it git push └─ GitHub Actions ├─ build generate the site ├─ verify dead links, missing images, bad metadata, broken redirects ├─ Lighthouse fail if performance/accessibility/SEO drop below budget └─ deploy upload to Cloudflare The important part is that deploy depends on the checks . A broken build never reaches the internet. Pull requests get a preview URL; merges to main go live. The triggers and permissions that make that safe: ci-cd.yml — triggers and permissions name : CI/CD # Build and verify every change; deploy previews for PRs and production from main. # Deploy jobs depend on the quality gates, so nothing ships unverified. on : push : branches : [ main ] # The SEO watch ledger is machine-written, is never part of dist/, and is # committed daily. Deploying the site again for it would be pure noise — and # would re-trigger the SEO ping through workflow_run every single day. paths-ignore : - ' _source/seo-watch.json' pull_request : branches : [ main ] workflow_dispatch : # Least privilege by default; individual jobs elevate only what they need. permissions : contents : read # Supersede in-flight runs for a branch, but never interrupt a production deploy. concurrency : group : ci-cd-${{ github.ref }} cancel-in-progress : ${{ github.ref != 'refs/heads/main' }} env : # wrangler ships as a pinned devDependency; never phone home from CI WRANGLER_SEND_METRICS : " false" permissions: contents: read at the top means every job starts with the minimum, and only the one that comments on pull requests gets more. The c

Cloudflare AI Search is a built-in search and retrieval service designed to give AI agents and applications a ready-to-use search engine over custom data. It supports agent integration, multimodal search, and seamless integration with other Cloudflare tools. By Sergio De Simone

Cloudflare KV for Session Caching in Multi-Tenant FastAPI: Reducing PostgreSQL Load Without Redis Complexity Every SaaS I've built hits the same wall: session validation on every request hammers PostgreSQL. You add Redis, suddenly you're managing another service, debugging cache invalidation, and paying for redundancy you don't need. Then I discovered Cloudflare KV sits between your users and origin server. It's not a replacement for PostgreSQL—it's a read cache positioned at the edge that auto-syncs on writes. For multi-tenant session and permission data, this eliminates 60–80% of auth-related database queries without the operational complexity of Redis. This is the approach I use in CitizenApp. Here's why it works, how to implement it, and where I nearly broke production. Why Cloudflare KV Beats Redis for Session Caching Redis requires: A separate service deployment (Render, AWS ElastiCache) Connection pooling logic in your app Cache invalidation strategies you'll get wrong Monitoring for memory leaks and eviction Cost that scales with your hot data size Cloudflare KV requires: A binding in your edge worker (one line of config) Simple key-value storage at 200+ edge locations Automatic TTL expiration Zero operational overhead—Cloudflare manages it Here's my honest take: I prefer KV because I don't have to think about it. My workers validate JWT tokens and fetch session data from KV before even routing to my FastAPI origin. Cache misses flow to PostgreSQL and write back to KV. No connection pools. No eviction policies. No debugging Redis memory fragmentation at 3 AM. The tradeoff? KV is slower than in-memory Redis (ms vs microseconds), but for session lookups happening 200+ times per second per user at global scale, edge-cached responses beat origin-fetched ones every time. Architecture: Edge Validation + Origin Sync Your flow looks like this: Request hits Cloudflare Worker Worker checks KV for session + permissions (hit = serve immediately) KV miss → fetch from Fas

Cloudflare Workers can now accept inbound TCP connections through a new connect(socket) handler routed via Spectrum, ending an eight-year restriction to HTTP. Containers get full-duplex gRPC in any language, while Workers get unary and server-streaming through automatic gRPC-web translation. Everything is private beta. By Steef-Jan Wiggers

Cloudflare announced Wallets, giving agents a stablecoin balance and spending controls, though only handle claiming is live and squatting complaints have already surfaced. Payments run on x402, now hosted by the Linux Foundation. The controls bound single payments, not sequences, leaving composition to the application above. By Steef-Jan Wiggers

Honestly, when I saw Cloudflare's announcement, my first reaction wasn't "oh cool, something new"—it was "there goes another giant company proving the thing I've been saying all along." What Cloudflare Actually Did On August 4, Cloudflare (yes, the infrastructure giant that blocks traffic and runs CDNs for half the internet) launched "Cloudflare Wallets" and something called cloudflare.pay. It gives AI agents three things they didn't have before: An identity —a recognizable wallet handle so others know exactly which agent is paying A wallet —funded with stablecoins, so the agent can actually pay A spending cap —and this one is enforced by Cloudflare's infrastructure itself The structure here is what I think matters most. You (the human) hold an Account Wallet where the funds live; then, through an API key, you grant a limited slice of spending power to individual Virtual Wallets that your agents actually use. Here's the analogy that makes it click: the Account Wallet is your company's master account, and each Virtual Wallet is a prepaid card with a spending limit that you hand to one of your AI employees. The only difference is these "employees" are AI, and the limit on the card isn't managed by a credit card company's risk engine—it's written directly into Cloudflare's infrastructure. Payments run through the now widely-discussed x402 protocol: an agent wants to buy a service, and it pays for that one transaction on the spot with stablecoins. I should be upfront about something: it's not fully usable yet. As of August 5, it's in a "launched, you can reserve your cloudflare.pay name" state. The real funding, Virtual Wallets, and programmatic spend controls are, per Cloudflare, coming "over the next few months." So this is a clear directional statement, not a mature product you can fully adopt today. Why I'm Not Reading This as "One New Product"—I'm Reading It as an Industry Consensus If this were just Cloudflare doing its own thing, I wouldn't bother writing about i

Cloudflare recently open-sourced Cloudflare OS. It allows enterprise teams to output work artifacts grounded in enterprise knowledge, know-how, and provisioned connectors, automate repetitive workflows with optimized token cost (with AI assistance only where needed), and build personal, shareable, customizable work software that caters to specific, complex use cases within a secure sandboxed model By Bruno Couriol

Cloudflare recently introduced Kitesurf, a lightweight browser built for automated workloads. Kitesurf runs browser components in isolated WebAssembly/Rust environments on Cloudflare Workers and supports the Chrome DevTools Protocol, allowing tools such as Playwright and Puppeteer to drive it with lower resource overhead than a full Chromium browser. By Renato Losio

Cloudflare, Astro, AI agents, GitHub Actions, issue triage, agentic AI, software architecture, open source, developer tools, AI automation, automated testing, human in the loop, agent workflows, GitHub, software engineering, AI software development, bug triage, continuous integration, developer productivity, autonomous agents, AI coding, Cloudflare Workers, Flue, triagebot By Leela Kumili

Cloudflare is introducing WriteGuard, now in private beta, to provide fine-grained security controls for MCP (Model Context Protocol) servers. It aims to make AI agents safer by controlling their access to tools that can modify data or perform actions, rather than simply read information. By Sergio De Simone

Eight user agents, and the one that decides whether ChatGPT cites you is not among them. An r/SEO post from April, 53 points and 40 comments, says Cloudflare quietly cut the author's site off from ChatGPT, from Perplexity and from Google's AI Overviews. I went and read the block. It names ChatGPT's training crawler and not its search crawler, and it never mentions Perplexity at all. The Google agent it does name is one Google says does not affect inclusion or ranking in Search. Take just two of the eight agents. GPTBot governs inclusion in OpenAI's training data, and Google-Extended governs grounding in Gemini Apps. Those are real things to give up. They are not the three things the warning names. What is actually in the file Cloudflare prints the whole block on its managed robots.txt page : # BEGIN Cloudflare Managed content User - Agent : * Content - signal : search = yes , ai - train = no , use = reference Allow : / User - agent : Amazonbot Disallow : / User - agent : Applebot - Extended Disallow : / User - agent : Bytespider Disallow : / User - agent : CCBot Disallow : / User - agent : ClaudeBot Disallow : / User - agent : Google - Extended Disallow : / User - agent : GPTBot Disallow : / User - agent : meta - externalagent Disallow : / # END Cloudflare Managed Content Read that against Cloudflare's own crawler reference table and a pattern falls out. GPTBot is in, OAI-SearchBot is out. ClaudeBot is in, Claude-SearchBot is out. For both of those pairs the table calls the blocked agent an AI Crawler and the one left alone AI Search. Applebot-Extended is in and plain Applebot is not, the same split again. The table has no Applebot-Extended row, so it cannot tell you what Cloudflare calls that one. The block runs along the training and search seam, and that looks deliberate to me. OpenAI's side of it is one line: "Each setting is independent of the others". I worked through the three OpenAI bots and which one governs search visibility in an earlier post , so I will

Cloudflare has released cloudflare/ci, a CI SDK that defines pipelines in TypeScript on top of Cloudflare Workflows, giving each step durable retries and replay, concurrent steps by default and Sandbox snapshot caching. It targets the Workers runtime and depends on Artifacts, still in private beta, so the transferable lesson is the durable-step model rather than a drop-in CI replacement. By Mark Silvester

I build a web service that preserves evidence of harassment on social platforms. The core feature is a single thing: automatically capture a real screenshot of the offending post. There was no substitute for it. I built an alternative that pulled the text through an API and rendered a tidy "evidence card" image, and threw it away. An image you can author freely afterwards proves nothing. Here's the conclusion first. Third-party wrappers eventually die, and when they do, the failure comes back as a plausible-looking image rather than an error. The first approach was refused by the other side I started with Cloudflare Browser Rendering. The wiring worked. The capture didn't. X blocks headless browsers. The request times out YouTube refuses script injection under a Trusted Types CSP. There's no way to make it render the comment Neither is a bug in my implementation — that is how they are built. So I declared Cloudflare alone impossible for this and moved to a service with a real browser and bot avoidance behind it. Both captures started working. For X, open the post page and clip the tweet element. For YouTube, open the URL with &lc= and screenshot just that comment element. Element screenshots have one trap worth knowing: selector_algorithm=clip returns a blank image when the element sits below the fold. The selector matches, the capture "succeeds," and the file is empty. That took a while to see. ytd-comment-thread-renderer :has ( a [ href *= "lc=ID" ]) A parameter that had worked started returning 400 I wanted timestamps rendered in Japan time, so I passed time_zone: Asia/Tokyo . One day every request started coming back 400. Every capture failed. The provider had narrowed which timezones they accept. Nothing changed on my side. I could diagnose it immediately only because I was storing the raw error body in the database. The response went into rawPayload.screenshotError , so opening one row told me why. Without that, this starts as "captures stopped working, no ide

Cloudflare launched agent tracing, adding spans for agent invocations, model calls, tool runs, and approvals to existing Workers traces. Sessions replay turn by turn, though the docs warn traces are not lossless and payloads may be truncated. Payload recording defaults differ by framework, and from October 1, 2026 every span counts as a billable event. By Steef-Jan Wiggers

Cloudflare recently introduced Cache Response Rules, a rules engine that operates after an origin server responds but before content is written to Cloudflare's cache. Previously, Cache Rules operated only on request attributes. Cache Response Rules add a response phase that evaluates origin responses before they are cached. By Renato Losio