标签：#robots

Your robots.txt may express an AI policy you did not write. We checked the homepage and robots.txt of 9,037 live AI tools listed on directree on 6 and 7 September 2026. Of those, 945 explicitly disallow OpenAI’s GPTBot in its own user-agent group: 10.5% of the sample. Treat AI crawler rules as deployment configuration. Review them when you change hosting, enable a CDN feature, adopt a starter template, or hand site operations to someone else. Read the full research and methodology . GPTBot, search, and user browsing are separate A common configuration blocks model training while keeping a site available in AI-assisted search and browsing: User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / These are separate crawlers with separate purposes. In our sample, 839 of the 945 sites that block GPTBot, or 88.8%, still allow OAI-SearchBot. That is a deliberate and useful distinction if your goal is to opt out of training while remaining eligible to be cited in ChatGPT search. The same pattern appears across AI labs. ClaudeBot is explicitly blocked by 10.1% of the 9,037 tools, while Claude-SearchBot is blocked by just 0.1%. Google-Extended is blocked by 9.9%, but its purpose is also distinct from ordinary Google Search crawling. Do not assume a broad-looking rule has the result you want. Check the actual crawler names and decide which capabilities you want to permit. A safe way to review your file Start by opening the public URL: https://your-domain.example/robots.txt Then look for three things: A named crawler group, such as User-agent: GPTBot . A Disallow: / directly inside that group. A wildcard group, User-agent: * , that could affect all crawlers. Our measurement only counts a site as blocking GPTBot when the named GPTBot group itself contains Disallow: / . This matters because ordinary technical exclusions are widespread. Only 31 sites in the 9,037-site sample, or 0.3%, block every crawler outright. Meanwhile, 44% have a path-level Disallow rule in a wildc

The company is testing robots on tasks that can performed by technicians.

The company is testing robots that can swap cables, reset servers, and take on other tasks performed by technicians, fueling concerns among some workers that their jobs could be at risk.

The potential for AI to automate scientific research and manufacturing must be balanced with new risks, Anthropic says.

Beijing’s endlessly delightful Robot Games featured tons of impressive stunts. But the most mind-blowing tricks challenged the humanoid’s brain, not its brawn.

Record-breaking robot races are less substantial than household chore challenges.

FCC ban on foreign-made robots accelerated RoboStore’s US manufacturing plans.

“We're not necessarily building in a dogmatic fashion towards full autonomy.”

The Unitree G1 has found online fame as a relatively affordable robot that can charm a crowd. But can it ever hold down a real job?

Bartesian's cocktail makers would be best described like a Keurig or Nespresso machine, but for alcoholic drinks.

Ati Robotics assembles its robots in India and uses just a few Chinese parts—a strategy that could pay off as the Trump administration cracks down on Chinese humanoids.

July 2026 is over. The month that opened with AUTONOMOUS 2026 and WAIC 2026 running simultaneously on opposite sides of the Pacific closed with the sector tallying what it built. The number that defines the period is $55.8 billion in robotics funding across H1 - nearly double the prior full-year record. But the more durable signal from this week is operational rather than financial: Neura Robotics has a confirmed deployment date at a Schaeffler facility in December, NVIDIA's simulation-to-real pipeline is now functional at production scale, and five simultaneous shifts are reshaping factory floors right now, not in 2027. The questions that drove the first half of 2026 - does Physical AI work, is the funding real, will the robots actually arrive - are no longer interesting. H2 starts with harder ones. Stats: Value Description $55.8B Robotics funding raised in H1 2026, nearly double the prior annual record $8.6B Humanoid startup funding in H1 2026 alone, 1.8x all of 2025 December 2026 Confirmed first deployment of Neura Robotics humanoids at Schaeffler's German facilities 5 Simultaneous operational shifts reshaping factory floors identified in the mid-2026 analysis Neura Robotics Has a Deployment Date: December 2026 in a Schaeffler Factory Most Physical AI deployment announcements are directional. "We are partnering with X to explore robotics in our facilities" is a press release. A confirmed month and a specific facility is a contract. Neura Robotics confirmed that Schaeffler - one of the key investors in its $1.4 billion Series C alongside Amazon, Nvidia, Qualcomm, and the European Investment Bank - plans to deploy Neura's humanoids in its German facilities in December 2026 . Schaeffler manufactures precision bearings and components for electric vehicles, operating in environments where dimensional tolerances are measured in micrometers. Deploying a humanoid robot in that context is a fundamentally different challenge than warehouse pick-and-place or automotive sequ

The latest version of Google DeepMind's AI model includes a significant jump into “physical AGI.” But plopping AI into the real world comes with risks.

A German startup sent a camera-wearing chef to my apartment. In exchange for a free lunch, I let them record every chop and stir to train future humanoids.

Most lists that claim to help you block AI crawlers are copy-pasted and dangerously wrong about the two tokens that actually matter. Sorting them properly is not an abstract taxonomy exercise. It is the single decision that determines whether your content vanishes from AI answers while training continues — or vice versa. We maintain the crawler registry in lib/ai-crawlers.ts that powers techpotions’ free AI robots.txt generator . Every agent string and description was verified against the operator’s own crawler documentation. The registry holds 15 verified bots across four categories, and that four-way split is this article’s structure, because the categories map directly to what blocking costs you. Two tokens almost everyone gets wrong Before the list, the single most important correction to make, and almost every listicle on this query gets it wrong: Google-Extended and Applebot-Extended are not crawlers. They are robots.txt tokens — product controls that govern whether your content is used for Gemini and Apple foundation-model training. Blocking Google-Extended does not affect Google Search crawling, Google ranking, or regular Applebot search indexing. People block them believing they are opting out of AI Overviews, and are actually opting out of nothing they think they are, while leaving search indexing completely untouched. Platforms have started wiring these tokens into one-click controls. Cloudflare’s managed robots feature, released mid-2025, lets you add AI crawler rules through a dashboard toggle rather than editing a raw file — but the underlying token logic above still applies. Block AI crawlers: the four categories that decide the cost Every AI crawler we track belongs to one of four categories. The category tells you the cost of blocking it. Training crawlers scrape pages to feed a model that may never cite you. Blocking them is a defensive data decision. Assistant crawlers fetch pages to answer a live user’s question and can cite and link you. Blockin

With “blessings” from the original Jibo founders, iKairos is a wearable or desk-mounted “AI journal” that turns your family moments into AI images and video.

Union previously warned automaker that any robot deployment must be negotiated.

Ukraine's battlefield surge of robots now features airdrops and beach assaults.

The CEO of Foundation Future Industries, which counts the president’s son as its chief strategy adviser, tells WIRED it’s exploring some “kinetic things.”

Hyundai aims to deploy 25,000 Atlas robots starting with US factories in 2028.