标签：#t

Last week, I asked my AI agent to scrape 300 job listings from a recruiting site. It broke at row...

Everyone talks about AI speeding up coding. Nobody talks about debugging AI-generated code. Last month, I spent three hours hunting down a bug in a 20-line function that an LLM wrote in thirty seconds. That's not a productivity gain—that's a productivity swap. You trade typing speed for debugging speed, and most of the time the trade is terrible. I've been using AI assistants for about a year now, mostly Claude and GPT-4, and I've noticed a pattern. The first version of any moderately complex piece of code always has at least one subtle mistake. Not syntax errors—those are easy. I'm talking about logical off-by-ones, missing edge cases, or completely hallucinated API calls. And the worst part? The AI writes the code with such confidence that you assume it's correct. You run it, it crashes, and you spend ten minutes thinking you misused the function before you finally look at the generated code with a suspicious eye. Let me show you a concrete example. I was building a small Node.js service that fetches data from a paginated REST API and merges the results. I asked the AI to write a function that handles pagination with a while loop and an offset parameter. Here's what it gave me: async function fetchAllPages ( baseUrl , limit = 100 ) { let offset = 0 ; let allData = []; let hasMore = true ; while ( hasMore ) { const response = await fetch ( ` ${ baseUrl } ?limit= ${ limit } &offset= ${ offset } ` ); const data = await response . json (); allData = allData . concat ( data . results ); hasMore = data . results . length === limit ; offset += limit ; } return allData ; } Looks clean, right? I pasted it in, ran my test, and got an infinite loop. The server returned a 400 error after a few requests, but the function kept going because response.ok was never checked. The AI assumed every call succeeds. I spent forty-five minutes debugging that—not because the bug was hard, but because I trusted the output. I added a try/catch and a status check, and then I found the real is

Before adopting DSPy, prove the LM program has a contract DSPy is easy to undersell. If you describe it as "a nicer way to write prompts", you will probably test the wrong thing. The better first test is this: can your language-model workflow be expressed as a small program with declared inputs, declared outputs, measurable examples, and an optimizer that is allowed to change prompts without changing the business boundary? That is the point where DSPy starts to make sense. The upstream README positions DSPy as a framework for programming, rather than prompting, foundation models. The current Doramagic project pack for stanfordnlp/dspy also points at the same starting command: pip install dspy . The package metadata I checked today declares name="dspy" , version 3.3.0b1 , and Python >=3.10,<3.15 . The GitHub API snapshot showed the repository was still active, with a push on 2026-07-05 and recent pull requests around empty evaluation sets, document formatting, and GEPA trace attribution. That activity is useful, but it is not the adoption proof. The proof is whether your own task can survive three separations. 1. Separate the task contract from the prompt In DSPy terms, the first artifact worth reading is not a clever prompt. It is the Signature. A Signature says what goes in and what must come out. A Module wraps that contract into something callable. An Adapter turns the contract into the provider-facing prompt format and parses the model response back into fields. That gives you a practical review question: Can the team name the output fields before anyone starts tuning wording? If the answer is no, DSPy will not rescue the workflow. You are still negotiating the task itself. The first pass should be a tiny contract: input text, retrieved context if needed, answer, citations, confidence, or whatever fields the product actually needs. Do not start with agents, tools, or multi-stage pipelines. 2. Separate compilation from runtime correctness DSPy optimizers can impr

Most AEO/GEO advice hands you a checklist: add structured data, write answer-first, put a date on it, get a score. You do all of it, and the AI answer still quotes someone else. The checklist skipped the only question that decides the outcome first: for this particular query, can an independent site get cited at all? Getting cited by ChatGPT, Perplexity, or Google's AI Overviews is a two-stage funnel, and the stages fail for completely different reasons. Grade your page without knowing which stage you're stuck at and you'll spend a day tuning headings on a page that was never eligible. Here's the model, and how to run the check yourself before you touch the formatting. Stage 1: eligibility — can the engine retrieve you at all? Answer engines are retrieval-augmented. Before anything gets generated, a retriever picks a small set of candidate pages. If you're not in that set, nothing about your writing matters. Three things decide it, and only some are visible in your HTML. The part you can check on-page — the hard disqualifiers: noindex . A <meta name="robots" content="noindex"> (or an X-Robots-Tag header) keeps you out of the indexes these engines lean on. Easy to ship by accident on a templated page. AI crawlers blocked in robots.txt . GPTBot, PerplexityBot, ClaudeBot, and Google-Extended are distinct user agents. A Disallow: / for any of them means that engine can't fetch you even if Googlebot can. Check each one by name: curl -s https://example.com/robots.txt | grep -iA2 -E 'GPTBot|PerplexityBot|ClaudeBot|Google-Extended' Content that only exists after JS runs. If your article body is injected client-side and the server returns an empty shell, a fetch-based crawler sees nothing. Compare raw HTML to rendered: curl -s https://example.com/post | grep -c "a distinctive sentence from your article" Zero means your content isn't in the served HTML. Server-render it or pre-render it. The part you cannot check on-page — and this is where honesty matters — is domain authori

在《南方公园》（South Park）长达二十余年的播映史中，观众们见证了无数荒诞的奇迹。我们曾以为这群四年级的孩子会永远停留在那个没有手机、只有雪地的虚构小镇里。然而，随着特辑《后新冠时代》（Post Covid）的推出，观众们终于看到了主角四人组人到中年的模样——斯坦变成了自私的中年危机男，凯尔成了秃顶的心理咨询师，卡特曼甚至讽刺地皈依了犹太教，而肯尼则成为了伟大的科学家。 “长大”这一曾经被视为不可触碰的禁忌剧情，最终还是真切地呈现在了我们面前。 那么，顺着这个逻辑，下一个尚未开拓、但 注定会到来 的剧情处女地是什么？ 答案只有一个： 现任父母一代（兰迪、杰拉德、史蒂芬、谢拉等）在南方公园度过的童年往事。 这不仅仅是一个粉丝的狂想，而是《南方公园》在叙事结构、商业逻辑以及核心主题演变下， 必定会发生且正在酝酿的终极剧情。 以下我们将从四个维度，深度解析为什么“父母辈童年篇”的出现是历史的必然。 一、 角色重心的位移：兰迪·马什已经成为事实上的“男主角” 要理解为什么父母的童年很重要，首先要看《南方公园》现在的核心是谁。 在早期的节目中，家长的功能非常单一——他们是孩子闯祸后的惩罚者，或者是荒谬社会现象的背景板。但随着创作者特雷·帕克（Trey Parker）和马特·斯通（Matt Stone）步入中年，他们的视角不可避免地发生了偏移。 斯坦的爸爸——兰迪·马什（Randy Marsh），已经从一个功能性配角，逐渐篡位成为了本剧事实上的第一男主角。 从种植大麻的“特种大麻（Tegridy Farms）”主线，到他各种中年危机的狂欢，兰迪的戏份和角色深度甚至超越了孩子们。观众不仅想看斯坦和凯尔，更想看兰迪又作了什么新死。 既然兰迪已经成为了灵魂人物，那么探索他的“前传”就具有了极高的叙事价值。兰迪是如何从一个地质学家变成如今的疯癫模样的？他和杰拉德（凯尔的爸爸）在童年时期有着怎样相爱相杀的关系？史蒂芬（巴特斯的爸爸）那近乎病态的严厉性格，是不是源于他自己童年时遭受的更可怕的“禁足”？ 给头牌角色写前传，是所有长寿美剧在后期挖掘角色深度、延长生命周期的必经之路。 二、 历史的镜像：南方公园是一个“代际宿命”的闭环 《南方公园》最核心的喜剧和讽刺张力，往往来源于 “历史的重复” 。 在《后新冠时代》中，我们看到长大的孩子们不可避免地活成了他们父母的样子：斯坦和兰迪一样酗酒、焦虑、与世界妥协。这种“长大后我就成了你”的幻灭感，是本剧最深刻的黑色幽默。 如果这个闭环要完整，我们就必须看到父母辈的童年。 我们可以预见到这样一幅充满讽刺艺术的画面： 在1980年代的南方公园，小兰迪、小杰拉德、小史蒂芬和小斯图尔特（肯尼的爸爸）也曾组成过一个“四人组”。 他们当时可能也面临着和今天的斯坦、凯尔一模一样的困境——愚蠢的父母、荒诞的镇长、莫名其妙的末日危机。 小兰迪可能曾是一个像斯坦一样理智、对世界充满正义感的孩子，他发誓“我以后绝对不要成为我爸那样无聊的中年人”；而小杰拉德可能比凯尔更具道德洁癖。 这种“屠龙少年终成恶龙”的跨时空对比，将产生无与伦比的戏剧冲击力。 它能将《南方公园》的荒诞主义升华为一种关于“宿命”和“时间”的哲学思考：每一个愚蠢的中年人，都曾是那个试图拯救世界的孩子。 三、 终极的情怀武器：80年代的怀旧经济学 从商业和流行文化的角度来看，主打“80年代怀旧”是当今影视圈的财富密码（想想《怪奇物语》的爆火）。 《南方公园》的创作者特雷和马特出生于60年代末、70年代初，他们的童年恰好度过在70年代末到80年代。对于这两个天才创作者来说， 解构并重塑自己的童年，是他们创作生涯中迟早要交出的一张答卷。 在“父母辈童年篇”中，他们可以肆无忌惮地致敬和讽刺他们自己成长年代的产物： 雅达利游戏机、卡式录音带、初代的MTV。 冷战时期的恐慌、里根时代的保守主义。 经典的80年代怪兽电影和校园青春片范式。 这不仅能吸引那些看着《南方公园》长大的老观众（他们现在也为人父母，有着强烈的怀旧需求），还能为剧集提供源源不断的全新文化素材，避免现代科技（AI、短视频）题材带来的创作疲劳。 四、 尚未填补的巨大剧情坑（Canon） 在现有的剧情碎片中，创作者其实已经有意无意地暗示了父母辈丰富的童年/青年往事，这些“坑”都在等待着被填满： 兰迪与杰拉德的大学基情 ：他们曾提到在大学时期探索过性取向，这段荒唐的青春期是如何过渡的？ 兰迪的男孩天团梦 ：兰迪年轻时曾是男子组合“Ghetto Avenue Boys”的成员，大红大紫后迅速过气，这段经历如何塑造了他渴望关注的性格？ 镇上大人们的恩怨 ：为什么卡特曼的妈妈莱安娜年轻时是全镇的交际花？莫普斯托博士的小白鼠实验在几十年前给小镇带来了什么灾难？ 这些零散的设定就像一颗颗散落的珍珠，急需一根名为“童年

If someone compromised one of your project's dependencies today, would they be able to steal your...

🎲 A Tale of Inference Woes 🎲 Thy context window overflows with dread, Thy API keys scattered 'cross thy thread. Thou switchest providers mid-conversation, And pray thy tokens find the right foundation. No ward stands guard when tools go rogue, No grimoire saves a session from the fog. Thy agents wander, masterless and blind, Thy prompts untested—leaving truth behind. Thy wallet weeps. Thy latency doth creep. Thy model's fine. Thy infrastructure? Not so deep. Sound familiar? I'm excited to share the public README for Arcanum — a .NET 10, single-binary, Native AOT, local-first AI inference hub that treats your infrastructure with the seriousness of a dungeon master and the organization of a well-kept grimoire. Arcanum is one self-contained native executable. No runtime prerequisite. No "install the framework first." Just arcanum serve and you're running a full inference platform on loopback. What's in the bag of holding: 🏰 Local-first & encrypted — SQLCipher-encrypted Grimoire persists every session, entry, and memory. Your data never leaves your machine unless you tell it to. ⚔️ Multi-provider native engine — Any OpenAI-compatible API (DeepSeek, Groq, Ollama via /v1, LM Studio, etc.) plus local GGUF models via a managed llama-server lifecycle. One hub, zero vendor lock-in. 🔮 OpenAI API compatible — POST /v1/chat/completions and GET /v1/models work with existing OpenAI clients out of the box. Drop-in replacement for your local stack. 🛡️ Wards & Sanctum — High-risk tools require operator approval before execution. Per-campaign sandboxes enforce path containment, network policy, and OS-level CPU/memory/FD limits via cgroups v2 and setrlimit. 📜 Spells, not prompts — Versioned markdown workflows with dependency resolution, tool allowlisting, and semantic routing. Dry-run cast previews before spending a single token. 🧙 Autonomous Apprentices — Goal-driven agents with plan generation, retry/backoff, autonomous plan revision, DM escalation, and parallel step execution. 🏰 The

Most SaaS backends were built around a simple assumption: The user pays a subscription, then uses the product. That assumption breaks down for AI apps. An AI app does not just serve screens. It spends money while it works. A user searches the web. A model summarizes a report. An image model generates a draft. An agent calls an MCP tool. A workflow buys data from an API. A future x402 endpoint charges for a capability call. Every one of those actions can have a marginal cost. That means the backend for an AI app is no longer just a place to store users, projects, and settings. Increasingly, it is a system of record for economic activity. In other words: AI app backends are becoming accounting systems. The old SaaS model was simpler Traditional SaaS could survive with coarse billing. You had: monthly subscriptions seats tiers maybe a usage limit somewhere That worked because the marginal cost of most product actions was close enough to zero. If a user clicked a button, edited a document, opened a dashboard, or created a project, the backend cost was usually small compared with the subscription price. The business could average it out. AI apps are different. The product may call paid APIs on almost every useful action. Search once. Summarize once. Generate once. Transcribe once. Call an agent tool once. The unit economics are inside the interaction loop. If the product cannot see who spent what, when, and why, the business is flying blind. Usage billing is not an add-on For AI apps, usage billing is often treated like a pricing feature. I think that is too narrow. Usage billing is really a cost ledger. It answers: which user triggered the cost? which project or app did it belong to? which capability was called? what did it quote before execution? what did it actually cost? was it retried? was it idempotent? did the end user pay for it? is there a payment or checkout record attached? If you cannot answer those questions, you do not have a reliable production backend for

TL;DR 23 commits across 4 repos, one theme: opening apps to the outside world, safely. Public: kickoff v1.32.0 ships SDK-free support-widget integration stubs. Private: external intake channels (token-authed API, cookie-free widget, signed webhooks) on a helpdesk product; signed public API + rebuild webhooks on an event platform. Everything today was about external surfaces — letting the outside in without leaving the door unlocked. What shipped Where What kickoff v1.32.0 (public) SDK-free support-widget integration stubs: settings class + migration, Livewire admin settings page, Blade component, docs, Pest coverage Helpdesk product (private) External intake channels: token-authed API, magic-link requester view, cookie-free embeddable widget, signed outbound webhooks, hardening pass from an adversarial review Event platform (private) Signed public event API + landing-page rebuild webhooks, persona nav overhaul, 15 new MCP tools, offline PWA check-in, plan-limit enforcement Event platform docs (private) Tracker updates + before/after UX screenshots Stubs, not SDKs kickoff now ships a support-widget integration as stubs — settings class, migration, admin page, Blade component — copied into your app. No composer dependency for glue code: you own it, you can read it, you can change it. For ~100 lines of integration code, a stub beats a package. Intake is three problems The helpdesk work was the day's core: letting outside systems and end users create tickets. Every inbound surface splits into the same three problems — who gets in (token auth, magic links), what they can do (rate limits, severity clamps, single-use entry), and what you send back out (signed, idempotent webhooks). An adversarial review caught four real issues before launch; that story gets its own post, next. Static pages, fresh data The event platform got a signed public API plus webhooks that fire on content changes — so landing pages can be static builds that rebuild themselves when an event changes. C

TL;DR Built an embeddable support widget for a helpdesk product: no cookies — a short-lived bearer token in a header, hashed at rest. Entry is an HMAC-signed assertion from the host page. An adversarial review caught four real holes before launch. Outbound webhooks: sign the exact bytes, dedupe key for idempotency, SSRF guard on destination URLs. The requirement: end users file tickets from pages the product doesn't own. That means an embeddable widget — and embeddable means everything you know about sessions stops working. Why cookie-free The widget lives on customers' domains, so any cookie it sets is a third-party cookie — blocked or partitioned by modern browsers. Fighting that means flaky sessions, so: no cookies at all. The entry exchange mints a short-lived session token the widget sends in a header, and the server caches the session keyed by sha256(token) — a cache dump yields nothing replayable. Sessions last 60 minutes, and expiry shows a real recovery path in the UI instead of dying silently. customer backend widget (on customer page) helpdesk API | signs ref|email|name | | | into HMAC assertion ---> |-- redeem assertion (single use) ->| | |<-- session token (60-min TTL) ---| | |-- X-Widget-Token: ... ---------->| What the adversarial review caught Finding Fix Replay burn keyed by client-chosen nonce Burn by HMAC signature — a leaked assertion can't mint extra sessions `\ ` accepted inside signed fields Origin check failed open when Origin/Referer absent Fall back to the unspoofable Sec-Fetch-Dest header to enforce embedding Widget could request critical severity Clamp effective severity (including the channel default) to the widget's allowlist My favourite is the delimiter one. If you sign ref|email|name and accept | inside a field, two different identity tuples can share one valid signature. Canonicalization bugs, not crypto bugs. Webhooks out: sign the exact bytes Outbound webhooks get composed once at enqueue time and stored; the delivery job re-encod

We’ve all been there: it’s 7:00 PM, you’re exhausted after a long sprint, and you open a food delivery app. Your brain screams "Double Cheeseburger," but your body is still recovering from that mid-afternoon sugar spike. What if your phone was smart enough to say, "Hey, your blood sugar is currently 160 mg/dL and rising—maybe skip the extra fries?" In this tutorial, we are building a Chief Health Officer (CHO) Agent . This isn't just a simple chatbot; it’s a sophisticated AI Agent using LangGraph to bridge the gap between real-time medical data (CGM) and real-world actions (Food Delivery APIs). By leveraging automation , function calling , and state machines , we’ll create a system that actively protects your metabolic health. The Architecture: How the CHO Agent Thinks To build a reliable agent, we need a "stateful" workflow. We aren't just sending a prompt to an LLM; we are creating a loop that monitors glucose levels, analyzes food options, and interacts with the browser. graph TD A[Start: Hunger Trigger] --> B{Fetch CGM Data} B -->|Sugar High/Unstable| C[Constraint: Low GI Only] B -->|Sugar Stable| D[Constraint: Balanced Meal] C --> E[Scrape Delivery App Menu] D --> E E --> F[Agent: Analyze Ingredients & GI Index] F --> G[Selenium: Mark/Filter Non-Compliant Items] G --> H[End: Safe Ordering] subgraph "The LangGraph Loop" C D E F end Prerequisites Before we dive into the code, ensure you have the following: LangGraph & LangChain : For the agent's cognitive architecture. Dexcom API Credentials : To fetch real-time Continuous Glucose Monitor (CGM) data. Selenium : For interacting with food delivery web interfaces (Meituan/Ele.me). OpenAI API Key : Specifically for GPT-4o’s reasoning and function-calling capabilities. Step 1: Defining the Agent State In LangGraph, everything revolves around the State . Our CHO agent needs to track the current glucose level, the user's health constraints, and the list of available food items. from typing import TypedDict , List , Anno

With over 5 years in customer support and retention, I've lost count of how many times I've seen the same pattern: a customer explains an issue, gets it "resolved," and then has to explain the same problem again weeks later, as if the first conversation never happened. Support systems forget. Customers don't. That frustration, seen over years on the support side, is what led me to this hackathon project. Most support systems and most AI chatbots treat every interaction as isolated. They don't remember. So patterns that should be obvious (repeated complaints, dropping usage, unresolved issues) never get connected until a customer just leaves. That became the seed for my project: the Retention Risk Agent. The Problem With "Forgetful" AI Most AI tools answer questions in the moment, then forget everything. Ask a chatbot about a customer's history, and it only knows what's in that single message, not what happened last week, last month, or across five different support tickets. For churn prediction, that's a fatal flaw. Churn isn't a single event. It's a pattern, a series of small signals that only make sense when viewed together over time. This is something I understand deeply from years of watching it happen firsthand. Cognee is an open-source memory layer for AI agents. Instead of treating each interaction as isolated, it builds a knowledge graph, connecting facts, relationships, and context across everything you feed it. That's exactly what churn detection needed. What I Built I created a Python script that: Ingests customer records (support tickets, usage patterns, plan changes) Uses Cognee to build a memory graph connecting these signals Asks a simple question: "Which customers show signs of churn risk, and why?" The result wasn't a keyword match; it was reasoning. The agent correctly flagged a customer whose usage dropped 80% and who'd ignored two check-in emails. It flagged another who'd complained twice about slow support and mentioned a competitor. And critica

I built a 36-pattern checklist to catch AI writing tells in my own drafts, calibrated against...

AI Music Tutor that gives real-time feedback on your playing Discussion | Link

In the first article of this series, we saw how a custom build-time compiler can transform a legacy Angular.js template into raw, optimized JavaScript. To recap, starting from this template: <!-- simple.html --> <p> Hello {{ name }}! </p> Our Go compiler generates the following JavaScript module: // simple.js export function template () { const p_0 = document . createElement ( " p " ); const text_1 = document . createTextNode ( "" ); p_0 . append ( text_1 ); return { mount ( container ) { container . append ( p_0 ); }, update ( change ) { if ( " name " in change ) { text_1 . data = " Hello " + change . name + " ! " ; } } } } This is incredibly clean. By running template() , we get an object with mount and update methods. Using mount is fully intuitive: we pass a reference to a DOM element, and it injects our empty paragraph ( p_0 ) into it: import { template } from ' ./simple.js ' ; const { mount , update } = template (); const container = document . getElementById ( ' view-container ' ); mount ( container ); // The DOM now contains: <p></p> (waiting for data) However, the paragraph remains empty until we call update with a change object like this: let changes = { name : " Mario " , }; update ( changes ); // The DOM surgically updates to: <p>Hello Mario!</p> But who is responsible for tracking changes in our application state, building this changes object, and calling update ? The answer lies in marrying the legacy Angular.js $scope with the modern JavaScript Proxy API . The Legacy State Pattern In a traditional Angular.js application, developers mutate the state directly inside a controller by assigning properties to the $scope object: // simple-controller.js export function SimpleController ( $scope ) { $scope . name = " Mario " ; } To bridge the gap between this legacy controller and our new build-time template, we need a way to automatically capture the assignment $scope.name = "Mario" and translate it into a structured update: let changes = { name : " Mario " }

I want to share some thoughts about Reverse Engineering. I love JavaScript and TypeScript, but sometimes I don't want to create something, but the opposite, to understand how others programs work. Working with IDA and x64dbg is the best thing in the world. Btw, at the beginning I used Ghidra, but after I tried IDA, I liked it more. What do you think, what tool is better, IDA or Ghidra? 🤗

Sync your mailbox with your issue tracker Discussion | Link

Back in February, Uber announced ambitious plans to launch in seven new European markets in 2026 — but now five of those launches are reportedly on hold.

Here's the failure mode I keep running into. A team gives a coding agent a repo, a task, and maybe a README. The agent can find files and write code, but it still has to guess the operating rules. It guesses the package manager. It guesses which checks matter. It guesses whether generated files are safe to edit. It guesses what "done" means. A README is usually for humans: what the project is, how to run it, and where the important docs live. A coding agent needs different context. Setup rules. Test commands. Boundaries. Completion criteria. That's the gap AGENTS.md fills. The official AGENTS.md guidance describes it as a predictable place for coding-agent instructions: setup commands, test commands, code style, security considerations, and nested instructions for large monorepos. I find the split useful in a more boring way. The README answers, "What is this project?" AGENTS.md answers, "What should an agent know before touching it?" That second question is where the work usually gets fragile. Where Goose Fits Goose makes this less theoretical because it isn't just a chat box. It's an open source local AI agent with a desktop app, CLI, API, MCP extensions, and skills. Without AGENTS.md , I find myself writing prompts like this: Update the docs, but don't touch generated files, use pnpm, run the lint and test commands, keep the PR small, and tell me what you couldn't verify. With AGENTS.md , the prompt can get shorter: Update the quickstart docs for the new config flag. Goose can run the task in the repo. The repo can carry the standing instructions. I noticed this on a small docs/config update where generated files sat near source files. Without repo instructions, the prompt had to carry the package manager, generated-file boundary, checks, and the "tell me what you could not verify" rule. Once those rules lived in AGENTS.md , the prompt became just the task. Not magic. Just fewer chances to forget the boring parts. Where Skills Fit I would add one more layer once

Introduction Although we interact with LLMs using natural language, these models never processes raw text directly. Before a prompt reaches the model, it is converted into a sequence of tokens , the fundamental units that the model understands. Tokenization is one of the earliest stages of the inference pipeline and influences everything from context windows and API pricing to latency and memory usage. What Is a Token? A token is the smallest unit of text processed by a language model, it is not necessarily a word. Depending on the tokenizer, a token may represent: an entire word part of a word punctuation whitespace numbers symbols emojis Different models use different tokenizers, so the same text may be split differently depending on the model. Why Tokens? Simply because language models operate on numbers, not text. Before the transformer can perform any computation, the input must be converted into a numerical representation. The preprocessing pipeline looks like this: Raw Text │ ▼ Tokenizer │ ▼ Tokens │ ▼ Token IDs │ ▼ Embedding Layer │ ▼ Embedding Vectors │ ▼ Transformer The tokenizer splits the input into tokens and each token is then mapped to a unique integer called a token ID , which are passed through the model's embedding layer, which converts them into dense vectors that become the actual input to the transformer. A Real Example Instead of using hypothetical examples, let's look at how OpenAI's tokenizer processes text. Input: I have no enemies. OpenAI tokenizes it to: ["I", " have", " no", " enemies", "."] with the following token IDs: [40, 679, 860, 33974, 13] that have been generated by OpenAI Tokenizer for the "GPT-5.x & O1/3" models. The transformer never sees the original sentence, it only receives the corresponding sequence of token IDs. Token IDs After tokenization, every token is replaced with an integer. Conceptually: " have" → 679 " no" → 860 " enemies" → 33974 ... The exact numbers differ between models because each tokenizer has its own voca