今日已更新 160 条资讯 | 累计 24117 条内容
关于我们

I built a CLI that tells you if your codebase fits an LLM's context window

Michael 2026年07月26日 05:19 1 次阅读 来源:Dev.to

Every time I wanted to paste a whole project into Claude or ChatGPT, I ended up guessing whether it would even fit — and often found out the hard way, mid-conversation, that it didn't. So I built Tokenazire, a small CLI tool that solves exactly that. What it does Scans a local folder or a GitHub repo (just pass the URL, it clones it for you) Counts tokens per file using tiktoken (the same tokenizer OpenAI models use, a solid approximation across most LLMs) Shows a color-coded breakdown (green → yellow → orange → red) so you instantly see which files are "heavy" Calculates what percentage of a model's context window (default 200k, configurable) your whole project takes up Ignores .git, venv, node_modules, and other noise automatically Has an --export flag that bundles the entire project — folder structure plus every file's content — into a single text file, ready to paste straight into an LLM chat I kept hitting the same annoying loop: copy a project into a chat, get cut off or told the input's too long, then manually trim files and try again. This automates the "will it fit, and if not, what's taking up the most space" question up front. The --export step came later — once I knew what would fit, I still had to manually copy-paste files one by one into the chat. Now it just spits out one clean file with a project tree on top and clearly separated file contents, ready to paste. Tech stack Plain Python, tiktoken for tokenization, rich for the terminal output (tables, colors, progress bar). No config files, no external services beyond git for cloning. Try it Repo: https://github.com/DeKlain4ik/token-counter (MIT licensed) Still early — feedback, issues, and PRs are welcome.

本文内容来源于互联网,版权归原作者所有
查看原文