标签：#Python

GitHub热门项目 | Specification and documentation for Agent Skills | Stars: 21,425 | 47 stars today | 语言: Python

Build a Real-Time Crypto Trading Dashboard with Python, WebSockets, and React Real-time data is the difference between catching a move and reading about it later. This tutorial walks through a minimal but complete stack: a Python WebSocket client pulling live prices, a simple signal generator, and a React frontend displaying everything. You will end up with a dashboard that shows live Binance prices, basic momentum signals, and auto-updates without polling. Why this stack Binance provides a clean WebSocket API for tickers and trades. Python handles the backend connection and lightweight analysis. React keeps the UI reactive and simple to extend. No heavy frameworks, no paid data feeds. Prerequisites Python 3.11+ Node 20+ A Binance API key (read-only is fine for prices) Step 1: Python price stream Install the client library: pip install python-binance pandas Create price_stream.py : import asyncio import json from binance import AsyncClient , BinanceSocketManager import pandas as pd from datetime import datetime async def main (): client = await AsyncClient . create () bm = BinanceSocketManager ( client ) ts = bm . trade_socket ( ' BTCUSDT ' ) async with ts as tscm : while True : res = await tscm . recv () price = float ( res [ ' p ' ]) qty = float ( res [ ' q ' ]) ts = datetime . fromtimestamp ( res [ ' T ' ] / 1000 ) print ( f " { ts } | BTCUSDT { price : . 2 f } | { qty : . 4 f } BTC " ) if __name__ == " __main__ " : asyncio . run ( main ()) Run it: python price_stream.py You should see a live feed of trades. Keep this running as your data source. Step 2: Add a simple momentum signal Extend the script to calculate a 20-trade rolling average and flag when price deviates more than 0.3%: # inside the loop, after parsing res prices . append ( price ) if len ( prices ) > 20 : prices . pop ( 0 ) avg = sum ( prices ) / len ( prices ) deviation = ( price - avg ) / avg * 100 if abs ( deviation ) > 0.3 : print ( f " ⚡ Signal: { deviation : + . 2 f } % from 20-trade avg " )

A pre-install supply-chain gate returns ALLOW or DENY for each package your AI agent proposes, before npm install runs, keyed on provenance: is the name in a vouched snapshot or a popular baseline, and is the .npmrc registry trusted. An SBOM taken after resolve cannot answer that question. In this post's attack manifest, supply_chain_gate.py returns 2 DENY and exits 1. AI disclosure: I wrote supply_chain_gate.py with an AI assistant and ran it myself, offline, before publishing. Every number in the output blocks below is pasted from a real local run on Python 3.13.5, standard library only, no network. I checked the exit codes (0 / 1 / 2), hashed the STDOUT twice to confirm it is byte-for-byte deterministic, and edited every line. The external figures I cite (the USENIX 2025 package-hallucination study) are the researchers' numbers, not mine, and I link the source and say how they measured. I keep their numbers and my run's numbers in separate paragraphs on purpose. In short: An SBOM and a CVE scan run after npm install . They record what resolved and whether it has a known CVE. Neither can say whether your agent should have proposed that name in the first place. A coding agent recommends a dependency with the same flat confidence whether the name is real, hallucinated, or one letter off a real one. That confidence is exactly what a post-resolve scan cannot see through: a name registered yesterday has no CVE yet, so a known-CVE scan lists it as clean. supply_chain_gate.py reads a manifest (the packages the agent proposed, your vouched snapshot, and your .npmrc ) and returns ALLOW or DENY per package against a bundled popular baseline, before install. The result that carries the argument: the same 277-name baseline that ALLOWs express (exact match) DENYs expresss in a sibling manifest. One letter flips the verdict. What flips it is default-deny against a vouched baseline, not a static blocklist of known-bad names; the edit-distance check only labels the DENY ( TYPOSQU

⚠️ This article contains affiliate advertising (promotions). A portion of revenue generated through linked sites is paid to the author, but this does not affect the purchase price for readers in any way. Hey — I'm a working engineer running a side hustle in tech writing and e-commerce. Here's the bottom line upfront: by the time you finish this article, you'll have a Python script that extracts "side-hustle promo noise" from your X (Twitter) timeline and auto-adds it to a mute list , plus a Claude Haiku classifier that labels each tweet as signal or noise for roughly ¥0.02 per tweet — copy-paste ready, just swap in your API keys. My own information-gathering time dropped from 90 minutes to 12 minutes a day (7-day average; details below). Why Manual Muting Breaks Down on X: The 30-Item Wall Muting on X via the GUI is one entry at a time. In my case, roughly 70% of the 480 accounts I follow are genuinely useful — but 30% are promotional, making them "almost good" accounts. Muting at the account level kills the useful tweets too. So I turned to keyword muting, which becomes unmanageable past 30–40 keywords manually. Add "free," "limited time," "LINE sign-up," and "#RT please" to the list and you start catching legitimate tech tweets as collateral damage. One failure story: early on I added "side hustle" as a mute keyword, missed an entire high-quality thread squarely in my interest zone, missed the viral wave, and conservatively lost about ¥3,000 in affiliate opportunity. Word filters don't have the precision. That's the starting point for this article. Extracting "Promo Templates" Mechanically with Tweepy and Filter Rules First, using the X API v2 (read access is available even on the free tier) and Tweepy, I pull tweets equivalent to my home timeline and numerically score structural features common in promotional content. The trick is to score on three axes — emoji density, URL count, and call-to-action verbs — rather than keyword matching. import re import tweepy cl

I used to spend ten hours every week doing content research manually. Checking competitor blogs. Scanning Reddit threads. Copying and pasting search results into a spreadsheet. Trying to spot patterns in an ocean of unstructured text. It was exhausting, slow, and completely unnecessary. Once I learned to automate this with Python and a few affordable APIs, I cut that ten-hour grind down to under thirty minutes. Here is the exact system I built, what it costs, and how you can replicate it yourself. The Quick Answer To automate content research with Python, combine a search API like Serper to pull structured Google search data, BeautifulSoup or requests-html to parse page content, and an LLM API like Gemini to synthesize insights into actionable content briefs. Connect these three components in a sequential Python pipeline and you have a fully automated research agent that runs in minutes instead of hours. What I Actually Built I needed a system that could do three things automatically: First, find what real people are asking about any topic across Reddit, Quora, and Google search. Second, identify what my top competitors have written about that topic and where the gaps are. Third, summarize everything into a clean content brief I can use to write or generate an article. I built this using Python with three core components: the Serper API for search data, BeautifulSoup for page parsing, and the Google Gemini API for synthesis. Total monthly cost: about twelve dollars. I document the full working version of this system — including the Flask web interface and WordPress publishing integration — at https://zerofilterdiary.com Step-by-Step Build Guide Step 1: Install the Required Libraries pip install requests beautifulsoup4 python-dotenv google-generativeai Step 2: Set Up Your API Keys Create a .env file in your project root: SERPER_API_KEY=your_serper_key_here GEMINI_API_KEY=your_gemini_key_here Step 3: Search for Real Discussions Using Serper API import requests import

We’ve all been there. You get a notification from your smartwatch saying your heart rate has been a bit funky, or your blood oxygen is dipping. Usually, we ignore it until it becomes a problem. But what if your personal AI was looking out for you? 🤖 In this tutorial, we are building an Autonomous Health Agent . This isn't just a notification bot; it's a proactive system that uses Playwright browser automation , OpenAI Function Calling , and Python to monitor your health trends and—if things look suspicious for three days straight—literally opens a browser and books a doctor's appointment for you. By leveraging Autonomous AI Agents and Playwright automation , we are moving from "Passive Monitoring" to "Active Intervention." This is the future of Health Tech Automation . 🏗 The Architecture Before we dive into the code, let's look at how the data flows from a "scary heart rate" to a "confirmed appointment." graph TD A[Wearable Data/Health Logs] --> B{3-Day Anomaly Check} B -- Normal --> C[Stay Healthy! 🟢] B -- Abnormal --> D[Trigger AI Agent 🤖] D --> E[OpenAI Function Calling] E --> F[Playwright Browser Automation] F --> G[Hospital Booking Platform] G --> H[Appointment Confirmation 🏥] H --> I[Notify User via SMS/Email] 🛠 Prerequisites To follow along, you’ll need: Python 3.10+ Playwright : The king of modern browser automation. OpenAI API Key : For the "brain" of our agent. A healthy dose of curiosity! 🥑 pip install playwright openai pydantic playwright install chromium 👨‍💻 Step 1: Defining the "Brain" (OpenAI Function Calling) We don't want the LLM to just "talk" about booking an appointment; we want it to actually execute the action. We'll use OpenAI's Function Calling to bridge the gap between text and code. import json from openai import OpenAI client = OpenAI () # Define the tool our agent can use tools = [ { " type " : " function " , " function " : { " name " : " book_doctor_appointment " , " description " : " Books a medical appointment based on department and s

CodeTrace-AI v1.0.1 — Stop Reading Code. Start Understanding It. Every developer has experienced this. You clone a repository, open it, and suddenly you're staring at thousands of files. You spend hours answering questions like: Where is this function called? Which files depend on this module? What happens if I modify this class? Is this code even used anymore? Traditional tools like grep , IDE search, or AI chat assistants can help you find code. They don't help you understand the architecture . That's why I built CodeTrace-AI . What is CodeTrace-AI? CodeTrace-AI is an AI-powered code intelligence tool that transforms your repository into a searchable structural knowledge graph. Instead of treating your project as plain text, it understands your codebase structurally by analyzing: 📂 Folder hierarchy 📄 Files 🏛 Classes ⚙ Functions 📦 Imports 🔗 Function calls 🌐 Cross-file dependencies Think of it as having an AI Software Architect that understands your entire repository. 🚀 What's New in v1.0.1 This release focuses on speed, privacy, and understanding large repositories. 🕸 Interactive Code Graph One of the biggest additions is the interactive repository graph. Instead of reading hundreds of files manually, you can visualize relationships between: Folders Files Classes Functions Imports Function calls Understanding a new project becomes dramatically easier. ⚡ SHA-256 Delta Sync Engine One feature I'm particularly proud of is the new Incremental Indexing Engine. Most code intelligence tools rebuild their entire index every time. CodeTrace-AI doesn't. It computes a SHA-256 fingerprint for every tracked file and detects: ✅ Modified files ➕ Newly added files ❌ Deleted files Only those files are: Re-parsed Re-embedded Re-added to the knowledge graph Everything else is skipped. This makes repeated indexing dramatically faster, especially for large repositories where only a few files change between runs. Under the hood The sync engine includes: SHA-256 fingerprinting Parallel f

A note before we start: this is about the machine, not the money. I'm not going to show you returns, positions, or a single "this strategy made X%." Partly because that's a regulatory minefield, and partly because the returns aren't the interesting part — the engineering is. If you came for a get-rich screenshot, this isn't that. If you came to see how one person ships production infrastructure with an AI, pull up a chair. The thing I built Over the last few months I built, with an AI coding agent as my pair-programmer, a fleet of ~35 automated trading bots. They run across five equity markets plus crypto. Each one is a long-running service. They share a single database, post to a live dashboard, fire alerts to my phone, and — the part that took the longest — they're built to survive restarts, reconcile against reality, and refuse to do anything stupid. I'm one person. I am not a team. The "team" is me plus an AI in a terminal, working the way you'd work with a very fast, very literal junior engineer who never gets tired and occasionally needs to be talked out of a bad idea. Here's how it's put together, and the handful of lessons that cost me the most to learn. The architecture, in one breath One Postgres database is the brain — every trade, signal, and piece of state lives there. Around it sit ~35 containerized bots, each isolated (its own tables, its own config, its own identity), orchestrated with Docker Compose. A Streamlit dashboard reads the database and renders the whole fleet — open positions, P&L curves, health. A notification layer pushes Telegram alerts on every meaningful event. Schema changes go through migrations so a new bot is never born with a stale database shape. Each bot is the same skeleton wearing a different hat: a signal module (the strategy logic), a trader that turns signals into orders, a storage layer that persists everything, a runner loop on a schedule. Strategies are swappable. The infra underneath them is identical. That sameness is

Hey DevHunt community! 👋 I'm incredibly excited to launch Scankii! As developers, we are building more and more AI Agents using frameworks like LangChain, OpenHands, and AutoGen. The standard paradigm is giving these agents "skills" or "tools" — which are basically just Python functions combined with Natural Language instructions (prompts or docstrings). But here is the problem: Standard secret scanners (like GitLeaks or TruffleHog) are blind to AI-specific vulnerabilities. They only scan source code for hardcoded secrets. But what if your Python code securely loads an API key, and your English instructions accidentally trick the agent into printing that key to stdout? The agent framework captures that output, injects it into the LLM context window, and your secret is suddenly exposed. We call this Cross-Modal Leakage. Enter Scankii. 🛡️ Scankii solves this by analyzing the intersection of your Natural Language and your code. It uses a dual-engine pipeline (NL Semantic Analyzer + AST Syntax Analyzer) to track variable flows between your prompts and your code sinks. ✨ Core Features: Dual-Engine Scanning: Correlates English instructions with Python ASTs. Local-First & Fast: Your proprietary agent tools and code never leave your machine. CI/CD Ready: Outputs standard SARIF reports. Drop it into GitHub Actions or use it as a pre-commit hook. Framework Agnostic: Works with LangChain, AutoGen, CrewAI, MCP, or any custom python agent framework. I built Scankii to give developers peace of mind when scaling their agent toolchains. Security shouldn't be an afterthought when building autonomous systems. I would love for you to try it out on your agent repos, star the project, and leave any feedback or questions below! I'll be here all day answering them. 👇 GitHub Repository: https://github.com/ashp15205/scankii Installation: pip install scankii

TL;DR: The idea is to use a set of ordinary JPEG images as a distributed medium for an encrypted container. The data is split into redundant fragments and distributed across the images, allowing the container to be recovered even if some of the JPEG files are lost. We will not examine the code; instead, we will focus only on the general principle behind the approach. Where the idea came from Most steganographic systems follow a "one image — one file" model. If the image is lost or damaged, the embedded data is lost with it. The idea was to distribute the data container across multiple JPEG images while preserving the ability to recover the data even if some of those images are lost. I should point out right away that this is not steganography in the traditional sense. The additional data is written to JPEG images after the EOI ( End Of Image ) marker, where it can be easily detected by even the simplest analysis. The goal is not to completely hide the presence of the data, but to use ordinary JPEG images as a distributed medium for an encrypted container. At the same time, in everyday use the JPEG images remain fully functional, look like ordinary photographs, and do not attract attention. Even if the additional data is detected, it does not reveal whether it is an encrypted container, write artifacts, or simply an arbitrary sequence of bytes. Why JPEG First, JPEG is the most widely used image format. Photographs in this format naturally accumulate on computers and phones and are shared between people. Second, this format has one important property: a JPEG file ends with a special EOI marker. Everything written after it is ignored during image decoding, so the additional data does not affect how the image is displayed. This is the area we will use: JPEG JPEG Additional data EOI data │ │ │ ...░░░░░░░░░░░░░▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒ At first glance, the solution looks extremely simple: just split the encrypted container into fragments and write one fragment to the end of each JP

GitHub热门项目 | The uncompromising Python code formatter | Stars: 41,656 | 22 stars today | 语言: Python

GitHub热门项目 | Turn any technical book PDF into a Claude Code skill — ready to study, reference, and use while you work. | Stars: 7,325 | 205 stars today | 语言: Python

GitHub热门项目 | User-friendly AI Interface (Supports Ollama, OpenAI API, ...) | Stars: 143,680 | 171 stars today | 语言: Python

GitHub热门项目 | Toolkit for linearizing PDFs for LLM datasets/training | Stars: 18,087 | 295 stars today | 语言: Python

Pernah kepikiran, "Sebenarnya AI agent saya inget apa aja sih soal saya?" Kalau iya, tulisan ini buat kamu. Masalahnya: Memori AI Itu Kotak Hitam Kalau kamu pakai AI agent yang punya memori jangka panjang (persistent memory), kamu mungkin pernah ngerasa gak nyaman karena beberapa hal ini: Gak tahu persis apa yang diingat AI tentang kamu Gak tahu file-nya disimpan di mana Gak bisa edit memori itu tanpa ngetik perintah lewat chat Takut kalau file memorinya rusak, semua informasi hilang begitu saja Studi kasus di tulisan ini pakai Hermes Agent , agent open-source besutan Nous Research. Sebagai konteks buat yang belum familiar: Hermes Agent adalah agent AI open-source yang berjalan sebagai proses (daemon) mandiri di server milikmu sendiri, mengumpulkan memori lintas sesi, menjalankan tugas terjadwal, terhubung ke belasan platform pesan, dan menulis skill-nya sendiri dari pengalaman. Framework berlisensi MIT ini dirilis Februari 2026 dan dengan cepat menarik perhatian komunitas open-source AI. Hermes menyimpan memorinya di dua file utama: USER.md (profil tentang kamu) dan MEMORY.md (catatan agent soal lingkungan kerja, kebiasaan, dan pelajaran yang dipetik), plus satu file lagi SOUL.md untuk "kepribadian" si agent. Semuanya disimpan dalam format teks polos yang dipisah pakai karakter § , seperti ini: Preferensimu: komunikasi singkat dan langsung § Namamu Budi, awal 30-an, tinggal di Surabaya § Penggemar PKM / Building a Second Brain Format ini fungsional, tapi ada beberapa kekurangan: Susah diedit langsung karena bukan format yang ramah manusia Gak ada riwayat versi — sekali salah edit, informasi bisa hilang selamanya Gak ada tampilan visual — susah lihat semua catatan sekaligus Gak ada antarmuka grafis — harus lewat chat agent atau edit file mentah Solusinya: pindahkan memori itu ke Obsidian , aplikasi catatan berbasis markdown yang mendukung riwayat versi lewat git dan bisa diedit bebas. Arsitektur Sistemnya Sistem sinkronisasi ini punya empat lapisan: ┌───────────────

**Python Selenium Architecture** Introduction: Selenium automates web browsers. The Python Selenium architecture consists of four main components that work together to control a browser. 1.Selenium Client Library (Python Language Binding) Python developers write automation scripts using the standard Selenium API. This library converts your Python code into a standardized format. It sends these commands as programmatic requests to the browser driver. 2.W3C WebDriver Protocol/JSON Wire Protocol This is the communication channel between the code and the driver. Historically, Selenium used the JSON Wire Protocol over HTTP. Modern Selenium (Version 4+) uses the standardized W3CWebDriver Protocol. Commands and responses are transferred directly without any middle translation. 3.Browser Drivers Every web browser has its own specific executable driver. Examples include ChromeDriver for Chrome and GeckoDriver for Firefox. The driver acts as a secure HTTP server that receives commands. It passes these requests directly to the browser and returns results. 4.Web Browsers This is the final layer where execution happens physically. Supported browsers include Google Chrome, Mozilla Firefox, Microsoft Edge, and Safari. The browser receives commands through its native OS-level API. It executes actions like clicking, typing, or fetching text. Significance of Python Virtual Environments: A Python Virtual Environment is an isolated directory containing its own Python installation and independent packages. It prevents dependency conflicts across different software projects. Conclusion: Why It Matters? Avoids Dependency Hell: Different projects can use different versions of the same library. Protects System Python: Prevents breaking system-wide packages required by your operating system. Ensures Reproducibility: Allows developers to easily recreate the exact environment on other machines. No Admin Privileges Needed: Allows installing packages without sudo or administrator rights. Real-Wo

The problem every AI coding session hits You open Claude or Copilot, paste in your task, and immediately hit the wall: the codebase is too big. You either: Dump everything and burn 80% of your context window on irrelevant files Hand-pick files and miss the one import that breaks everything Pay for a bigger context window and repeat the problem at scale I got tired of this and built ContextOS — a local CLI that acts as an intelligent context layer between your repo and your AI agent. What it does pip install rm-contextos cd your-project contextos scan contextos pack --task "add rate limiting to the auth endpoint" --budget 8000 Output: a Markdown (or JSON) context pack with only the files that matter for that task — ranked by keyword match, import graph centrality, AST symbol overlap, and git churn. Secrets redacted automatically. Token savings report on every pack: Packed 12 files · ~6,840 tokens · saved ~47,200 tokens (87%) vs full repo How ranking works Five signals combine into a score per file: Signal What it catches Keyword match Files whose content/name overlap with your task Import graph centrality Files that everything else imports (critical shared modules) AST symbol overlap Function/class names, not just grep strings Git churn score Recently modified files are probably active code Secret penalty Credential files silently excluded No LLM calls. No cloud. Fully offline. MCP server (for Claude Desktop / Claude Code) pip install "rm-contextos[mcp]" contextos serve --stdio Register in claude_desktop_config.json and your AI agent can call pack_context , scan_repo , list_files , get_file , churn_report directly as tools — no CLI needed. What's shipped 980 tests, 96% coverage Apache-2.0, no telemetry, no accounts Python 3.11–3.13, Linux + macOS Export formats: Claude, Codex, Cursor, Aider, JSON Incremental scan cache — re-scans only changed files pip install rm-contextos pip install "rm-contextos[mcp]" # + MCP server pip install "rm-contextos[all]" # everything Git

从思想到工程：FROST 与 FROST-SOP 的双生之旅 你知道吗？每一个伟大的工程背后，都有一个简单而深刻的起点。 引言：从 500 行到 5000 行的演进 在开源世界里，我们见过太多"大而全"的框架——它们功能丰富，却让人望而生畏。今天我想分享一个反其道而行之的项目： FROST 。 FROST （Fractal Runtime of Orchestrated Skills & Tasks）最初只是一个 500 行的教学框架，用最朴素的代码讲述 Agent 的本质。它的核心哲学只有一句话： 细胞会死，但谱系会存续。Agent 会消亡，但宪法会传承。资产会永存。 而今天我要介绍的，是 FROST 思想"开花结果"后的工程实践—— FROST-SOP ，一个拥有 5000+ 行代码的完整 Agent 工程平台。 FROST：理解 Agent 的钥匙 FROST 不是什么复杂的框架。它的设计哲学是 最小原子集 + 分形宪法 ： 四个原子，理解一切 原子 职责 生物学类比 Store 记忆容器，只做 save/load/delete 细胞核 Skill 纯能力单元，无状态无副作用 蛋白质 Agent 膜包裹的细胞，拥有 Store + Skills 神经细胞 SOP 有序步骤列表，可教学、校验、优化 宪法文本 from core import Store , Agent , skill_set , skill_get store = Store () agent = Agent ( " cell " , store , skills = { " set_context " : skill_set , " get_context " : skill_get }) result = agent . run ( sop_steps = [ " set_context " , " get_context " ], initial_context = { " key " : " message " , " value " : " FROST is alive " } ) # result["_result"] == "FROST is alive" 5 行代码，你就能理解整个 Agent 系统的运作原理。这就是 FROST 的魔力—— 用最少的代码，讲述最深的道理 。 五维元模型（V4.0/V5.0） FROST V4.0 引入的五维元模型，将框架从扁平升级为多维度 Agent 编排系统： 武器注册表 ：能力的元数据管理与发现 任务注册表 ：DAG 任务编排与图谱 SOP 事件编目 ：态势感知与双模式事件分析 平台注册表 ：外部能力的发现、调用与健康检查 规则注册表 ：可版本化的治理约束与合规检查 197 个测试用例保障质量，最新 Release： FROST v5.0.0 FROST-SOP：思想的工程化 FROST 教会我们理解，FROST-SOP 教会我们构建。 事件驱动的 Agent 家族 FROST-SOP 实现了完整的 祖辈 → 父辈 → 子辈 三层递归架构： import asyncio from core.event_bus import get_async_event_bus , Event , EventType from agents.ancestor import create_ancestor from agents.parent import create_parent async def main (): bus = get_async_event_bus () # 创建事件驱动的 Agent 家族 ancestor = create_ancestor ( constitution , asset , event_driven = True ) parent = create_parent ( " parent-1 " , store , event_driven = True , asset_store = asset , sop_id = " DEV-001 " ) # 发布任务，Agent 家族自动响应 await bus . publish ( Event ( event_type = EventType . TASK_CREATED , source = " user " , data = { " task_id " : " task-001 " , " task_description " : " 开发一个用户登录功能 " } )) await asyncio . sleep ( 10 ) asyncio . run ( main ()) 完整的工程特性 特性 说明 19 个内置 Skill

Web Scraping with Python in 2026: Best Libraries and Anti-Bot Strategies Web scraping in 2026 looks very different from 2020. Sites are smarter, anti-bot systems are more aggressive, and the legal landscape has evolved. Here's what actually works now. The 2026 Scraping Landscape Challenge 2020 Solution 2026 Solution Bot detection Rotate User-Agent Fingerprint randomization + residential proxies CAPTCHAs Manual solving Turnstile/hCaptcha solvers JavaScript rendering Selenium Playwright (faster, more reliable) Rate limiting Sleep between requests Adaptive pacing + request signing IP blocking VPN rotation Residential proxy pools Best Libraries in 2026 1. Playwright (Best for JS-heavy sites) from playwright.sync_api import sync_playwright def scrape_with_playwright ( url ): with sync_playwright () as p : browser = p . chromium . launch ( headless = True ) page = browser . new_page () page . goto ( url , wait_until = " networkidle " ) data = page . query_selector_all ( " .job-item " ) results = [] for item in data : title = item . query_selector ( " h2 " ). text_content () results . append ( title ) browser . close () return results 2. httpx + Selectolax (Fast, no JS needed) import httpx from selectolax.parser import HTMLParser def scrape_static ( url ): resp = httpx . get ( url , headers = { " User-Agent " : " Mozilla/5.0 " }) tree = HTMLParser ( resp . text ) for node in tree . css ( " .listing " ): print ( node . text ()) 3. API-First Approach (Always check first!) Many sites have hidden or public APIs that make scraping unnecessary: url = " https://www.freelancer.com/api/projects/0.1/projects/active/?query=python " data = httpx . get ( url ). json () Anti-Bot Strategies That Work 1. Request Fingerprint Randomization import random def get_random_headers (): browsers = [ " Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " , " Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " , ] return { " User-Agent " : random . choice ( browsers ), " A

Breaking long documents into overlapping chunks, preserving context, and reassembling with FastAPI At LectuLibre, we’ve built an AI‑powered platform that translates entire books—EPUBs and PDFs—using large language models. When we first hooked up Claude’s API, we naively fed it a 300‑page PDF in one request. It failed immediately. Claude 3 Opus has a 200K token window, but a 300‑page book can easily run to 300K tokens or more. Even if we squeezed it in, the output would be truncated and the quality would degrade at the extremes of the context window. So we faced a classic long‑document problem: how do you translate a book that’s larger than the model’s context window? Here’s the real approach we ended up with, the code we wrote, and the lessons we learned. The Problem: Token Limits Are Real Claude 3 Opus and Haiku models (and most LLMs) have a maximum context length—200,000 tokens for Opus. A token is roughly ¾ of a word. A 300‑page novel with ~75,000 words translates to about 100K tokens, so it should fit, right? But translations from English to Spanish can expand by 15–20%, and the prompt instructions, system message, and the user message itself all eat into that budget. Plus, we needed to send the entire source text in every call to give the model full context. That’s not feasible. We could have tried a simple split: cut the book at arbitrary page boundaries and translate piecemeal. That fails spectacularly. Narrative breaks mid‑sentence, and phrases like “the previous chapter” lose their referents. We needed a more intelligent chunking strategy. Our Approach: Sliding Window with Overlapping Paragraphs We settled on a sliding window chunking algorithm based on paragraphs, with a generous overlap. Here’s the idea: Split the source text into paragraphs (using \n\n ). Build chunks of max_chunk_tokens (we used 180,000 to keep a safety margin), adding paragraphs one by one and counting tokens with tiktoken . When the chunk exceeds the limit, we start a new chunk but we