今日已更新 35 条资讯 | 累计 23992 条内容
关于我们

标签:#us

找到 1178 篇相关文章

AI 资讯

Crushing 5GB of XML: Building a Blazing Fast Apple Health Parser with Rust and ClickHouse

We’ve all been there. You click "Export Health Data" on your iPhone, wait ten minutes, and receive a massive, bloated export.xml file. If you've tracked your fitness for years, this file can easily exceed 5GB. Try opening that in Python’s ElementTree or even pandas , and your RAM will cry for mercy. This is a classic Data Engineering challenge: transforming high-volume, semi-structured XML into actionable insights without waiting an eternity. In this tutorial, we are going to build a high-performance parser using Rust performance techniques, Rayon for parallelism, and ClickHouse for lightning-fast OLAP queries. By leveraging Rust's zero-cost abstractions, we'll turn a 20-minute Python slog into a sub-30-second sprint. 🚀 The High-Level Architecture Handling 5GB of XML requires a streaming approach. We cannot load the whole file into memory. We will stream the XML, parse segments in parallel, and ship them to ClickHouse using Protocol Buffers for maximum serialization efficiency. graph TD A[Apple Health export.xml] --> B[Streaming XML Reader] B --> C{Chunking Logic} C -->|Batch 1| D[Rayon Worker 1] C -->|Batch 2| E[Rayon Worker 2] C -->|Batch N| F[Rayon Worker N] D & E & F --> G[Protobuf Serialization] G --> H[(ClickHouse DB)] H --> I[Grafana / SQL Insights] Prerequisites To follow along, you'll need: Rust (Stable) Tech Stack : quick-xml (for streaming), serde (serialization), rayon (data parallelism), and clickhouse-rs . A running ClickHouse instance. 1. Defining the Data Schema Apple Health data (specifically Record types) consists of types, dates, and values. Since we want high performance, we'll use Protocol Buffers to define our intermediate format, ensuring minimal overhead when moving data through the pipeline. // Simplified representation of a Health Record use serde ::{ Deserialize , Serialize }; #[derive(Debug, Serialize, Deserialize, Clone)] pub struct HealthRecord { #[serde(rename = "@type" )] pub record_type : String , #[serde(rename = "@startDate" )] pub

2026-07-09 原文 →
AI 资讯

Did you ever face "stale singleton httpx connection" and "cold-start connection problem" problem, Well I did tonight.

It is been while I am learning and build around FastAPI. So there is a project where I was thinking how to add this new feature over exiting one. Like what changes I need to make in database which need to be reflected in my backend and frontend. I already lunched the web locally. Problem started When I when back to the web and reload it it shows this error: ERROR: ConnectTimeout: Unauthorized 401. I was like what? Why? I thougth there is some issue with login endpoint or refresh token function. When i did some debugging and found some new information which is: "Either Supabase's edge/pooler (or OS, or an intermediate proxy/NAT) silently kills those idle connections server-side after some timeout but client-side pool doesn't know that." As I was doing nothing in become idle state so to save the resources server side silently close that particular connection. So I came back and try to connect it give this error. First thought come it my mind after this was there should be a way to automatically check this idle state and if user was in ideal state then create a new connection. Proposed Solutions After a while I come up with these solution: Calculate the Idle time: if it is more then server connection timeout then establish new connection. Retry logic: retry once on the specific connection errors. I thought this will work but This again give me error then this new issue I faced. Cold-start connection problem There is something call dual-stack (IPv4 and IPv6) networks and Happy Eyeballs is a network mechanism which automatically move to IPv4 connection if IPv6 fails. But supabase-py uses httpx and it doesn't support Happy Eyeballs. So in first try after the connection time out it try to establish IPv6 connection which is not routeable in most Pakistani ISPs and ultimately it fails and wait for timeout. There is no way to try it again for IPv4. So we have to do it manually. So this error help me to learn many thing in process. Share your thoughts.

2026-07-09 原文 →
AI 资讯

Epoch Duel: Cyberpunk LLM Alignment Battle

Have you ever wondered how AI engineers fine-tune and align large language models? Under the hood, they run Supervised Fine-Tuning (SFT), optimize parameters using direct preference gradients (DPO), filter out low-quality pre-training corpuses (Pruning), and mitigate catastrophic drifts. To help you visualize how LLM alignment and parameter optimization work in a highly strategic way, I built a cyberpunk card battler inspired by Gwent: 🤖 Epoch Duel: Cyberpunk LLM Alignment Battle Play in Fullscreen Mode (if the embed sizing is tight) 🛠️ Tune Your Model Parameters Your mission as an alignment engineer is to play optimizer cards to outscore the adversarial baseline AI across 3 training Epochs: ⚙️ Logic & Coding: Run SFT code snippets, compile theorem provers, and deploy Python scripts to build your coding benchmark scores. 📖 Language & Speech: Train on multilingual datasets and summarization corpuses to maximize reading comprehension. 🛡️ Safety & Alignment: Implement red-team safeguards, configure RLHF preference pairs, and run DPO tuning to protect your model's outputs. ⚡ regularizers & Drifts: Deploy Regularization cards like Gradient Clipping (Scorch) and Model Pruning to destroy anomalies, or exploit Anomalous Drifts to collapse the AI's rows. 🧬 Playable ML Concepts Explained Here is how the card battle mechanics map to production machine learning pipelines: 1. ✂️ Model Pruning (Weight Compression) In-Game: Playing the Model Pruning card triggers a glitchy dissolution animation that purges the lowest-value card from the targeted board row, cleaning up noise. 💾 The Real-World Counterpart Model Pruning removes unimportant weights (often those closest to zero) from a trained neural network. It shrinks the memory footprint of the model, allowing it to run faster on edge devices. ⚠️ How it affects LLMs By stripping out low-impact weights, pruning compresses models by 30-50% with minimal loss in benchmark accuracy, making deployment significantly cheaper. 2. 🔀 DPO vs RL

2026-07-09 原文 →
AI 资讯

Git tells you what changed. Causari tells you why.

AI coding agents are becoming good enough to touch real codebases. They can refactor files, write tests, change architecture, move logic around, and sometimes modify more code in ten minutes than a human would in an afternoon. That is powerful. But it creates a new debugging problem. Git can tell you what changed . When an AI agent was involved, you often need to know something deeper: Why did this change happen? Which prompt caused this line? Which model produced it? What files did the agent read before writing it? What later changes depended on this agent action? That is the problem I wanted to solve with Causari . Causari is a local CLI for intent-addressable code . It records AI agent actions as causal events: prompts, models, reads, writes, diffs, reasoning, cost, and relationships between actions. The goal is simple: Git tracks bytes. Causari tracks intent and causality. Repository: https://github.com/croviatrust/causari Website: https://causari.dev The problem When a human developer changes code, there is usually some context. A commit message. A pull request. A ticket. A discussion. A design decision. With AI coding agents, the workflow is different. You ask something like this: Refactor the auth flow and add JWT refresh logic. The agent reads files, makes assumptions, writes code, maybe fixes tests, maybe changes something unrelated, then moves on. At the end, you have a diff. But the diff does not tell the full story. A suspicious line appears in auth.ts . Git can show when the line appeared. But Git cannot answer: which prompt produced this exact line? what completion did it come from? did the agent read the right files first? was this part of the original request or an accidental side effect? if I revert this, what downstream work am I also undoing? That gap becomes bigger as agents become more autonomous. The more work agents do, the more we need provenance. Not only code provenance. Intent provenance. The idea: intent-addressable code Causari treats an

2026-07-09 原文 →