标签：#r

We’ve all been there. You click "Export Health Data" on your iPhone, wait ten minutes, and receive a massive, bloated export.xml file. If you've tracked your fitness for years, this file can easily exceed 5GB. Try opening that in Python’s ElementTree or even pandas , and your RAM will cry for mercy. This is a classic Data Engineering challenge: transforming high-volume, semi-structured XML into actionable insights without waiting an eternity. In this tutorial, we are going to build a high-performance parser using Rust performance techniques, Rayon for parallelism, and ClickHouse for lightning-fast OLAP queries. By leveraging Rust's zero-cost abstractions, we'll turn a 20-minute Python slog into a sub-30-second sprint. 🚀 The High-Level Architecture Handling 5GB of XML requires a streaming approach. We cannot load the whole file into memory. We will stream the XML, parse segments in parallel, and ship them to ClickHouse using Protocol Buffers for maximum serialization efficiency. graph TD A[Apple Health export.xml] --> B[Streaming XML Reader] B --> C{Chunking Logic} C -->|Batch 1| D[Rayon Worker 1] C -->|Batch 2| E[Rayon Worker 2] C -->|Batch N| F[Rayon Worker N] D & E & F --> G[Protobuf Serialization] G --> H[(ClickHouse DB)] H --> I[Grafana / SQL Insights] Prerequisites To follow along, you'll need: Rust (Stable) Tech Stack : quick-xml (for streaming), serde (serialization), rayon (data parallelism), and clickhouse-rs . A running ClickHouse instance. 1. Defining the Data Schema Apple Health data (specifically Record types) consists of types, dates, and values. Since we want high performance, we'll use Protocol Buffers to define our intermediate format, ensuring minimal overhead when moving data through the pipeline. // Simplified representation of a Health Record use serde ::{ Deserialize , Serialize }; #[derive(Debug, Serialize, Deserialize, Clone)] pub struct HealthRecord { #[serde(rename = "@type" )] pub record_type : String , #[serde(rename = "@startDate" )] pub

Microservices have a marketing problem: they're associated with the engineering cultures of Netflix and Amazon, so ambitious teams assume adopting them is what serious companies do. But those companies moved to microservices to solve problems of enormous scale and huge headcount — problems you almost certainly don't have yet. For most products, splitting too early is one of the most expensive mistakes you can make. Here's the honest trade-off. What a monolith actually gives you A monolith is one deployable application. That simplicity is a feature, not a limitation, especially early: One codebase, one deploy. No orchestration, no service mesh, no distributed tracing just to understand a request. Simple debugging. A stack trace crosses your whole request. You're not correlating logs across five services to find one bug. Fast local development. Run the whole app on your laptop and iterate. Easy transactions. Data consistency is a database transaction, not a distributed saga you have to design and get right. The modern version isn't a big ball of mud. A modular monolith enforces clean internal boundaries — separate modules with clear interfaces — giving you much of the organization of microservices with none of the network overhead. What microservices actually cost Splitting into services doesn't remove complexity; it moves it from your code into the network, where it's harder to see and reason about. You inherit a long list of new problems: Distributed systems failure modes — partial failures, retries, timeouts, and eventual consistency become your daily reality. Data consistency across services — no more easy transactions; you're designing sagas and compensating actions. Operational overhead — every service needs deployment, monitoring, logging, and on-call. Slower local development and debugging — reproducing a bug can mean running half your architecture. For a small team, this overhead can consume the very velocity you were trying to gain. When microservices genuin

The problem Doc references to a line number or function name break the moment code gets refactored. You write "see line 142 of foo.py " or "see calculateRefund() " in a design doc, then six months later the line has shifted or the function got renamed, and the reference just quietly points at the wrong thing (or nothing). Nobody notices until someone follows it and lands in the wrong place. The fix Anchor Pointing is a tiny text convention that swaps the location for a fixed ID. You drop an ID at the destination you want to be a durable reference target: ap.<21-char-base62-id>.E and reference it from anywhere else, docs, other code, tickets, with: ap.ref.<21-char-base62-id>.E Both are just literal strings, so resolving a reference is a plain text search: grep , ripgrep , GitHub code search, Sourcegraph, your IDE, whatever you already use. The ap.ref. prefix exists specifically so a search for the anchor itself never also matches its own references. Rules One anchor per ID (a single, unambiguous destination) Unlimited references pointing at that ID ID is 21-char base62, making collisions a non-issue with no central coordination Spec: https://github.com/nickolay-kondratyev/anchor-points/blob/main/ANCHOR_POINTS_SPEC.md Feedback welcome, especially if this already exists under another name and I've reinvented something. submitted by /u/ThorgBuilder [link] [留言]

A post on X last week: "The fact that people can't even reply to posts without AI anymore says a...

Abertura Eu sempre participei de comunidades de tecnologia: grupos de estudo do Google,...

The saga of Musk's tussle with the SEC over how he disclosed his growing stake in Twitter (now X) has come to an end.

The $300 million round is expected to be led by Menlo Ventures, Sifted reported.

Meta might be the next company to make an always-on AI wearable. The company is working on prototype "super sensing" always-aware smart glasses that could continuously record audio and snap photos "every few seconds," according to the Financial Times. The wearer could then ask Meta AI about the captured audio and images. However, the images […]

Where human life runs with AI Discussion | Link

The National Highway Traffic Safety Administration said emergency scenes are not "edge cases."

Introduction My zsh profile is over 1000 lines at this point. A lot of that is functions I asked AI to generate for me, since it's fast, portable, and saves me a ton of typing. Here's the thing though: the shortcuts that save me the most time aren't the clever ones. They're the dumb ones. Things like clone instead of git clone && cd , or dir instead of mkdir -p && cd . Each one only saves a second or two, but I run them so often that it adds up fast. These are in no particular order, just the ones I reach for constantly. Git aliases for common commands A few one-liners I have set up as plain aliases: alias gcp = "git cherry-pick" alias git-append = "git commit --amend --no-edit -a" gcp is self-explanatory. git-append amends the last commit with your currently staged (and unstaged, thanks to -a ) changes without touching the commit message. Great for fixing up a commit you just made before you push. Create a branch or switch to it if it already exists One of my most-used functions. Normally you have to remember whether a branch exists before deciding between git checkout <branch> and git checkout -b <branch> . This just does the right thing either way: gb () { if git rev-parse --verify --quiet " $1 " > /dev/null ; then git checkout " $1 " else git checkout -b " $1 " fi } Nuke all local changes to reset the working tree When an experiment goes sideways or I just want to throw everything away and start clean, I run nah : nah () { git reset --hard git clean -df if [ -d ".git/rebase-apply" ] || [ -d ".git/rebase-merge" ] ; then git rebase --abort fi } This resets tracked changes, removes untracked files and directories. No confirmation prompt, so use it carefully. Print recent commits as ready-to-paste cherry-pick commands Useful when you need to cherry-pick a batch of commits from one branch onto another in order: logs () { if [[ -z " $1 " || " $1 " = ~ [ ^0-9] ]] ; then echo "Usage: logs <number_of_commits>" return 1 fi git log -n " $1 " --reverse --pretty = format: "g