标签：#learning

This isn't a wish for the internet to stop — just a moment to imagine what it'd mean to breathe without it. Not everyone, but a huge percentage of the world now relies heavily on the internet. What if it were unavoidably shut down for just 24 hours? How long would those hours actually feel — and how much would they reshape our daily routines? I see the irony everywhere already. The moment a page hangs, I instinctively dial a USSD code to check my data balance. I know someone who pings google.com just to see if he's still connected — using the internet to check whether the internet is still there. The first hour would probably be spent staring at the network icon, refreshing pages, waiting for life to resume. That's when we'd notice how much of the day quietly depends on the cloud: deliveries stall, payments freeze, navigation disappears, businesses pause. Millions would discover just how many invisible gears keep everyday life moving. Then the smaller shifts. Looking at the sky to guess the weather instead of opening an app. Realizing the only people who "exist" are the ones actually in front of you. Sitting in a room where the loudest sound is the silence of the feed. Maybe one day, staying offline will be a skill of its own. Have we gotten so used to consulting the network before taking a step that we've stopped trusting our own judgment? Perhaps 24 hours of silence wouldn't just be an outage. It would be a reminder — that before the cloud, there was memory. Before search engines, there was curiosity. Before notifications, there was presence. And before constant connection, we still knew how to walk on our own. If you asked me, What cloud or internet service would you miss most for a day? For me, I don't remember the last time I went 48 hours without Gemini.

Meta recently open-sourced Brain2Qwerty v2, a noninvasive Brain–Computer Interface (BCI) that can decode sentences from thoughts using electroencephalography (EEG) or magnetoencephalography (MEG) signals from the brain. In evaluations, the system achieved a word accuracy rate 61% on average, compared to 8% for other non-invasive methods. By Anthony Alford

When I first started building APIs with Express.js, every async controller looked the same. I would write a try block, perform some database operations, and then write a catch block that called next(error) . It worked, so I copied the same pattern into every controller. One controller became ten. Ten became fifty. Eventually, I realized that half of my controller code wasn't actually business logic, it was just repetitive error handling. That's when I discovered the Async Handler pattern. The Problem A typical Express controller often looks like this: export const getUser = async ( req , res , next ) => { try { const user = await User . findById ( req . params . id ); if ( ! user ) { throw new Error ( " User not found " ); } res . json ( user ); } catch ( error ) { next ( error ); } }; There's nothing wrong with this code. The problem is that every async controller ends up looking exactly the same. Every file contains: try, catch and next(error) over and over again. Besides being repetitive, it's also easy to forget. Miss one try-catch block, and Express won't automatically catch errors thrown inside async functions. What Is an Async Handler? An async handler is a small wrapper function that automatically catches errors from async controllers. Instead of every controller handling its own errors, the wrapper does it for you. A Simple Analogy Imagine an office where every employee has to stop working whenever someone rings the front door. Besides doing their own job, they also have to greet every visitor. This quickly becomes repetitive and inefficient. Instead, the company hires a receptionist to handle every visitor. Now the employees can focus on their actual work while the receptionist takes care of the door. An async handler works the same way. Controllers focus on handling requests, while the async handler catches errors and passes them to Express's error handler. Without an Async Handler export const createUser = async ( req , res , next ) => { try { const user

I Wish I Ran the Numbers on Open Source AI APIs Sooner Three months ago I would have told you self-hosting was the obvious move. "Open source means free, right?" I said that to a client while quoting them $3,500 for a GPU server setup. They smiled politely and went with someone else. That rejection sent me down a rabbit hole I wish I'd started years earlier, because the actual math — not the vibes-based math freelancers like me tend to do — completely flips the script. If you're running a solo practice or a tiny shop, you probably bill every minute of GPU babysitting straight out of your own pocket. That's time you could be shipping features, pitching clients, or — if we're being honest — sleeping. So let me walk you through what I learned the hard way, with all the pricing left exactly where it belongs. The Open Source Lineup That Actually Matters Right Now When I started this research, I assumed "open source AI API" was an oxymoron. If you're calling an API, somebody owns the server, so what's even the point of being open? Turns out the point is massive: open-weight models accessible through an API give you the pricing transparency of self-hosting without the DevOps funeral you're planning for your weekends. Here's the pricing matrix I put together from Global API's public rates. These are output token prices (input is usually cheaper), and yes — they're shockingly low compared to GPT-4o territory. Model License Output Price Self-Host Range DeepSeek V4 Flash Open weights $0.25/M $500-2,000/mo DeepSeek V3.2 Open weights $0.38/M $800-3,000/mo Qwen3-32B Apache 2.0 $0.28/M $400-1,500/mo Qwen3-8B Apache 2.0 $0.01/M $200-800/mo Qwen3.5-27B Apache 2.0 $0.19/M $300-1,200/mo ByteDance Seed-OSS-36B Open weights $0.20/M $500-2,000/mo GLM-4-32B Open weights $0.56/M $400-1,500/mo GLM-4-9B Open weights $0.01/M $200-800/mo Hunyuan-A13B Open weights $0.57/M $300-1,000/mo Ling-Flash-2.0 Open weights $0.50/M $300-1,000/mo Look at Qwen3-8B and GLM-4-9B at $0.01/M output tokens. A mi

I gotta say, i Spent a Month Testing Chinese AI APIs — Here's What Actually Wins Look, I'm just an indie hacker trying to ship products without going broke. For the past month I've been obsessively running the four biggest Chinese AI model families — DeepSeek, Qwen, Kimi, and GLM — through every test I could think of. And honestly? I wish someone had given me a breakdown like this before I started. So here's my attempt. No corporate fluff, no hand-wavy "it depends" answers. Just real data from someone who actually pays these bills. Why I Even Started Looking at Chinese Models Honestly, I was a GPT-4o loyalist for the longest time. Then I saw my December API bill and nearly choked. $400+ for what amounted to a few chatbot features and some content generation. That's when a friend told me to check out DeepSeek and Qwen. I was skeptical. Like, REALLY skeptical. Chinese models in 2023 were a joke for English tasks. But I kept hearing whispers from other indie hackers about how good things had gotten. So I decided to actually test them properly through Global API's unified endpoint (more on that later). What I found kinda blew my mind. The Quick Cheat Sheet Here's the TL;DR table I wish existed when I started. I'm putting it up top because, lets be real, you probably just want the bottom line: Feature DeepSeek Qwen Kimi GLM Developer DeepSeek (幻方) Alibaba (阿里) Moonshot AI (月之暗面) Zhipu AI (智谱) Price Range $0.25-$2.50/M $0.01-$3.20/M $3.00-$3.50/M $0.01-$1.92/M Best Budget Pick V4 Flash @ $0.25/M Qwen3-8B @ $0.01/M N/A GLM-4-9B @ $0.01/M Best Overall V4 Flash @ $0.25/M Qwen3-32B @ $0.28/M K2.5 @ $3.00/M GLM-5 @ $1.92/M Code Generation ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐ Chinese Language ⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ English Language ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐ Reasoning ⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ Speed ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐⭐ Vision/Multimodal Limited ✅ (VL, Omni) ❌ ✅ (GLM-4.6V) Context Window Up to 128K Up to 128K Up to 128K Up to 128K API Compatibility OpenAI ✅ OpenAI ✅ OpenAI ✅ OpenAI ✅ Alright, now let me act

I kept re-running the exact same input through my translation app. Same code. Same model. Same everything. And the word "machines" kept flipping between two different translations. Sometimes it came out as "機械" (machine). Sometimes as "あなたのPC" (your PC). No code changed between runs. No input changed either. My first assumption was a race condition somewhere in my pipeline. It wasn't. Where I actually looked I checked the obvious suspects first: caching, threading, anything stateful that could make the same input behave differently on different runs. All clean. So I went one level deeper, into how the model picks the winning word. Translation models score every candidate word and pick whichever scores highest. When I logged the actual scores for "machine" vs "your PC" on this input, they were almost exactly tied. That's the part that mattered. When two candidates are separated by a tiny margin, the order floating-point operations get summed in can nudge the score just enough to flip which one wins. Same math, same inputs, different accumulation order between runs — and a near-tie flips sides. Nothing was actually random. It was deterministic all the way down. It just wasn't deterministic in a way I could predict, because the thing that decided the winner was rounding noise several layers below anything I was testing. The fix wasn't "make it deterministic" Forcing strict floating-point determinism across an ML pipeline is its own rabbit hole, and not one I wanted to go down for one word. Instead, I looked at why the tie was so close in the first place. "Machine" and "your PC" were close enough in meaning, in this context, that the model wasn't confident either way. So I widened the margin instead of trying to eliminate the noise: I swapped the input word choice from "machines" to "equipment," which the model was much more decisively confident about. Scores stopped being close enough for rounding noise to matter. The flip-flopping stopped. I want to be honest about a

We all hear about "Not comparing yourself to others" and that "comparing yourself is the thief of joy". To be honest, I agree and it's strange that I am contradicting myself because I compare myself A LOT. The more I looked into it, the more I realized that we have a natural tendency to compare ourselves. It's a human thing to do. The issue is that we tend to be very excessive over comparing ourselves to others to the point where it takes a toll on us. For example, we are demotivated to see someone's success because we believe we can't reach the goal they are in. We all have jealousy. Big or small. Even where I am at right now, I am still jealous that many people I know that got into big tech companies like Microsoft. To get more context, I want to share a story with you. Story Time Back in the day, I remember it was the year of the ACT. For those who don't know: It's a Standardized test that is needed for the college admissions to determine if you are admitted to their program. I remember I got a national average of 21 as my composite score and I was proud of the score I got since it's the national average during that time. However, I remember the day where my friends talked about the ACT. The most common thing I heard was: "Oh I got a 30" "I got a 32" "Man I got a 35, it was sooo easy" Hearing that makes me feel not only bummed out, but felt left out. I was feeling that I wasn't smart enough to be in the group. What's worse is that they got accepted into colleges and programs that are well known. Then they start boasting about their accomplishments. I felt like I am the odd-one-out because of my scores and their accomplishments I could not match. Why am I Talking about this? Looking back and knowing where they are at now, I am proud of who I become today. It's not that they have fallen downhill (they are still successful), but the route they have taken that I definitely could not follow. For example, on GitHub, many people fill up their contribution graphs to the

Why build something? And what if nobody ends up using it? There are good answers to the first one. You build because you need a thing that doesn't exist yet. You build to see if you can, the technical challenge, the "is this even possible?" You build to impress someone, or just because you think it'll make people's day a little less annoying. All of those are real reasons, and at different points, I told myself most of them. Then, a few days ago, late in the day, at the end of a coding session, five months into the project, I asked myself those two questions back-to-back. And for the first time, I couldn't answer the second one. Zeri worked. Every feature did what it was supposed to do. Both processes handshake cleanly, a variable set in one context showing up in another a second later, the TUI rendering exactly as I'd pictured it. And I sat there and couldn't come up with one honest sentence explaining why anyone would actually download it. That gap, between something built well and something that has a reason to exist, turned out to be the most useful thing this whole project taught me. So I'm shipping it anyway, and I'll tell you why. What I built Zeri is a TUI multi-language REPL. You launch it, pick a language, Python , JavaScript (with Bun ), Ruby , or LuaJIT , and you get an interactive session in your terminal. You can switch languages mid-session, share variables across them, save and reload your work, manage snippets, and talk to a local LLM through a command running on Ollama . The feature list isn't the interesting part, though. The interesting part is what's underneath. Two processes, one app Zeri is split into two processes: a headless engine written in C++23 and a TUI frontend built in Go using Bubble Tea and Lip Gloss . The engine does all the evaluation, state, and runtime coordination. The frontend does rendering, input, and everything the user actually sees and touches. They talk to each other over a custom binary IPC protocol that I built from sc

The brain layer was scoring high because the test was leaking. The actual capability was being silently rejected by a misconfigured gate. Both findings landed in the same week. Part 18 ended on a clean diagnosis. The brain layer reasoned correctly when the encoder fed it correct inputs. The encoder didn't always feed it correct inputs. So the path forward was upstream: more physics-shaped training data for the encoder, retrain, re-validate. I wrote the drops, kicked off the retrain, and watched the held-out eval climb. It hit twenty-three out of twenty-six. Eighty-eight percent. The number I'd been chasing. I sat with that for an evening. Twenty-three of twenty-six on compositional reasoning probes the model had never seen during training. The Phase 8 cutover gate from Stage D had been sixty percent. I was thirty points past it. The brain layer had not only survived its missing-from-production months, it had come back stronger. The number was wrong. I figured this out the next morning while writing what was going to be the celebration commit. Something nagged about the eval set. The training data generator built the eval pairs independently from the training pairs, drawn from a different source list. That should have given me a clean train/test split. But I noticed the eval generator was running before the training generator wrote its file, and neither side knew about the other. I dropped into a Python shell and intersected the two pair sets by their input-output keys. Twenty-three of twenty-six held-out probes were also present in training data. Eighty-eight percent of my held-out eval wasn't held out. The model wasn't generalizing. It was memorizing the answers it had already been shown, then being graded on whether it remembered them. The three pairs that were genuinely unseen, I checked those separately. The model got one right. Three out of twelve when I went back through other historical evals and ran the same overlap check. About a quarter, with no statistica

Builder Journal · ARC Prize 2026 There is a specific horror in a detective story when you realize the witness everyone trusted has been lying, or just wrong, the whole time, and every conclusion built on their testimony has to come down with them. I had that moment with my own notes this month. The unreliable witness was me. Context, if you are new to this thread : I'm competing in the ARC Prize 2026, building an agent that has to win games it has never seen. It had been stuck, underperforming on the hidden test in a way I could see on the scoreboard but could not explain, and I had been hunting the cause across several sessions. The two comforting facts In two earlier work sessions I had written down, as settled conclusions, two things about why the agent was failing. One: the failure was a kind that only happens on the hidden online games, so it could not be taken apart and studied on my own machine. Two: the practice games I did have were useless for investigating it anyway, because they scored a flat zero on the relevant measure. Notice what those two beliefs do when you put them together. They say, in a calm and reasonable voice, that there is nothing to be done here. The problem is unreachable, the practice data is a dead end, the smart move is to spend your energy elsewhere. They were not just facts. They were permission to stop looking. So I stopped looking. Twice. The hour that knocked it all down Eventually I made myself do the one thing I had been quietly avoiding. Instead of rereading my own notes for the third time, I went and checked. I wrote small probes and ran them against the real artifacts, the actual code and the actual game data, rather than against my memory of what they did. Both beliefs collapsed inside an hour. The failure was not unreachable. It came apart cleanly, deterministically, on the games I already had sitting on my disk. And the "dead end" practice data was not a dead end at all. It showed the problem plainly the moment I asked it

Problem Statement For roughly a decade, vision-language models have been declared to be approaching or matching human performance on scene description (captioning). The evidence for that claim has almost always come from the same family of benchmarks—most famously MS-COCO. Those images are typically clean, well-lit, and depict either no people or people performing simple, isolated actions (sitting, walking, holding an object). They rarely require the model to parse multi-agent social dynamics, subtle intentions, or the kind of relational reasoning humans perform effortlessly when watching a movie scene or a street interaction. Because the evaluation data are easy, the reported numbers look excellent. Automatic metrics such as BLEU-4, CIDEr, or even embedding-based scores like BERTScore further inflate the impression of progress: they reward surface lexical overlap more than genuine semantic fidelity. At the same time, almost no work has systematically catalogued which visual-cognitive failures models still commit, or how those failure modes have changed as architectures moved from CNN+LSTM captioners to today’s multimodal large language models (MLLMs). The result is a field that can claim “human-level performance” while remaining largely blind to whether the models actually understand the scenes that matter most in real applications—scenes full of people interacting. The authors therefore set out to answer two concrete questions that the existing literature left open: (1) How much of the apparent progress is an artifact of easy data? (2) Which specific error types have been eliminated and which stubbornly remain? Core Idea The core insight is that progress looks dramatically different once you force models to describe complex social behavior and once you measure not only overall accuracy but a taxonomy of visual-cognitive errors. By constructing a new 100-image Complex Social Behavior (CSB) dataset drawn from movie frames that require reasoning about multi-person in

I recently found an old project I built with a friend around 2017–2018: a perk calculator for the game Firefall. The application allowed players to browse perks by category, drag them into a build, track the available perk points and automatically filter incompatible options based on the selected class. Looking at the code today, there are many things I would structure differently. The JavaScript could be better organised, responsibilities could be clearer, and the overall architecture would benefit from more modern practices. Still, I decided to preserve it as it is. Older projects are useful reminders that progress is not only visible in the technologies we use, but also in how we model problems, organise code and make technical decisions. It is not a showcase of how I would build the same application today. It is a snapshot of how I approached a real problem at that point in my career. Repository: https://github.com/lksvn/firefall-perk-calculator

skip에서 partition overwrite로: business_date 재처리를 Iceberg로 다시 표현하기 이전 글에서는 같은 source_hash 가 다시 들어왔을 때 기존 successful run을 재사용하는 idempotency를 다뤘다. 하지만 재처리에는 두 종류가 있다. 1. 같은 입력이 다시 들어온 경우 -> skip이 맞다. 2. 같은 business_date의 정정 입력이 들어온 경우 -> skip하면 안 된다. -> 같은 날짜의 gold 결과를 중복 없이 교체해야 한다. manufacturing-data-platform-mini 의 B5 slice는 두 번째 문제를 아주 작게 다룬다. 전체 Spark pipeline을 만든 것이 아니다. gold_daily_metrics Iceberg table 하나를 local Spark에서 만들고, business_date partition overwrite와 snapshot evidence만 검증했다. Scenario 이미 아래 gold row가 있다. business_date=2026-06-29 plant-a / line-1 / gearbox-a units_produced=120 defect_count=3 나중에 같은 business_date=2026-06-29 에 대한 정정 source가 들어온다. 운영자가 원하는 것은 append가 아니다. 원하지 않는 상태: 2026-06-29 old row 2026-06-29 corrected row -> 같은 날짜 결과가 중복됨 원하는 상태: 2026-06-29 corrected row만 남음 2026-06-30 같은 다른 날짜 partition은 그대로 유지됨 재처리 전후 snapshot evidence가 남음 그래서 이 slice의 질문은 이렇다. 같은 business_date의 정정 source를 처리할 때, gold table에서 해당 날짜 partition만 중복 없이 교체하고, 어떤 run이 어떤 Iceberg snapshot을 만들었는지 남길 수 있는가? Decision Pressure Slice1의 CSV pipeline은 already-successful source를 안전하게 skip할 수 있다. dataset_id + business_date + source_hash 이 key가 같으면 같은 입력이다. 다시 계산해도 같은 결과이므로 기존 run을 재사용한다. 하지만 source_hash 가 달라졌다면 의미가 다르다. same business_date different source_hash 이건 retry가 아니라 correction이다. CSV run-folder 방식에서는 새 run output을 만들 수는 있지만, "현재 gold table에서 해당 날짜를 원자적으로 교체한다"는 table-level 의미가 약하다. Iceberg를 붙이는 이유는 여기 있다. source_hash -> 같은 입력인지 판단하는 idempotency key business_date partition -> 정정 시 교체할 gold table 범위 snapshot_id -> table commit의 evidence 즉 Spark/Iceberg는 도구 이름을 추가하려고 붙인 것이 아니라, 재처리 상태 전이를 더 명확히 표현하기 위해 붙였다. Options Option 장점 문제 판단 same source면 항상 재계산 단순함 retry 때 불필요한 commit이 계속 생김 제외 corrected source를 append 구현 쉬움 같은 날짜 gold row가 중복될 수 있음 제외 whole-table overwrite 단순함 다른 날짜 partition까지 지울 위험 제외 business_date partition overwrite correction 범위가 명확함 Spark/Iceberg 설정과 test가 필요 선택 MERGE/upsert 강력함 이번 skeleton에 과함 backlog 이번 구현은 DataFrameWriterV2.overwritePartitions() 를 사용했다. corrected_d

wide CSV 여러 개를 EAV로 모아 gold mart 만들기 현실의 데이터 소스는 한 가지 모양으로 오지 않는다. 같은 의미의 값도 어떤 파일에서는 생산수량 , 다른 파일에서는 units , 또 다른 파일에서는 made 로 올 수 있다. 온도도 어떤 곳은 섭씨, 어떤 곳은 화씨일 수 있다. 이걸 매번 pipeline code에 if source == ... 로 박기 시작하면 source가 늘 때마다 코드가 지저분해진다. manufacturing-data-platform-mini 의 EAV mini slice는 이 문제를 작게 다룬다. 여러 wide CSV를 mapping config로 표준 attribute에 맞춘 뒤, EAV long format으로 모으고, 다시 gold metric mart로 pivot/aggregate한다. 데이터는 모두 synthetic이고, 회사 코드·고객 데이터·실제 schema는 쓰지 않았다. 1. Scenario 서로 다른 공장/라인/벤더에서 비슷한 제조 지표 파일이 들어온다. 예: plant_a.csv: 설비ID, 생산수량, 불량수, 온도C, 압력kPa plant_b.csv: machine_id, output_units, defects, temp_f, pressure_bar vendor_d.csv: unit_name, made, scrap, deg_c, kpa 비즈니스적으로는 같은 지표를 보고 싶다. units_produced defect_count temperature_c pressure_kpa 문제는 source마다 column name과 unit이 다르다는 점이다. 2. Decision Pressure 단순 구현은 source마다 코드를 늘린다. if source == "plant_a": 생산수량을 units_produced로 읽는다 if source == "plant_b": output_units를 units_produced로 읽는다 temp_f를 섭씨로 변환한다 if source == "vendor_d": made를 units_produced로 읽는다 이 방식은 작게는 빨라 보이지만 source가 늘수록 문제가 된다. 새 파일 형식마다 pipeline code를 고쳐야 한다. column mapping과 transform logic이 섞인다. unit conversion이 흩어진다. quality check가 source별로 갈라진다. gold mart grain을 설명하기 어려워진다. 그래서 mapping은 config로 빼고, pipeline은 표준 attribute를 처리하게 만들었다. 3. Options option result risk source별 hard-coded parser 처음엔 빠름 source가 늘 때 code change 반복 모든 source를 wide table 하나로 합치기 보기 쉬움 sparse/heterogeneous column 폭발 EAV long format 이종 attribute를 표준 형태로 모음 pivot/quality 설계가 필요 full mapping DSL/rules engine 유연함 mini project에는 과함 이 프로젝트의 선택은 단순한 JSON mapping + EAV long + gold pivot이다. 4. Decision 각 source는 JSON config로 자신의 column을 표준 attribute에 매핑한다. source column -> standard attribute output_units -> units_produced temp_f -> temperature_c with f_to_c pressure_bar -> pressure_kpa with bar_to_kpa pipeline 흐름: wide CSVs -> mapping configs -> EAV long rows -> gold entity_daily_metrics -> quality checks -> catalog/lineage EAV row의 핵심 shape: entity_id business_date attribute value v

schema drift를 fail이 아니라 warn으로 둔 이유 데이터 파이프라인에서 source schema가 바뀌는 순간은 애매하다. 무조건 무시하면 운영자는 입력 구조가 바뀐 사실을 모른다. 반대로 모든 schema 변화를 실패로 처리하면, 정상적인 컬럼 추가까지 daily run을 막아버린다. manufacturing-data-platform-mini 에서는 이 문제를 작게 다뤘다. synthetic manufacturing CSV의 실제 header를 기준으로 schema_hash 를 만들고, previous successful run과 비교해 달라졌으면 schema_drift quality check를 warn 으로 남긴다. 단, required column이 빠져 silver/gold contract를 만들 수 없는 경우는 현재 ValueError 로 빠르게 실패한다. 1. Scenario 어느 날 source CSV에 새 컬럼이 추가된다. 기존 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date 새 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date,operator_id operator_id 는 아직 silver/gold mart에서 쓰지 않는다. 하지만 source 구조가 바뀐 사실은 기록되어야 한다. 2. Decision Pressure schema drift에서 중요한 질문은 단순히 "바뀌었나?"가 아니다. 바뀐 것을 운영자가 알 수 있는가? 정상적인 컬럼 추가 때문에 pipeline을 멈춰야 하는가? downstream gold mart contract가 조용히 바뀌지는 않는가? 이전 successful run과 지금 run의 schema identity를 비교할 수 있는가? 초기 구현에서는 한 가지 실제 버그가 있었다. schema_hash 가 고정된 required column 목록에 너무 묶여 있어서, 추가 컬럼이 들어와도 hash가 바뀌지 않았다. 즉 operator_id 가 추가되어도 drift가 보이지 않았다. 이 문제를 고치기 위해 read_rows 가 실제 CSV header를 반환하고, 그 실제 header 기준으로 schema_hash 를 계산하도록 바꿨다. 3. Options option result risk ignore drift pipeline은 계속 돈다 source 변화가 보이지 않음 fail every drift 변화에 강하게 반응 정상적인 컬럼 추가도 막음 warn and continue 변화가 보이고 run도 계속됨 warning을 inspect해야 함 auto-evolve silver/gold 새 컬럼을 바로 사용 가능 downstream contract가 조용히 바뀔 수 있음 full schema registry production에 가까움 mini slice에는 무거움 이 프로젝트의 선택은 warn and continue 다. 4. Decision 현재 contract는 이렇다. previous successful run이 없으면: schema_drift = pass baseline schema established current schema_hash == previous successful schema_hash: schema_drift = pass current schema_hash != previous successful schema_hash: schema_drift = warn quality_passed는 true 유지 run/lineage record에 previous/current schema_hash 저장 required column missing: V

source_hash로 같은 입력 재처리를 안전하게 skip하기 작은 데이터 파이프라인도 한 번만 실행된다고 가정하면 금방 거짓말이 된다. 실제로는 같은 파일을 다시 실행할 수 있다. 실패한 run을 재시도할 수도 있고, 과거 날짜를 backfill할 수도 있고, 운영자가 실수로 같은 입력을 다시 넣을 수도 있다. 이때 결과가 중복되면 gold metric은 더 이상 믿을 수 없다. 이 글은 개인 포트폴리오 프로젝트 manufacturing-data-platform-mini 에서 source_hash 를 이용해 같은 입력 재처리를 안전하게 skip하도록 만든 작은 설계 판단을 정리한 글이다. 데이터는 모두 synthetic이며, production platform이 아니라 검증 가능한 mini slice다. 1. Scenario 같은 business_date 의 제조/로봇 이벤트 파일을 다시 처리해야 하는 상황이 있다. 예: retry: 앞 run이 중간에 실패해서 다시 실행한다. backfill: 과거 날짜를 다시 채운다. operator mistake: 같은 파일을 실수로 다시 실행한다. 단순히 매번 append하면 같은 날짜의 gold metric이 중복될 수 있다. 2. Decision Pressure 단순 CSV pipeline은 보통 이렇게 끝난다. CSV 읽기 -> silver 만들기 -> gold 집계 -> 결과 저장 하지만 운영 관점에서는 질문이 생긴다. 이 입력은 전에 처리한 파일과 같은가? 같은 파일을 다시 돌리면 중복 output이 생기나? 다른 파일로 같은 날짜를 다시 돌리면 어떻게 구분하나? 어떤 run이 어떤 source에서 만들어졌나? 그래서 재실행을 판단할 identity가 필요했다. 3. Options option result problem always append 모든 run 결과를 계속 추가 같은 입력 재실행 시 중복 always overwrite 결과를 항상 덮어씀 이전 결과/원인 추적이 약함 skip by business_date only 같은 날짜면 무조건 skip 정정 파일을 반영할 수 없음 skip by dataset_id + business_date + source_hash 같은 입력만 no-op 정정 파일은 새 run으로 처리 가능 이 프로젝트의 Slice1은 마지막 선택지를 쓴다. 4. Decision 현재 mini pipeline은 입력 파일의 content hash를 source_hash 로 계산한다. idempotency key: dataset_id + business_date + source_hash 이미 성공한 run이 있으면 새로 처리하지 않고 기존 run을 재사용한다. same dataset_id same business_date same source_hash prior successful run exists => status = skipped 이 선택은 작지만 중요하다. 같은 파일 재실행: skip -> 중복 없음 같은 날짜의 정정 파일: source_hash가 다름 -> skip하지 않고 새 run으로 처리 가능 단, 여기서 조심해야 할 경계가 있다. Slice1은 다른 source_hash 를 새 run으로 처리할 수 있지만, 이전 gold partition을 원자적으로 교체하는 Iceberg-style overwrite까지 구현한 것은 아니다. 그 문제는 다음 Slice2의 business_date partition overwrite 주제다. 5. Evidence 관련 코드와 검증 evidence: src/manufacturing_data_platform/pipeline/lakehouse.py tests/test_lakehouse_pipeline.py VERIFICATION_LOG.md README.md 검증 로그: 2026-07-08 publication readiness check: pytest: 33 passed lakehouse JSON CLI: passed, status=processed, quality_passed=true EAV JS

Stop Guessing: How I Pick AI API Architecture at Every Scale I've been on both sides of this. Two years ago I was the lone backend engineer at a Series A startup, duct-taping API calls together at 2 AM because the founders wanted a chatbot demo by morning. Last quarter I sat in a procurement meeting at a Fortune 500 where we spent six weeks evaluating three vendors for a single inference workload. Same job title on LinkedIn, wildly different problems. Most AI API guides I've read treat both scenarios like they're the same conversation. They're not. The startup CTO optimizing for burn rate and the enterprise architect worrying about a 99.9% uptime SLA are solving fundamentally different equations. After enough of these conversations, I've developed a framework I'd like to share — and yes, I'll talk about Global API because it's what I actually use, but I'll also explain the reasoning behind each choice so you can adapt it to your own stack. What I Look at First: The p99 Question Before I look at price, I look at the latency distribution. Specifically, the p99. Mean latency tells you almost nothing useful. If your median response is 200ms but your p99 is 4 seconds, your users will see janky behavior on the long tail and you won't know why until production is on fire. For startups in the MVP phase, you can usually get away with best-effort routing. A p99 of 2-3 seconds is fine if you're building an async summarization feature. But the moment you put AI in the synchronous request path — like a customer-facing chatbot or a real-time code suggestion — p99 starts to bite. I learned this the hard way when our startup's "AI assistant" feature had users complaining about slowness that I couldn't reproduce locally. The culprit? Provider cold starts hitting our 1% of users who happened to get routed to a freshly spun-up instance. For enterprises, p99 isn't a nice-to-have, it's a contractual obligation. Most B2B SLAs I've negotiated pin uptime at 99.9% and require reporting on m

Detecta si tu modelo de materiales hace trampa con la "huella bibliográfica" Un modelo de ML puede predecir la propiedad de un material sin entender la química: basta con que "aprenda" qué autores, revistas o años suelen ir con cada resultado. Esta herramienta aplica el test de falsificación de Clever Materials para descubrirlo. El problema: cuando el modelo lee el membrete, no la ciencia Imagina que entrenas un modelo para predecir si un material es estable. El modelo no mira la química: descubre que los artículos del grupo X (publicados en la revista Y, en torno al año Z) casi siempre reportan "estable". Así que aprende a clasificar por el membrete bibliográfico , no por la estructura. Funciona en el papel y se rompe en la práctica. A esto se le llama confounding bibliográfico (o leakage por metadata). No es un error de código: es una señal espuria que el modelo aprovecha. El paper Clever Materials (Jablonka et al., 2026) mostró que este patrón está generalizado en cinco tareas reales de materials science. Qué hace la herramienta materials-confounding-check es una CLI ( mcc check ) que corre cuatro sub-tests de falsificación sobre tu dataset (descriptores químicos + metadata bibliográfica + propiedad objetivo): Clasificador de metadata — ¿se puede predecir la bibliografía (autor/revista/año) a partir de los descriptores químicos? Si es above-chance , hay una señal bibliográfica presente. Huella bibliográfica — ¿un modelo que usa solo la metadata predicha se acerca al modelo con descriptores? Entonces el dataset no descarta hacer "trampa" por bibliografía. Split por grupo/tiempo — ¿colapsa el rendimiento si separas por autor/año en vez de al azar? Veredicto — un score low / medium / high de riesgo de confounding. El rigor que exige el test (para especialistas) El punto delicado de cualquier "test de significancia" es fijar el umbral a mano. Si ajustas el margen hasta que tu fixture pase, el test no prueba nada: es el anti-patrón Clever-Hans que el propio proyecto d

You have seen the posts. AI is coming for the junior engineer first. Why hire someone to write code a model can write for free? The career ladder's bottom rung is gone, so start saving your pity for anyone about to graduate into this market. I think the premise is wrong, and it is wrong in a specific, fixable way. Look closely at what these predictions actually describe. Not a junior engineer. A person whose entire job is turning a finished spec into working code. That role is real, and it is shrinking fast, but it was never the same thing as "junior engineer." We just let the two collapse into one job title for forty years because, until recently, spec-to-code translation was the canonical, critical thing a junior had the skill to do. The task and the title are not the same thing. AI is eating the task. It does not follow that it eats the title too, unless we insist on keeping them welded together. So the real question is not "does the junior engineer survive." It is "what do we train a junior engineer to do now that the translation work is cheap." And the honest answer is: not much of what we have been doing. I think we landed on "junior engineers are doomed" for a reason that has nothing to do with whether it is true. It is the easy conclusion. It requires nothing from us. Training a junior into a senior was never straightforward, even in the old world, and figuring out how to do it without the years of tickets we used to lean on is genuinely hard. "They're doomed" lets everyone off the hook. "How do we train juniors into seniors now" does not, but it is the question with a future in it. The first one just has a shrug. The apprenticeship we built no longer exists For as long as I have been in this field, the plan was the same. Hire someone who can code. Hand them small, well-specified tickets. Let them grind through years of execution: bugs, edge cases, code review, the slow accumulation of pattern recognition that eventually turns into judgment. Somewhere around

Image-to-video generation is often described as a simple interaction: upload image -> describe motion -> get video That description hides the real problem. A single still contains only one view of a subject. When we ask a model for a fast camera orbit, a full-body walk, or expressive gestures, we are asking it to invent information that was never present in the source. That is where identity drift, unstable lighting, texture flicker, and waxy faces come from. The useful way to approach Seedance 2.0 image-to-video is not as a prompt-writing contest. It is a constraint-management workflow. Give the model a strong identity anchor, request motion that the source image can support, and evaluate one variable at a time. This post explains that workflow in a way that is useful whether you are animating a product render, a character portrait, an approved client still, or a visual asset for a prototype. Note: Model capabilities, pricing, model availability, and input limits change quickly. Check the current documentation and the terms of the platform you use before committing a production workflow. Why image-to-video is different from text-to-video Text-to-video is excellent when invention is the point. You describe a scene and let the model make creative decisions about characters, lighting, composition, and motion. Image-to-video is the better tool when those decisions have already been made and must remain stable. Situation Better starting mode Why Product hero shot Image-to-video Label, shape, material, and color must remain recognizable Character-led sequence Image-to-video One strong reference can anchor a character across clips Approved campaign still Image-to-video The source already represents the accepted art direction Atmospheric B-roll Text-to-video Exact subject identity matters less than visual exploration Abstract concept film Text-to-video Inventing a scene is more valuable than preserving one Existing brand-photo library Image-to-video Stills become reusable