标签：#p

schema drift를 fail이 아니라 warn으로 둔 이유 데이터 파이프라인에서 source schema가 바뀌는 순간은 애매하다. 무조건 무시하면 운영자는 입력 구조가 바뀐 사실을 모른다. 반대로 모든 schema 변화를 실패로 처리하면, 정상적인 컬럼 추가까지 daily run을 막아버린다. manufacturing-data-platform-mini 에서는 이 문제를 작게 다뤘다. synthetic manufacturing CSV의 실제 header를 기준으로 schema_hash 를 만들고, previous successful run과 비교해 달라졌으면 schema_drift quality check를 warn 으로 남긴다. 단, required column이 빠져 silver/gold contract를 만들 수 없는 경우는 현재 ValueError 로 빠르게 실패한다. 1. Scenario 어느 날 source CSV에 새 컬럼이 추가된다. 기존 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date 새 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date,operator_id operator_id 는 아직 silver/gold mart에서 쓰지 않는다. 하지만 source 구조가 바뀐 사실은 기록되어야 한다. 2. Decision Pressure schema drift에서 중요한 질문은 단순히 "바뀌었나?"가 아니다. 바뀐 것을 운영자가 알 수 있는가? 정상적인 컬럼 추가 때문에 pipeline을 멈춰야 하는가? downstream gold mart contract가 조용히 바뀌지는 않는가? 이전 successful run과 지금 run의 schema identity를 비교할 수 있는가? 초기 구현에서는 한 가지 실제 버그가 있었다. schema_hash 가 고정된 required column 목록에 너무 묶여 있어서, 추가 컬럼이 들어와도 hash가 바뀌지 않았다. 즉 operator_id 가 추가되어도 drift가 보이지 않았다. 이 문제를 고치기 위해 read_rows 가 실제 CSV header를 반환하고, 그 실제 header 기준으로 schema_hash 를 계산하도록 바꿨다. 3. Options option result risk ignore drift pipeline은 계속 돈다 source 변화가 보이지 않음 fail every drift 변화에 강하게 반응 정상적인 컬럼 추가도 막음 warn and continue 변화가 보이고 run도 계속됨 warning을 inspect해야 함 auto-evolve silver/gold 새 컬럼을 바로 사용 가능 downstream contract가 조용히 바뀔 수 있음 full schema registry production에 가까움 mini slice에는 무거움 이 프로젝트의 선택은 warn and continue 다. 4. Decision 현재 contract는 이렇다. previous successful run이 없으면: schema_drift = pass baseline schema established current schema_hash == previous successful schema_hash: schema_drift = pass current schema_hash != previous successful schema_hash: schema_drift = warn quality_passed는 true 유지 run/lineage record에 previous/current schema_hash 저장 required column missing: V

source_hash로 같은 입력 재처리를 안전하게 skip하기 작은 데이터 파이프라인도 한 번만 실행된다고 가정하면 금방 거짓말이 된다. 실제로는 같은 파일을 다시 실행할 수 있다. 실패한 run을 재시도할 수도 있고, 과거 날짜를 backfill할 수도 있고, 운영자가 실수로 같은 입력을 다시 넣을 수도 있다. 이때 결과가 중복되면 gold metric은 더 이상 믿을 수 없다. 이 글은 개인 포트폴리오 프로젝트 manufacturing-data-platform-mini 에서 source_hash 를 이용해 같은 입력 재처리를 안전하게 skip하도록 만든 작은 설계 판단을 정리한 글이다. 데이터는 모두 synthetic이며, production platform이 아니라 검증 가능한 mini slice다. 1. Scenario 같은 business_date 의 제조/로봇 이벤트 파일을 다시 처리해야 하는 상황이 있다. 예: retry: 앞 run이 중간에 실패해서 다시 실행한다. backfill: 과거 날짜를 다시 채운다. operator mistake: 같은 파일을 실수로 다시 실행한다. 단순히 매번 append하면 같은 날짜의 gold metric이 중복될 수 있다. 2. Decision Pressure 단순 CSV pipeline은 보통 이렇게 끝난다. CSV 읽기 -> silver 만들기 -> gold 집계 -> 결과 저장 하지만 운영 관점에서는 질문이 생긴다. 이 입력은 전에 처리한 파일과 같은가? 같은 파일을 다시 돌리면 중복 output이 생기나? 다른 파일로 같은 날짜를 다시 돌리면 어떻게 구분하나? 어떤 run이 어떤 source에서 만들어졌나? 그래서 재실행을 판단할 identity가 필요했다. 3. Options option result problem always append 모든 run 결과를 계속 추가 같은 입력 재실행 시 중복 always overwrite 결과를 항상 덮어씀 이전 결과/원인 추적이 약함 skip by business_date only 같은 날짜면 무조건 skip 정정 파일을 반영할 수 없음 skip by dataset_id + business_date + source_hash 같은 입력만 no-op 정정 파일은 새 run으로 처리 가능 이 프로젝트의 Slice1은 마지막 선택지를 쓴다. 4. Decision 현재 mini pipeline은 입력 파일의 content hash를 source_hash 로 계산한다. idempotency key: dataset_id + business_date + source_hash 이미 성공한 run이 있으면 새로 처리하지 않고 기존 run을 재사용한다. same dataset_id same business_date same source_hash prior successful run exists => status = skipped 이 선택은 작지만 중요하다. 같은 파일 재실행: skip -> 중복 없음 같은 날짜의 정정 파일: source_hash가 다름 -> skip하지 않고 새 run으로 처리 가능 단, 여기서 조심해야 할 경계가 있다. Slice1은 다른 source_hash 를 새 run으로 처리할 수 있지만, 이전 gold partition을 원자적으로 교체하는 Iceberg-style overwrite까지 구현한 것은 아니다. 그 문제는 다음 Slice2의 business_date partition overwrite 주제다. 5. Evidence 관련 코드와 검증 evidence: src/manufacturing_data_platform/pipeline/lakehouse.py tests/test_lakehouse_pipeline.py VERIFICATION_LOG.md README.md 검증 로그: 2026-07-08 publication readiness check: pytest: 33 passed lakehouse JSON CLI: passed, status=processed, quality_passed=true EAV JS

Enhancing CI/CD and E2E Testing with Sentry Integration in tvview TL;DR: I integrated Sentry for error tracking and improved End-to-End (E2E) testing in the tvview project, enhancing the CI/CD pipeline. This resulted in a score increase from 85 to 95+. The Problem The tvview project lacked comprehensive error tracking and E2E testing, making it difficult to identify and resolve issues in production. The existing CI/CD pipeline needed improvement to ensure smoother deployments and better code quality. What I Tried First Initially, I focused on setting up E2E tests using Vitest, but encountered issues with the test configuration. I also attempted to integrate Sentry, but faced challenges with the DSN (Data Source Name) configuration. The Implementation Step 1: Configuring Sentry To integrate Sentry, I created separate configuration files for the client, edge runtime, and server: // sentry.client.config.ts import * as Sentry from " @sentry/nextjs " ; Sentry . init ({ dsn : " https://385038c88b6eb6ddac52d05a144ab8c1@o4511628189630464.ingest.us.sent " , // Additional configuration options }); // sentry.edge.config.ts import * as Sentry from " @sentry/nextjs " ; Sentry . init ({ dsn : " https://385038c88b6eb6ddac52d05a144ab8c1@o4511628189630464.ingest.us.sent " , // Additional configuration options }); // sentry.server.config.ts import * as Sentry from " @sentry/nextjs " ; Sentry . init ({ dsn : " https://385038c88b6eb6ddac52d05a144ab8c1@o4511628189630464.ingest.us.sentry.io " , // Additional configuration options }); Step 2: Enhancing CI/CD Pipeline I updated the .github/workflows/ci-e2e.yml file to include Sentry configuration and E2E testing: name : 📺 CI + E2E — TVView on : push : branches : [ main ] workflow_dispatch : {} schedule : - cron : " 35 6 * * *" jobs : build-and-test : runs-on : ubuntu-latest steps : - name : Checkout code uses : actions/checkout@v2 - name : Install dependencies run : npm install - name : Generate Prisma client env : DATABASE_URL : " postgre

Stop Guessing: How I Pick AI API Architecture at Every Scale I've been on both sides of this. Two years ago I was the lone backend engineer at a Series A startup, duct-taping API calls together at 2 AM because the founders wanted a chatbot demo by morning. Last quarter I sat in a procurement meeting at a Fortune 500 where we spent six weeks evaluating three vendors for a single inference workload. Same job title on LinkedIn, wildly different problems. Most AI API guides I've read treat both scenarios like they're the same conversation. They're not. The startup CTO optimizing for burn rate and the enterprise architect worrying about a 99.9% uptime SLA are solving fundamentally different equations. After enough of these conversations, I've developed a framework I'd like to share — and yes, I'll talk about Global API because it's what I actually use, but I'll also explain the reasoning behind each choice so you can adapt it to your own stack. What I Look at First: The p99 Question Before I look at price, I look at the latency distribution. Specifically, the p99. Mean latency tells you almost nothing useful. If your median response is 200ms but your p99 is 4 seconds, your users will see janky behavior on the long tail and you won't know why until production is on fire. For startups in the MVP phase, you can usually get away with best-effort routing. A p99 of 2-3 seconds is fine if you're building an async summarization feature. But the moment you put AI in the synchronous request path — like a customer-facing chatbot or a real-time code suggestion — p99 starts to bite. I learned this the hard way when our startup's "AI assistant" feature had users complaining about slowness that I couldn't reproduce locally. The culprit? Provider cold starts hitting our 1% of users who happened to get routed to a freshly spun-up instance. For enterprises, p99 isn't a nice-to-have, it's a contractual obligation. Most B2B SLAs I've negotiated pin uptime at 99.9% and require reporting on m

Enhancing CraveView's CI/CD Pipeline with Sentry and E2E Tests TL;DR: I upgraded CraveView's CI/CD pipeline by integrating Sentry for error tracking and implementing End-to-End (E2E) tests, boosting the score from 85 to 95+. This technical deep-dive explores the architecture decisions, code changes, and lessons learned. The Problem The initial problem wasn't a single error message but a series of inefficiencies in the CI/CD pipeline. The existing setup lacked comprehensive error tracking and test coverage, leading to potential issues in production. Specifically, the pipeline didn't have: Robust Error Tracking : No integrated system for capturing and analyzing errors. End-to-End Tests : Limited test coverage, which could lead to undetected issues in production. What I Tried First Initially, I focused on enhancing the test suite. I explored various testing frameworks but decided to implement E2E tests using Vitest, given its compatibility with the existing tech stack. The first approach involved setting up a basic E2E test framework. However, I encountered issues with the test environment configuration, particularly with database connectivity. The tests required a realistic database setup, which wasn't properly simulated. The Implementation Step 1: Configuring Sentry To integrate Sentry, I created configuration files for client, edge, and server initialization: sentry.client.config.ts import * as Sentry from " @sentry/nextjs " ; Sentry . init ({ dsn : " https://385038c88b6eb6ddac52d05a144ab8c1@o4511628189630464.ingest.us.sentry.io/4511629 " , // Additional config options }); sentry.edge.config.ts and sentry.server.config.ts follow a similar structure, adjusted for their respective environments. Step 2: Implementing E2E Tests I added a new test file e2e-production.test.ts in src/__tests__ : import { test , expect } from ' @playwright/test ' ; test ( ' should render the homepage ' , async ({ page }) => { await page . goto ( ' https://craveview.vercel.app ' ); await expe

Upgrading CI Workflows: From Node 20 to Node 22 and Actions v5/v6 TL;DR: I upgraded the CI workflows for the content-automation repository from Node 20 to Node 22 and Actions v5/v6, addressing compatibility issues and improving performance. Key changes included updating upload-artifact from v5 to v7 and implementing retry with backoff. The Problem The CI workflows for the content-automation repository were using Node 20 internally, despite the configuration specifying Node 20. This discrepancy caused compatibility issues with newer versions of the GitHub Actions. Specifically, the upload-artifact action was still on version 5, which was internally targeting Node 20. What I Tried First Initially, I attempted to update the upload-artifact action to version 7, which supports Node 22. However, this change alone did not resolve the issue, as other actions like checkout and setup-python were still on older versions. The Implementation To address the compatibility issues, I updated the following actions: upload-artifact from v5 to v7 checkout to v5 setup-python to v6 Here are the specific code changes: // .github/workflows/main.yml steps: - name: Checkout code uses: actions/checkout@v5 - name: Setup Python uses: actions/setup-python@v6 - name: Upload artifact uses: actions/upload-artifact@v7 Additionally, I implemented a retry mechanism with backoff for the CI workflows: // .github/workflows/main.yml steps : - name : Retry with backoff run : | for i in {1..3}; do if ./script.sh; then break else echo "Retry $i failed, backing off..." sleep $((i * 2)) fi done Key Takeaway The key takeaway from this experience is the importance of keeping CI workflows up-to-date with the latest versions of GitHub Actions. This not only ensures compatibility but also improves performance and reliability. What's Next Next, I plan to monitor the CI workflows for any issues and continue to optimize the retry mechanism for better performance. I will also explore other ways to improve the reliabili

Disallow: GPTBot is a wall. Walls don't pay rent, and the crawlers that matter most either ignore them or route around them. If your content is worth training on, the interesting question isn't "how do I keep the bots out" — it's "what do they owe me, and how do I say so in a way a machine can read." That's what RSL (Really Simple Licensing) is for. It shipped 1.0 in December 2025 with around 1,500 publishers behind it — Reddit, Yahoo, Quora, O'Reilly, Medium, Vox. This post is a from-scratch walkthrough of what the format actually is, the six places you can put it, the one mistake that makes crawlers silently ignore your terms, and where the declaration stops and enforcement begins. No tooling required to follow along — it's all plain XML and HTTP. The format is an XML vocabulary, not a config file An RSL document says: for this content, here's what's permitted, what's prohibited, and what it costs. Minimal example: <?xml version="1.0" encoding="UTF-8"?> <rsl xmlns= "https://rslstandard.org/rsl" max-age= "7" > <content url= "/" > <license> <permits type= "usage" > search </permits> <prohibits type= "usage" > ai-train </prohibits> <payment type= "crawl" > <amount currency= "USD" > 0.015 </amount> </payment> </license> </content> </rsl> Read it out loud: search engines may index this; training on it is prohibited; if you want to crawl it anyway, the rate is $0.015. usage tokens include search , ai-train , ai-use (inference/grounding), and a few more. You can scope rules by user and geo too. One rule that trips people up: prohibition wins . If the same token shows up under both permits and prohibits , the content is prohibited. Don't try to express "allowed except for X" by listing X in both — just prohibit X. The namespace is the thing crawlers actually key on The single most common way to publish RSL that quietly does nothing: getting the namespace wrong. It must be exactly: xmlns="https://rslstandard.org/rsl" http instead of https , a trailing slash, or a plausible

submitted by /u/Either_Collection349 [link] [留言]

submitted by /u/Either_Collection349 [link] [留言]

submitted by /u/Either_Collection349 [link] [留言]

The Chinese company is allowed to sell its impressive phones in the States, but it doesn't. Here's why.

SteelSeries has the Arctis Nova Pro Wireless gaming headset on sale for $239.99 (currently between $300 and $350 at other retailers). The Xbox version that supports a host of other platforms including PlayStation, PC, and Switch is available in both black and white. The PlayStation version, which supports the same platforms except for Xbox, is […]

The state of the smart home can be frustrating, because it is just so obvious how things ought to work. You should be able to control everything from everywhere. Your spaces should adapt to what you're doing and how you're feeling. Making your home smart shouldn't require renovating, and the smarts should be mostly invisible. […]

submitted by /u/mitousa [link] [留言]

Know where to place your router and how to position its antennas should get you better coverage.

GitHub热门项目 | 🧰 The Rust SQL Toolkit. An async, pure Rust SQL crate featuring compile-time checked queries without a DSL. Supports PostgreSQL, MySQL, and SQLite. | Stars: 17,291 | 15 stars today | 语言: Rust

GitHub热门项目 | CloudFlare free temp domain email 免费收发 临时域名邮箱 支持附件 IMAP SMTP TelegramBot | Stars: 10,331 | 38 stars today | 语言: TypeScript

GitHub热门项目 | There can be more than Notion and Miro. AFFiNE(pronounced [ə‘fain]) is a next-gen knowledge base that brings planning, sorting and creating all together. Privacy first, open-source, customizable and ready to use. | Stars: 70,354 | 48 stars today | 语言: TypeScript

GitHub热门项目 | Pure Javascript OCR for more than 100 Languages 📖🎉🖥 | Stars: 38,359 | 167 stars today | 语言: JavaScript

GitHub热门项目 | Zapret (Запрет обход блокировки Дискорда и Ютуба) | Stars: 1,335 | 18 stars today | 语言: Python