标签：#data

Cargar 112.647 filas en una tabla de hechos debería tardar segundos. A mí me tardaba más de 25 minutos, y acababa cancelando la query. Los datos estaban bien, el SQL estaba bien, las dimensiones se poblaban sin problema. El culpable era otro, y descubrirlo fue la parte más instructiva de todo el proyecto. Todo esto surgió construyendo un Data Warehouse en estrella sobre datos reales de e-commerce: no una tabla bonita para hacer un SELECT * , sino un modelo dimensional completo, reproducible desde cero, capaz de responder preguntas de negocio de verdad. El dataset Trabajé con el Brazilian E-Commerce Public Dataset by Olist : pedidos reales de un marketplace brasileño entre septiembre de 2016 y octubre de 2018. Son 9 CSV relacionados entre sí: 99.441 pedidos y 112.650 líneas de venta 103.886 pagos y 104.719 reseñas 32.951 productos, 3.095 vendedores 1.000.163 registros de geolocalización Y con trampas de datos reales que hay que ver antes de que te muerdan: Un pedido puede tener varios pagos y varias reseñas. Si los unes tal cual a la tabla de hechos, duplicas ventas . Es el error clásico y silencioso: los totales salen inflados y nadie se entera. customer_id no es un cliente. Olist crea uno por cada pedido; la persona real es customer_unique_id . Contar mal aquí te cambia el KPI: hay 99.441 cuentas frente a 96.096 personas. El CSV de productos trae una errata en la cabecera ( product_name_lenght , con "lenght"). Si tu esquema la escribe bien y cargas por interfaz gráfica (que empareja por nombre ), esas columnas se quedan vacías sin que nadie avise. El proceso Monté una arquitectura en capas: CSV → staging → modelo dimensional → vistas → análisis , todo en cuatro scripts ejecutables en orden y idempotentes (el esquema se recrea desde cero, se puede relanzar mil veces). El modelo es un star schema : una tabla de hechos fact_sales al grano de línea de producto dentro de un pedido , y cinco dimensiones (cliente, producto, vendedor, pago y fecha), con claves sustitutas,

No hacking — a passive scan only looks at what your browser already downloads when it opens a page. Here's what I found: 6 of 15 load their Supabase database directly client-side. The public API key sits in the page source. That's fine if Row-Level Security is configured right — but it's one wrong setting away from "anyone can read the whole table." 14 of 15 ship no Content-Security-Policy — a simple, high-value hardening against script injection, almost always missing. Is this theoretical? No. Two apps I audited with the owner's permission: A social app: the profiles table — user names, cities, and a password hash — readable by a logged-out stranger. Closed in an afternoon. A paid learning app: 155 paid study sheets and 4,872 answers were readable by anyone, with no account and no subscription — its entire paid catalogue, a single API call away. The paywall lived only in the front-end; the database served everything to everyone. Loading Supabase in the browser isn't the mistake. Not enforcing access in the database (RLS) is. And the tools you build with won't tell you — they'll happily ship it. If you built something on Lovable / Bolt / Replit with real users (or paying ones), it's worth 60 seconds to check what a stranger can already see. I made a free tool that runs the surface check (passive, no signup): sealdy.dev Happy to answer questions on how RLS leaks happen and how to lock them down.

If you’ve ever struggled with CALCULATE() or wondered why SUMX() behaves differently from SUM() , this guide is for you. DAX (Data Analysis Expressions) is the language that powers Power BI , Analysis Services , and Power Pivot — enabling dynamic calculations, filtering, and time intelligence. Below is a categorized cheat sheet of essential DAX functions , plus examples showing how to use each in real-world Power BI scenarios. Filtering & Context These functions control how filters are applied and evaluated in your calculations. Function Example Description CALCULATE() CALCULATE(SUM(Sales[Amount]), Region[Name] = "Nairobi") Changes filter context to calculate total sales for Nairobi. FILTER() FILTER(Sales, Sales[Amount] > 10000) Returns a table filtered by condition. ALL() CALCULATE(SUM(Sales[Amount]), ALL(Region)) Ignores filters on Region. REMOVEFILTERS() CALCULATE(SUM(Sales[Amount]), REMOVEFILTERS(Region)) Removes filters from Region. VALUES() VALUES(Customer[City]) Returns unique list of cities. SELECTEDVALUE() SELECTEDVALUE(Product[Category], "All") Returns selected category or “All” if none. TREATAS() TREATAS(VALUES(Temp[City]), Customer[City]) Applies one table’s values as filters on another. KEEPFILTERS() CALCULATE(SUM(Sales[Amount]), KEEPFILTERS(Product[Category] = "Electronics")) Keeps existing filters and adds new ones. ALLSELECTED() CALCULATE(SUM(Sales[Amount]), ALLSELECTED(Region)) Respects user selections in visuals. ALLEXCEPT() CALCULATE(SUM(Sales[Amount]), ALLEXCEPT(Sales, Sales[Year])) Removes all filters except Year. Aggregation Summarize or aggregate data across rows or columns. Function Example Description SUM() SUM(Sales[Amount]) Adds all sales amounts. AVERAGE() AVERAGE(Sales[Amount]) Calculates mean value. COUNT() COUNT(Customer[ID]) Counts non-blank entries. COUNTROWS() COUNTROWS(Sales) Counts rows in a table. DISTINCTCOUNT() DISTINCTCOUNT(Customer[ID]) Counts unique customers. MIN() MIN(Sales[Amount]) Finds smallest sale. MAX() MAX(Sales[Amo

We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana Soumyadeep Dey Soumyadeep Dey Soumyadeep Dey Follow Jul 12 We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana # devchallenge # weekendchallenge # snowflake 5 reactions 1 comment 16 min read

skip에서 partition overwrite로: business_date 재처리를 Iceberg로 다시 표현하기 이전 글에서는 같은 source_hash 가 다시 들어왔을 때 기존 successful run을 재사용하는 idempotency를 다뤘다. 하지만 재처리에는 두 종류가 있다. 1. 같은 입력이 다시 들어온 경우 -> skip이 맞다. 2. 같은 business_date의 정정 입력이 들어온 경우 -> skip하면 안 된다. -> 같은 날짜의 gold 결과를 중복 없이 교체해야 한다. manufacturing-data-platform-mini 의 B5 slice는 두 번째 문제를 아주 작게 다룬다. 전체 Spark pipeline을 만든 것이 아니다. gold_daily_metrics Iceberg table 하나를 local Spark에서 만들고, business_date partition overwrite와 snapshot evidence만 검증했다. Scenario 이미 아래 gold row가 있다. business_date=2026-06-29 plant-a / line-1 / gearbox-a units_produced=120 defect_count=3 나중에 같은 business_date=2026-06-29 에 대한 정정 source가 들어온다. 운영자가 원하는 것은 append가 아니다. 원하지 않는 상태: 2026-06-29 old row 2026-06-29 corrected row -> 같은 날짜 결과가 중복됨 원하는 상태: 2026-06-29 corrected row만 남음 2026-06-30 같은 다른 날짜 partition은 그대로 유지됨 재처리 전후 snapshot evidence가 남음 그래서 이 slice의 질문은 이렇다. 같은 business_date의 정정 source를 처리할 때, gold table에서 해당 날짜 partition만 중복 없이 교체하고, 어떤 run이 어떤 Iceberg snapshot을 만들었는지 남길 수 있는가? Decision Pressure Slice1의 CSV pipeline은 already-successful source를 안전하게 skip할 수 있다. dataset_id + business_date + source_hash 이 key가 같으면 같은 입력이다. 다시 계산해도 같은 결과이므로 기존 run을 재사용한다. 하지만 source_hash 가 달라졌다면 의미가 다르다. same business_date different source_hash 이건 retry가 아니라 correction이다. CSV run-folder 방식에서는 새 run output을 만들 수는 있지만, "현재 gold table에서 해당 날짜를 원자적으로 교체한다"는 table-level 의미가 약하다. Iceberg를 붙이는 이유는 여기 있다. source_hash -> 같은 입력인지 판단하는 idempotency key business_date partition -> 정정 시 교체할 gold table 범위 snapshot_id -> table commit의 evidence 즉 Spark/Iceberg는 도구 이름을 추가하려고 붙인 것이 아니라, 재처리 상태 전이를 더 명확히 표현하기 위해 붙였다. Options Option 장점 문제 판단 same source면 항상 재계산 단순함 retry 때 불필요한 commit이 계속 생김 제외 corrected source를 append 구현 쉬움 같은 날짜 gold row가 중복될 수 있음 제외 whole-table overwrite 단순함 다른 날짜 partition까지 지울 위험 제외 business_date partition overwrite correction 범위가 명확함 Spark/Iceberg 설정과 test가 필요 선택 MERGE/upsert 강력함 이번 skeleton에 과함 backlog 이번 구현은 DataFrameWriterV2.overwritePartitions() 를 사용했다. corrected_d

wide CSV 여러 개를 EAV로 모아 gold mart 만들기 현실의 데이터 소스는 한 가지 모양으로 오지 않는다. 같은 의미의 값도 어떤 파일에서는 생산수량 , 다른 파일에서는 units , 또 다른 파일에서는 made 로 올 수 있다. 온도도 어떤 곳은 섭씨, 어떤 곳은 화씨일 수 있다. 이걸 매번 pipeline code에 if source == ... 로 박기 시작하면 source가 늘 때마다 코드가 지저분해진다. manufacturing-data-platform-mini 의 EAV mini slice는 이 문제를 작게 다룬다. 여러 wide CSV를 mapping config로 표준 attribute에 맞춘 뒤, EAV long format으로 모으고, 다시 gold metric mart로 pivot/aggregate한다. 데이터는 모두 synthetic이고, 회사 코드·고객 데이터·실제 schema는 쓰지 않았다. 1. Scenario 서로 다른 공장/라인/벤더에서 비슷한 제조 지표 파일이 들어온다. 예: plant_a.csv: 설비ID, 생산수량, 불량수, 온도C, 압력kPa plant_b.csv: machine_id, output_units, defects, temp_f, pressure_bar vendor_d.csv: unit_name, made, scrap, deg_c, kpa 비즈니스적으로는 같은 지표를 보고 싶다. units_produced defect_count temperature_c pressure_kpa 문제는 source마다 column name과 unit이 다르다는 점이다. 2. Decision Pressure 단순 구현은 source마다 코드를 늘린다. if source == "plant_a": 생산수량을 units_produced로 읽는다 if source == "plant_b": output_units를 units_produced로 읽는다 temp_f를 섭씨로 변환한다 if source == "vendor_d": made를 units_produced로 읽는다 이 방식은 작게는 빨라 보이지만 source가 늘수록 문제가 된다. 새 파일 형식마다 pipeline code를 고쳐야 한다. column mapping과 transform logic이 섞인다. unit conversion이 흩어진다. quality check가 source별로 갈라진다. gold mart grain을 설명하기 어려워진다. 그래서 mapping은 config로 빼고, pipeline은 표준 attribute를 처리하게 만들었다. 3. Options option result risk source별 hard-coded parser 처음엔 빠름 source가 늘 때 code change 반복 모든 source를 wide table 하나로 합치기 보기 쉬움 sparse/heterogeneous column 폭발 EAV long format 이종 attribute를 표준 형태로 모음 pivot/quality 설계가 필요 full mapping DSL/rules engine 유연함 mini project에는 과함 이 프로젝트의 선택은 단순한 JSON mapping + EAV long + gold pivot이다. 4. Decision 각 source는 JSON config로 자신의 column을 표준 attribute에 매핑한다. source column -> standard attribute output_units -> units_produced temp_f -> temperature_c with f_to_c pressure_bar -> pressure_kpa with bar_to_kpa pipeline 흐름: wide CSVs -> mapping configs -> EAV long rows -> gold entity_daily_metrics -> quality checks -> catalog/lineage EAV row의 핵심 shape: entity_id business_date attribute value v

schema drift를 fail이 아니라 warn으로 둔 이유 데이터 파이프라인에서 source schema가 바뀌는 순간은 애매하다. 무조건 무시하면 운영자는 입력 구조가 바뀐 사실을 모른다. 반대로 모든 schema 변화를 실패로 처리하면, 정상적인 컬럼 추가까지 daily run을 막아버린다. manufacturing-data-platform-mini 에서는 이 문제를 작게 다뤘다. synthetic manufacturing CSV의 실제 header를 기준으로 schema_hash 를 만들고, previous successful run과 비교해 달라졌으면 schema_drift quality check를 warn 으로 남긴다. 단, required column이 빠져 silver/gold contract를 만들 수 없는 경우는 현재 ValueError 로 빠르게 실패한다. 1. Scenario 어느 날 source CSV에 새 컬럼이 추가된다. 기존 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date 새 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date,operator_id operator_id 는 아직 silver/gold mart에서 쓰지 않는다. 하지만 source 구조가 바뀐 사실은 기록되어야 한다. 2. Decision Pressure schema drift에서 중요한 질문은 단순히 "바뀌었나?"가 아니다. 바뀐 것을 운영자가 알 수 있는가? 정상적인 컬럼 추가 때문에 pipeline을 멈춰야 하는가? downstream gold mart contract가 조용히 바뀌지는 않는가? 이전 successful run과 지금 run의 schema identity를 비교할 수 있는가? 초기 구현에서는 한 가지 실제 버그가 있었다. schema_hash 가 고정된 required column 목록에 너무 묶여 있어서, 추가 컬럼이 들어와도 hash가 바뀌지 않았다. 즉 operator_id 가 추가되어도 drift가 보이지 않았다. 이 문제를 고치기 위해 read_rows 가 실제 CSV header를 반환하고, 그 실제 header 기준으로 schema_hash 를 계산하도록 바꿨다. 3. Options option result risk ignore drift pipeline은 계속 돈다 source 변화가 보이지 않음 fail every drift 변화에 강하게 반응 정상적인 컬럼 추가도 막음 warn and continue 변화가 보이고 run도 계속됨 warning을 inspect해야 함 auto-evolve silver/gold 새 컬럼을 바로 사용 가능 downstream contract가 조용히 바뀔 수 있음 full schema registry production에 가까움 mini slice에는 무거움 이 프로젝트의 선택은 warn and continue 다. 4. Decision 현재 contract는 이렇다. previous successful run이 없으면: schema_drift = pass baseline schema established current schema_hash == previous successful schema_hash: schema_drift = pass current schema_hash != previous successful schema_hash: schema_drift = warn quality_passed는 true 유지 run/lineage record에 previous/current schema_hash 저장 required column missing: V

source_hash로 같은 입력 재처리를 안전하게 skip하기 작은 데이터 파이프라인도 한 번만 실행된다고 가정하면 금방 거짓말이 된다. 실제로는 같은 파일을 다시 실행할 수 있다. 실패한 run을 재시도할 수도 있고, 과거 날짜를 backfill할 수도 있고, 운영자가 실수로 같은 입력을 다시 넣을 수도 있다. 이때 결과가 중복되면 gold metric은 더 이상 믿을 수 없다. 이 글은 개인 포트폴리오 프로젝트 manufacturing-data-platform-mini 에서 source_hash 를 이용해 같은 입력 재처리를 안전하게 skip하도록 만든 작은 설계 판단을 정리한 글이다. 데이터는 모두 synthetic이며, production platform이 아니라 검증 가능한 mini slice다. 1. Scenario 같은 business_date 의 제조/로봇 이벤트 파일을 다시 처리해야 하는 상황이 있다. 예: retry: 앞 run이 중간에 실패해서 다시 실행한다. backfill: 과거 날짜를 다시 채운다. operator mistake: 같은 파일을 실수로 다시 실행한다. 단순히 매번 append하면 같은 날짜의 gold metric이 중복될 수 있다. 2. Decision Pressure 단순 CSV pipeline은 보통 이렇게 끝난다. CSV 읽기 -> silver 만들기 -> gold 집계 -> 결과 저장 하지만 운영 관점에서는 질문이 생긴다. 이 입력은 전에 처리한 파일과 같은가? 같은 파일을 다시 돌리면 중복 output이 생기나? 다른 파일로 같은 날짜를 다시 돌리면 어떻게 구분하나? 어떤 run이 어떤 source에서 만들어졌나? 그래서 재실행을 판단할 identity가 필요했다. 3. Options option result problem always append 모든 run 결과를 계속 추가 같은 입력 재실행 시 중복 always overwrite 결과를 항상 덮어씀 이전 결과/원인 추적이 약함 skip by business_date only 같은 날짜면 무조건 skip 정정 파일을 반영할 수 없음 skip by dataset_id + business_date + source_hash 같은 입력만 no-op 정정 파일은 새 run으로 처리 가능 이 프로젝트의 Slice1은 마지막 선택지를 쓴다. 4. Decision 현재 mini pipeline은 입력 파일의 content hash를 source_hash 로 계산한다. idempotency key: dataset_id + business_date + source_hash 이미 성공한 run이 있으면 새로 처리하지 않고 기존 run을 재사용한다. same dataset_id same business_date same source_hash prior successful run exists => status = skipped 이 선택은 작지만 중요하다. 같은 파일 재실행: skip -> 중복 없음 같은 날짜의 정정 파일: source_hash가 다름 -> skip하지 않고 새 run으로 처리 가능 단, 여기서 조심해야 할 경계가 있다. Slice1은 다른 source_hash 를 새 run으로 처리할 수 있지만, 이전 gold partition을 원자적으로 교체하는 Iceberg-style overwrite까지 구현한 것은 아니다. 그 문제는 다음 Slice2의 business_date partition overwrite 주제다. 5. Evidence 관련 코드와 검증 evidence: src/manufacturing_data_platform/pipeline/lakehouse.py tests/test_lakehouse_pipeline.py VERIFICATION_LOG.md README.md 검증 로그: 2026-07-08 publication readiness check: pytest: 33 passed lakehouse JSON CLI: passed, status=processed, quality_passed=true EAV JS

Losing PostgreSQL Gains? Blame Inline JSONB!! PostgreSQL's jsonb is a favorite among developers for its flexibility - but it hides a dark side. When used carelessly, especially in-line within rows under 2KB, it can silently destroy performance, even if you're using indexes. Here's why. 🔍 The Hidden Cost of JSONB (Inline Storage) PostgreSQL stores table rows in 8KB pages, packing as many tuples as possible. For a typical row with 10–12 columns, and small text/integers, 40–100 rows can easily fit per page. Typically row count = Page Size(8kb) / row size + row metadata (30-50 bytes approx.) But the game changes when you add jsonb. Example CREATE TABLE events ( id serial PRIMARY KEY, user_id int, action text, metadata jsonb ); Suppose metadata which is a jsonb column contains: { "ip": "127.0.0.1", "device": "Android", "country": "IN" } This JSON might be just 100–500 bytes, so PostgreSQL stores it in-line inside the same page (no TOASTing). Result Each row size jumps from ~80 bytes → ~200–400 bytes Row count per page drops from 100 → 20–40 Index scan still needs to read each page for matching rows More pages = more I/O, slower performance 🔢 Real Benchmark Insight Performance comparisonEven with a GIN or B-tree index on the JSONB column, PostgreSQL still needs to scan all matching pages to retrieve the full tuple. 🧠 Why Index Doesn't Save You Say you index a JSONB key like: CREATE INDEX ON events ((metadata->>'ip')); And query: SELECT * FROM events WHERE metadata->>'ip' = '127.0.0.1'; PostgreSQL will: Use the index to find matching tuples Still need to fetch the row from disk Because JSONB is in-line, many pages are touched More page fetches = more IO = slower queries 🩹 What You Can Do ✅ Force TOAST: Add padding to make JSONB exceed 2KB: UPDATE events SET metadata = metadata || jsonb_build_object('padding', repeat('x', 2000)); ✅ Split into separate table: If JSONB is rarely queried ✅ Stick to well defined schema and avoid using jsonb unless absolutely necessary. 🧾 TL;DR

In Part 2 , we learned why AI sometimes hallucinates. One of the biggest reasons is that an LLM can only answer based on what it learned during training and the information available in its context window. We also introduced grounding -providing the model with reliable information at runtime instead of expecting it to know everything. But that raises an important question: Where does that information come from? Modern AI applications don't simply dump an entire database or a thousand-page PDF into the prompt. Instead, they first identify the most relevant pieces of information and only send those to the model. In this article, we'll learn how that works. Running Example Let's continue building our AI-powered Travel Planner . So far, it can answer general travel questions using the knowledge it learned during training. Now we want to make it much smarter by uploading several documents into our application: Lonely Planet's Japan travel guide A PDF containing train schedules A document listing recommended local restaurants Hotel information Internal travel policies for our company Together, these documents contain hundreds of pages. Now the user asks: I'm staying near Tokyo Station. Which ramen restaurant from our travel guide is within walking distance and is known for vegetarian options? Somewhere in those hundreds of pages is the answer. The challenge is no longer generating text-it's finding the right information first. The Problem: An LLM Can't Read Your Entire Knowledge Base Every Time A common misconception is that AI applications simply send all their documents to the model. Imagine our travel guide contains 450 pages, thousands of restaurant listings, hotel descriptions, transportation details, and sightseeing recommendations. Sending all of that to the LLM every time someone asks "Where should I eat tonight?" creates several problems. First, many documents are simply too large to fit inside the model's context window. Second, even if they did fit, making the

Every data strategy document written this decade contains some version of the same sentence: we need a single place to access all our data. The sentence is right. The trouble starts on the next page, because there are two fundamentally different ways to build that single place, and the industry has spent years arguing about them as if they were rivals. Road one is consolidation: bring the data together. Land everything in one governed store, in this era an open lakehouse, Apache Iceberg tables on object storage, and point every consumer at it. Road two is federation: leave the data where it lives and bring the access together instead. A query engine that speaks to your databases, warehouses, lakes, and applications in place, presenting one surface over many sources, with no copies made. I work at Dremio, a company whose platform is built on the conviction that this is a false choice, that the right architecture uses both roads with judgment, and I will declare that bias now and then earn it with an honest treatment. Because the truth practitioners live is messier than either camp's marketing: federation without a lakehouse hits performance and scale ceilings, a lakehouse without federation spends years and fortunes migrating the long tail, and the teams that win treat the two as phases and partners rather than competitors. So this article is the full playbook. What federation and the lakehouse each actually are, mechanically. The honest strengths and limits of each, including the failure modes their advocates gloss over. A concrete decision framework for when each one carries a workload. The lifecycle pattern that connects them, federate first, promote deliberately. And the unified architectures, mine included, that put both behind one governed door, which matters more than ever now that the consumers walking through that door increasingly are AI agents. Why Unify at All: The Cost of the Status Quo Before the two roads, the destination deserves a paragraph, because

series: Prediksi Survival Klub Debutan Kenapa Project Ini? Setiap musim, klub yang promosi ke liga top (Premier League, La Liga, dst.) menghadapi risiko besar: sekitar 2 dari 3 klub yang naik biasanya kembali terdegradasi di musim pertama mereka. Saya penasaran — bisakah performa di beberapa laga awal musim memberi sinyal dini soal peluang klub tersebut bertahan? Ini jadi project portofolio pertama saya sebagai data scientist yang baru mulai (0-1 tahun pengalaman). Saya sengaja pilih topik yang saya suka (sepak bola) supaya prosesnya tetap enjoyable, bukan cuma "tutorial project" generik. Rencana Project Pertanyaan utama: Berdasarkan performa 8 laga pertama musim debut, seberapa besar peluang klub promosi bertahan hingga musim berikutnya (tidak degradasi)? Data yang dipakai: football-data.co.uk — data hasil pertandingan tiap musim sejak 1993/1994 Wikipedia (halaman musim liga) — daftar klub promosi & klasemen akhir musim Tech stack: pandas , requests untuk data collection scikit-learn untuk modeling (mulai dari Logistic Regression sebagai baseline) imbalanced-learn untuk handle class imbalance Streamlit + Plotly untuk dashboard interaktif Deploy ke Streamlit Community Cloud Timeline (Build in Public) Saya bikin timeline ini publik supaya ada tekanan yang sehat untuk benar-benar menyelesaikannya, bukan cuma jadi ide yang menguap: Checkpoint Target Tanggal Yang Harus Selesai Part 1 (post ini) 11 Juli 2026 Kickoff, rencana, environment siap Part 2 15 Juli 2026 Dataset jadi, push ke GitHub Part 3 17 Juli 2026 EDA selesai, insight awal Part 4 24 Juli 2026 Model final dipilih + evaluasi Part 5 31 Juli 2026 Dashboard live di Streamlit Cloud Part 6 (final) 8 Agustus 2026 Project selesai, recap lengkap Tantangan yang Sudah Saya Antisipasi Data leakage — fitur harus dihitung dari laga awal musim saja, bukan seluruh musim, biar model beneran memprediksi bukan "menyontek" hasil akhir Dataset kecil — kemungkinan hanya ~60-100 sampel klub, jadi saya mulai dari model sederhana (Lo

By the team at MailTester Ninja — a real-time email verification API that stores nothing. Everyone says "IPv6 is here." For the web, mostly true. For email , it is a mirage. We resolved the MX records of 50,000 of the most-linked domains and checked whether any of their mail servers publish an AAAA record, meaning they can actually receive over IPv6. No sending, no personal data, just DNS. 55.2% of mail-enabled domains have at least one IPv6-capable MX. That sounds healthy. It is not, because two companies carry almost the whole number: Email provider IPv6 MX Other / self-hosted ██░░░░░░░░ 18.4% Google Workspace / Gmail ██████████ 100% Microsoft 365 / Outlook █████████░ 91.3% Proofpoint ░░░░░░░░░░ 0.6% Mimecast ░░░░░░░░░░ 0% Tencent QQ ░░░░░░░░░░ 4.2% Namecheap ░░░░░░░░░░ 0.2% Cisco IronPort ░░░░░░░░░░ 4.5% Zoho ░░░░░░░░░░ 0% Barracuda ░░░░░░░░░░ 0% Google ( 100% ) and Microsoft ( 91.3% ) run IPv6 on nearly every inbox. Remove those two, the providers that already anchor most of the world's mail, and IPv6 email adoption falls from 55.2% to 12.9% . The enterprise security gateways that gate corporate mail, such as Proofpoint, Mimecast and Barracuda, are effectively not on IPv6 at all. Why it matters for deliverability. IPv6-only sending is a dead end. It reaches Gmail and Outlook and little else. Dual-stack is not optional. IPv4 is still the backbone of email, and that is where blocklists, FCrDNS and IP reputation are mature. The takeaway: IPv6 email is not adopted. Google and Microsoft adopted it for you. Plan your sending for an IPv4 world with two big IPv6 exceptions. Check any domain yourself — our free deliverability analyzer shows a domain's MX / SPF / DMARC in one click (no signup, nothing stored). Need to confirm whether a specific mailbox actually exists and is deliverable? That is exactly what MailTester Ninja's email verifier does in real time — and we store no data. Source: MailTester Ninja's open Email Infrastructure Index — a live DNS scan of 50,000 of

Adapted from an appendix of my MS thesis. Markov Chain Monte Carlo Almost as soon as computers were invented, they were used for simulation. Markov chain Monte Carlo (MCMC) was invested as Los Alamos, Metropolis et al (1953) simulated a liquid in equilibrium with its gas phase. Their tour de force was the realization that they did not need to simulate the exact dynamics, they only needed to simulate some Markov chain with the same equilibrium distribution. The Metropolis algorithm was widely used by chemists and physicists, but was not widely known among statisticians until after 1990. Hastings (1970) generalized the Metropolis algorithm, and simulations following his scheme are said to use the Metropolis-Hastings (MH) algorithm [1]. A special case of the MH algorithm was introduced by Geman et al (1984) discussing optimization to find the posterior mode rather than simulation. Algorithms following their scheme are said to use the Gibbs sampler. It took some time for the spatial statistics community to understand that the Gibbs sampler simulated the posterior distribution, thus enabling full Bayesian inference of all kinds. Gelfand et al (1990) made the wider Bayesian community aware of the Gibbs sampler, and then it was rapidly realized that most Bayesian inference could be done using MCMC, whereas very little could be done without MCMC. Green (1995) generalized the MH algorithm as much as it could be generalized [1]. Theoretical Foundations A sequence X 1 ​ , X 2 ​ , … of random elements of some set is a Markov chain if the conditional distribution of X n + 1 ​ given X 1 ​ , … , X n ​ depends on X n ​ only. The set in which the X i ​ take values is called the state space of the Markov chain. A Markov chain has stationary transition probabilities if the conditional distribution of X n + 1 ​ given X n ​ does not depend on n . This is the main kind of Markov chain of interest in MCMC. The joint distribution of a Markov chain is determined by the following [1]. The ma

Let's be real: our personal medical history is a mess. It’s a chaotic mix of PDF lab results, grainy scans of prescriptions, and cryptic Electronic Medical Records (EMR) scattered across different hospital portals. If you’ve ever tried to remember exactly when a specific symptom started or how your cholesterol has trended over the last decade, you know the "search" struggle is real. In this guide, we are moving beyond simple folders. We are architecting a Personal Health Knowledge Base using a modern Vector Database and RAG (Retrieval-Augmented Generation) pipeline. We’ll leverage Qdrant for high-performance similarity search, Unstructured.io for complex document parsing, and Sentence-Transformers to turn 10 years of medical jargon into searchable embeddings. By the end of this post, you'll have a system capable of cross-year symptom correlation and instant medical history retrieval. The Architecture: From Pixels to Insights 🏗️ The biggest challenge with medical records isn't storage; it's ingestion . Medical PDFs are notoriously difficult to parse because they often contain nested tables and checkboxes. Our pipeline handles this by isolating the layout before embedding. graph TD A[Raw Medical Data: PDFs, Scans, EMRs] --> B[Unstructured.io: Partitioning & OCR] B --> C[Text Chunking & Cleaning] C --> D[Sentence-Transformers: Vector Embedding] D --> E[(Qdrant Vector DB)] F[User Query: 'Show me my blood sugar trends since 2015'] --> G[FastAPI Interface] G --> H[Query Embedding] H --> I[Vector Search in Qdrant] I --> J[Contextual Results + LLM Synthesis] J --> K[Actionable Health Insight] Prerequisites 🛠️ To follow along, you'll need: Python 3.9+ Unstructured.io : For the heavy lifting of PDF/Image parsing. Qdrant : Our vector engine (run it via Docker: docker run -p 6333:6333 qdrant/qdrant ). Sentence-Transformers : To generate local embeddings without sending sensitive data to the cloud. FastAPI : To wrap it all in a slick API. Step 1: Parsing the Chaos with Unstructu

It's Friday evening. An endpoint that normally answers in 200 milliseconds is suddenly taking eight seconds. You open Grafana. Every graph is green. CPU is calm, memory is fine, the disk isn't full. By every dashboard you have, the database is healthy. It is not healthy. This is the failure mode monitoring is worst at: the server is unmistakably alive , so nothing alerts, while inside the database something is slowly rotting. A table has bloated. An index nobody uses is dragging down every INSERT . A forgotten transaction is sitting open, holding a lock and quietly making everything worse. None of it crashes. It just degrades, a little at a time, until one Friday evening it tips over. The good news is that Postgres will tell you all of this — you just have to ask. The queries below run on bare PostgreSQL (13 or newer; one version note along the way), need no agent and no paid monitoring, and use an extension in exactly one place where it genuinely earns it. Open psql and check your own database as you read. 1. The cheapest signal: dead rows Start here, because it costs nothing and catches the most. Postgres never deletes a row in place. An UPDATE or DELETE leaves behind a dead tuple — an old version of the row — and autovacuum cleans those up later. Until it does (or if it can't keep up), the dead rows sit in the table, taking space and forcing every scan to page past them. The fastest look is pg_stat_user_tables , always available, no extension: SELECT schemaname , relname AS table , n_live_tup , n_dead_tup , round ( n_dead_tup * 100 . 0 / nullif ( n_live_tup + n_dead_tup , 0 ), 1 ) AS dead_ratio , last_autovacuum FROM pg_stat_user_tables WHERE n_dead_tup > 0 ORDER BY n_dead_tup DESC LIMIT 20 ; A dead_ratio above ~20% on a large table is worth investigating. And watch for a table where the ratio is high and last_autovacuum is empty — that means autovacuum has never successfully run on it, which is its own red flag (we'll see why in section 5; the whole story conver

Reddit looks simple and hides two genuinely hard problems. Comments nest arbitrarily deep, and a naive tree structure makes loading a busy thread slow. The front page reorders itself constantly, so ranking cannot just count votes or old posts would never leave. Both problems have well-known answers, and both are good lessons in choosing the right model. The core problem A comment thread is a tree. Each comment can reply to any other, so depth is unbounded. If you store only "this comment's parent id" and then try to load a whole thread, you walk the tree one level at a time, one query per level, which gets slow for deep or wide threads. Loading a popular post with thousands of nested comments should not take thousands of queries. Ranking is the second problem. If the front page sorted by raw vote count, the highest-voted post of all time would sit at the top forever. If it sorted by newest, quality would drown in noise. You need a score that blends how good a post is with how fresh it is, so good new posts can climb and old ones fade even if they were once popular. Key design decisions Store the parent pointer, but do not traverse at read time. The simple model is a parent_id per comment, which is easy to write but expensive to read as a tree. To load a thread cheaply, fetch all comments for the post in one query, then assemble the tree in application memory. One read, in-memory tree building. This works because a single post's comments, while numerous, fit in memory to assemble. Consider a path or closure model for deep trees. For very deep threads, some systems store a materialized path on each comment, an encoded ancestor chain, so you can fetch an entire subtree with a single prefix query and sort by the path to get correct display order. Another option is a closure table that records every ancestor-descendant pair, which makes subtree queries direct at the cost of extra write work. The right choice depends on how deep threads get and how often you read subtrees

TL;DR Retailers process thousands of inventory transactions every second across physical stores, eCommerce platforms, warehouses, suppliers, and fulfillment centers. Yet many inventory systems still rely on scheduled synchronization, causing stock levels to become outdated within minutes. The result is overselling, delayed replenishment, inaccurate inventory visibility, and avoidable stockouts. Apache Kafka enables real-time inventory management by treating every inventory movement as an event that is streamed the moment it occurs. Sales, returns, warehouse transfers, supplier deliveries, and IoT sensor updates are continuously processed to maintain a consistent inventory view across all retail systems. This event-driven approach helps retailers improve inventory accuracy, automate replenishment, detect stockouts before they occur, and respond to changing demand in near real time. In this guide, you'll learn how Apache Kafka powers real-time inventory management, explore a production-ready reference architecture, understand how inventory events are processed across retail systems, and discover implementation best practices for building scalable, resilient inventory streaming applications. Introduction Retail inventory management has evolved far beyond tracking products on warehouse shelves. Today's retailers operate across physical stores, eCommerce platforms, online marketplaces, distribution centers, and supplier networks, where inventory levels change continuously throughout the day. Every sale, return, warehouse transfer, supplier delivery, and inventory adjustment impacts product availability, making accurate inventory visibility essential for delivering a seamless customer experience. However, many retailers still rely on scheduled synchronization between Point-of-Sale (POS) systems, Warehouse Management Systems (WMS), Enterprise Resource Planning (ERP) platforms, and online storefronts. While these systems perform different functions, they all depend on accur

Original Japanese article : Iceberg REST Catalogを直接叩いて、Glue Data CatalogとS3 Tablesの違いを理解する Introduction I'm Aki, an AWS Community Builder ( @jitepengin ). Most of the time, when working with Iceberg tables, we reach for PyIceberg or Spark. I'm no exception, and honestly there were parts of the PyIceberg configuration — rest.sigv4-enabled , rest.signing-name , warehouse — that I understood only vaguely. Iceberg defines a standard called the Iceberg REST Catalog Open API specification , and AWS implements it through two separate endpoints: The AWS Glue Iceberg REST endpoint ( https://glue.<region>.amazonaws.com/iceberg ) The Amazon S3 Tables Iceberg REST endpoint ( https://s3tables.<region>.amazonaws.com/iceberg ) If two implementations follow the same spec, sending the same requests to both and comparing the results should reveal what's actually different between them. In this article, I'll bypass clients like PyIceberg entirely and hit the REST API directly to explore the differences between the two endpoints. To state the conclusion up front: Even though both implement the same Iceberg REST Catalog specification, Glue is designed as an "entry point to multiple catalogs," while S3 Tables is designed as an "entry point to a single table bucket." That difference is visible just by looking at the URL paths. I previously wrote about the relationship between S3 Tables and Glue Data Catalog in another article — worth a read alongside this one: Does Amazon S3 Tables Replace AWS Glue Data Catalog? Understanding Their Relationship What Is the Iceberg REST Catalog? The Iceberg REST Catalog is a specification that standardizes Iceberg catalog operations as an HTTP API. It's published as an OpenAPI definition (YAML), and any catalog that conforms to it can be accessed the same way from clients such as PyIceberg, Spark, and Trino. The key points of the spec are: URL paths follow a pattern like GET /v1/{prefix}/namespaces , where {prefix} is a free-form segment Clients first call

I run a small AI trends site, and three weeks ago I started doing something simple: every day, snapshot the star count of every repo that crosses my GitHub trending scan for AI. No judgment, no curation, just append-only rows in a database. 611 repos and 2,671 data points later (June 19 to July 10), the picture of what's actually growing looks pretty different from what my feeds told me was hot. Here's what the data says. Before publishing this I re-checked every number below against GitHub's live API. Star counts drift by the hour, so treat them as of July 10. The top 10 risers, by raw stars gained Repo Gained Window From → To calesthio/OpenMontage +30,253 21 days 5,899 → 36,152 DeusData/codebase-memory-mcp +20,483 19 days 7,516 → 27,999 mattpocock/skills +19,053 15 days 137,485 → 156,538 obra/superpowers +16,887 20 days 232,908 → 249,795 NousResearch/hermes-agent +14,896 21 days 197,297 → 212,193 Panniantong/Agent-Reach +14,334 14 days 34,780 → 49,114 usestrix/strix +13,243 12 days 26,363 → 39,606 addyosmani/agent-skills +12,685 21 days 63,156 → 75,841 asgeirtj/system_prompts_leaks +11,720 21 days 43,415 → 55,135 msitarzewski/agency-agents +11,055 10 days 118,241 → 129,296 Windows differ because I only hold snapshots for the days a repo appeared in my scan; each row states its own real window. Three things in this data genuinely surprised me. 1. "Skills" are eating agent frameworks Four of the top ten are not agent frameworks. They are collections of packaged expertise that plug into an existing agent: obra/superpowers (still compounding at roughly 840 stars a day on a 250k base), mattpocock/skills, addyosmani/agent-skills, msitarzewski/agency-agents. A year ago this table would have been full of new frameworks. Now the framework layer looks settled and the growth is in what you load INTO the agent. The moat moved from orchestration code to encoded judgment. 2. The sharpest climbs are applications, not infrastructure The steepest sustained climb from a newcomer in