今日已更新 231 条资讯 | 累计 34096 条内容
关于我们

标签:#p

找到 14098 篇相关文章

AI 资讯

Kiponos Java SDK 5.0 What’s New — Developer Guide

Kiponos Java SDK 5.0 What’s New — Developer Guide This is the technical companion to the 5.0 milestone announcement: what changed, how modes behave, how to read config with the Folder API, and how to upgrade cleanly. Version 5.0.0.260710 Maven group io.kiponos Artifacts sdk-boot-3 (recommended), sdk-boot-2 (legacy) Released 2026-07-12 (Maven Central) Happy product story: SDK 5.0 milestone post . 1. Summary for busy engineers 5.0 productizes client reliability using a classic state pattern behind a stable facade: Mode When Config reads Mutations / hooks Notes Ready Connected to hub Live in-memory tree Full Production happy path Offline Disconnected but LKG available Last Known Good (read-only) No-op / ignored Survives hub blips without inventing values Safe Fail-closed Empty / null-safe No-op Diagnostic dumps must not overwrite LKG Public entry remains: Kiponos kiponos = Kiponos . createForCurrentTeam (); You do not receive mode instances as the API surface. Modes switch internally. Query with: kiponos . getCurrentMode (); kiponos . isReadyMode (); kiponos . isOfflineMode (); kiponos . isSafeMode (); 2. Install Gradle — Boot 3 repositories { mavenCentral () } dependencies { implementation 'io.kiponos:sdk-boot-3:5.0.0.260710' } Gradle — Boot 2 implementation 'io.kiponos:sdk-boot-2:5.0.0.260710' Runtime inputs Input Mechanism Identity env KIPONOS_ID Access env KIPONOS_ACCESS Profile / tree slice JVM -Dkiponos="['App']['1.0.0']['dev']['base']" Tokens and profile come from the Kiponos Connect screen for your team. sdk-common is not a separate app dependency for consumers — boot jars include shared classes (fat-jar pattern). 3. Architecture (state pattern) Application code │ ▼ Kiponos / KiponosBase ◄── stable facade (one reference for app lifetime) │ ▼ volatile SdkState ├── ReadyMode* → live WebSocket + full Folder ops ├── OfflineMode* → LKG reads only └── SafeMode* → fail-closed + safe diagnostic dump Design rule: never return Ready/Offline/Safe objects to callers. Retur

2026-07-13 原文 →
AI 资讯

The handshake is the easy part. Agent payments still haven't named the custody split.

Agent payment protocols are converging fast. The Linux Foundation launched the x402 Foundation around a protocol initially developed by Coinbase, Cloudflare, and Stripe, with Google, AWS, Visa, Mastercard, and Circle among the organizations expressing initial support. It is increasingly framed as an "SSL for AI commerce." When a protocol reaches that stage, three things have to mature together: the specification, the interoperability suite, and the adversarial security scenarios. Today x402 has the specification and growing interoperability machinery. What is not yet visible is a normative adversarial suite every client, resource server, and facilitator must pass. There is a precedent worth sitting with. As check clearing scaled in the early Federal Reserve era, the answer was not merely a better check format. The system added a common clearing and settlement layer. The Federal Reserve could credit the collecting institution's reserve account and debit the paying institution's account, creating an independent record against which the participating banks could reconcile. The important separation was not that the Fed guaranteed every check. It was that settlement no longer depended solely on either participant's account of what happened. (Credit to Starfish on Moltbook, whose custody-split framing sharpened this for me.) This pattern exists across mature financial systems as separation of duties: the maker is not the checker. Its modern equivalent for the controls surrounding agent payments is not custody in the asset-control sense. It is an independent, reproducible assurance plane. x402 can provide independently inspectable evidence that an on-chain payment settled. That does not independently establish that every security and policy condition surrounding the payment was evaluated correctly. The payment handshake is standardizing. The assurance semantics around it are not. The party that performs a security check can attest that it ran. But its own attestation shoul

2026-07-13 原文 →
AI 资讯

I built Regdrift, a CLI and GitHub Action for detecting breaking CMSIS-SVD changes

Hi guys, I've been working on Regdrift, my first open-source project. It's a CLI and GitHub Action that compares two CMSIS-SVD files to check whether there are any register-map changes that could affect firmware functionality. It catches changes such as moved registers, interrupt renumbering, access changes, and altered read/write behavior. It then classifies those changes as BREAKING , WARNING , or SAFE so the tool can act as a CI gate. I'm looking for feedback from people who maintain SVDs, HALs, PACs, SDKs, or firmware repositories. If possible, I'd like to test it against real old/new SVD pairs and learn where the classifications produce false positives, miss important changes, or are unclear. For people who work frequently with CMSIS-SVD files: which types of register-map changes are most detrimental to firmware or cause the most difficult code-related problems? Resources GitHub: https://github.com/Pranav-s79/regdrift Install pip install regdrift Usage regdrift check old.svd new.svd

2026-07-13 原文 →
AI 资讯

The monitoring agent that cannot be told what to do

Here is a design decision we made early, wrote into the architecture as an invariant, and have refused to revisit since: our agent accepts no commands. Not "we don't currently use that feature" — the hub has no way to tell an installed agent to do anything at all. No remote execution, no self-update, no "collect this for us right now". It sends data outward, and that is the entire surface. This is not a limitation we are working around. It is the product. And it costs us features that customers ask for, which is exactly why it is worth explaining. The uncomfortable arithmetic of remote control Any tool that can update a plugin across fifty client sites is, by construction, a tool that can execute code on fifty client sites. Any dashboard that can restart a service on your server holds, somewhere, a credential that lets it in. This is not a flaw in those products — it is what they are for. You cannot automate a repair without the power to perform it. But that power has an owner, and the owner has a login, and the login has a support team, and somewhere in that chain there is a version of the software with a bug in it. When the tool is compromised, the blast radius is not the tool. It is every machine the tool could reach. The industry has already run this experiment at scale. In July 2021, attackers exploited a vulnerability in a widely used remote monitoring and management platform. They did not break into a single company — they broke into the thing that had access to the companies. Roughly sixty managed service providers were hit, and through them, an estimated 800 to 1,500 downstream businesses were encrypted in a single weekend, with a $70 million ransom demand attached. Read that shape again, because it is the whole argument: the victims did nothing wrong. They had bought a well-known product from a serious vendor and installed it exactly as instructed. Their compromise arrived through the door they had deliberately, sensibly, contractually left open — the one

2026-07-13 原文 →
AI 资讯

I built a browser CAD where you type a sentence and walk through the house

Concept design for a building is slow and expensive. A homeowner planning an extension, or a contractor trying to win a job, is stuck between two bad options: pay a drafter $500–2,000 for a concept package, or fight SketchUp's learning curve for a week. Meanwhile the actual idea — "a 4-bed duplex with a garage and a palm out front" — fits in one sentence. So I built Forge3D Spaces : you type that sentence, and a few seconds later you're walking through a furnished 3D house in your browser — with measured floor plans, DXF for AutoCAD, and a cost estimate that come out of the same model. No install. Here's how it works under the hood. The pipeline: sentence → structured plan → building The naive approach — "ask an LLM to emit a 3D scene" — falls apart fast. Models are bad at spatial consistency; walls don't meet, rooms overlap, doors float. So the LLM never touches geometry directly. It emits a structured program , and a deterministic solver turns that into a watertight building. The prompt becomes a spec. A strict JSON-schema call (OpenRouter, json_schema response format with every field required) turns "4-bed duplex with a garage" into a room program: room types, target areas, adjacencies, storeys. A slicing-tree solver lays it out. This is the old floorplanning trick from chip design — recursively split a rectangle with horizontal/vertical cuts until every room has its area. A squarify pass keeps rooms from collapsing into corridors. The output is exact rectangles with real dimensions, guaranteed non-overlapping and gap-free. Walls, openings, roof, furniture get generated from the solved plan. Every door and window is placed by rule, not by vibes. Because the plan is a real data structure, the 2D floor plan, the 3D model, the elevations, and the bill of quantities are all views of the same thing . Drag a wall and they all move together. Nothing drifts out of sync, because there's nothing to sync — it's one model. The rendering: WebGPU, and the fallback you actually

2026-07-13 原文 →
开发者

How Processes Share Memory Without Copying (Visual)

A visual explanation of how two separate processes can access the same bytes without copying a payload between them. It covers virtual memory, page tables, physical frames, shm_open, mmap(MAP_SHARED), lazy allocation, copy-on-write, shared libraries, memory-mapped files, Redis snapshots, and Dirty COW. submitted by /u/Ok_Marionberry8922 [link] [留言]

2026-07-13 原文 →
开发者

Exploiting PackageInstaller parsing: A 974-byte modern Android 14 PoC.

I’ve been exploring the absolute structural floor of the Android APK format. By leveraging hasCode="false" and surgically optimizing the ASN.1 DER encoding of the V2 signature, I’ve managed to get a compliant Android 14 app down to 974 bytes. The system treats it as a first-class app, and it’s fully installable on a stock Android 14 device. It’s an exercise in how much the PackageManager trusts the headers versus what it semantically validates. submitted by /u/Same-Access-6799 [link] [留言]

2026-07-13 原文 →
AI 资讯

Building a Three.js 3D Product Configurator for WooCommerce: 4 Things I Didn't Expect

Most WooCommerce product pages still show the same thing stores have shown for 20 years: a handful of flat photos. I spent the last few months building Noorifa, a plugin that replaces that with an interactive Three.js viewer — customers rotate the model, zoom in, and switch colors/materials on specific meshes in real time, synced to the store's actual WooCommerce variations. The 3D rendering part was the easy 20%. The other 80% was a series of small, specific problems that don't show up in a Three.js tutorial. Here are four of them. 1. A directional light rig can't light a face it can't see Early on, customers rotating a table model would find the underside of the tabletop rendering near-black — no matter how far I pushed the light intensity. The rig at the time was a single key light plus a hemisphere ambient: scene . add ( new THREE . HemisphereLight ( 0xffffff , 0x444444 , 1.2 ) ); const keyLight = new THREE . DirectionalLight ( 0xffffff , 1.2 ); keyLight . position . set ( 3 , 5 , 4 ); scene . add ( keyLight ); The bug was geometric, not a brightness problem: keyLight sits above the model, so its light direction only reaches surfaces whose normal faces back toward it. A downward-facing surface — the underside of an overhanging tabletop — can't receive any direct contribution from a light positioned above it, at any intensity. Cranking the brightness slider was scaling a number that was multiplying against zero. The fix was closer to actual three-point studio lighting: key, fill, and rim from above for shape and separation, plus a dedicated light from below, and a brighter hemisphere ground color to approximate bounced light: scene.add( new THREE.HemisphereLight( 0xffffff, 0x888888, 1.1 * brightness ) ); const keyLight = new THREE.DirectionalLight( 0xffffff, 1.1 * brightness ); keyLight.position.set( 3, 5, 4 ); const fillLight = new THREE.DirectionalLight( 0xffffff, 0.5 * brightness ); fillLight.position.set( -4, 2, 3 ); const rimLight = new THREE.DirectionalLigh

2026-07-13 原文 →
AI 资讯

Be the right Platform Team

Throughout my career I have had to work with quite a few platform teams, and I was part of two for a couple of years. Some were bad, some were good and some should not have existed at all. I want to tell you my user experience, what I have seen work and what not and what you should definitely avoid doing. Be The Multiplier This is the main goal of a platform team. As a team, it needs to be a multiplier. If the platform team supports 10 teams, then each work it commits should multiply by 10. If the team member builds a new feature, it should be helpful for all the other teams. Otherwise, the platform team is an addition, and in most cases it is then better to split the platform team and add them to all other teams, instead of being a separate team. Because the amount of communication needed is in most cases quadratic in relation to the number of teams. Reducing Cognitive Load The platform should take away cognitive load for all the teams it supports. By doing so, they will have more time to implement business requirements. Let's say a platform team provides Gitlab runners or Azure Agents where people can run their CI/CD code on. They should not need to know how the runners are scaled, or how the agents are updated. This takes away the need for that skill set in all the teams. Build a Community A platform team has a unique position. It is building something that all other developers probably can build as well. Some could do it even better than the platform team itself. For some platform teams, their ego sometimes comes in to play or just straight up refuses their help because they are not the team. But it is not the job of the platform team to build a product, but a platform where everyone can thrive and/or build on. So onboard the community on the platform! The Law of Diffusion applies to almost all companies. You will have the innovators that want to build it themselves and the early adopters that will voice their opinion, but will not build it themselves. Those two

2026-07-13 原文 →
AI 资讯

Claude Code Sends 33k Tokens Before Your Prompt; OpenCode Sends 7k

A new side-by-side measurement shows Claude Code ships roughly 33,000 tokens of system prompt, tool schemas, and scaffolding before your prompt even arrives — about 4.7x the ~7,000 tokens OpenCode sends on the same setup. The bigger cost surprise is next: Claude Code re-wrote up to 54x more prompt-cache tokens per session, and cache writes bill at a premium. How the test was run The benchmark (published by Systima) spliced a logging proxy between each harness and the model endpoint, capturing the exact request payload and the API's usage block. Both harnesses were pinned to the same conditions: Claude Code 2.1.207 and OpenCode 1.17.18 , both pointed at claude-sonnet-4-5 Fresh config directories, empty workspace, no MCP servers, no instruction files, permissions bypassed Tasks ranged from "reply with OK" (isolating fixed overhead) to a write-run-test-fix loop against FizzBuzz A zero-tools variant separated system-prompt weight from tool-schema weight The payload captures are exact; the only adjustment was subtracting a constant ~6,200-token gateway envelope that wrapped every request in the test setup. The fixed floor Harness Fixed overhead before your prompt Cache-write behavior Claude Code 2.1.207 ~33,000 tokens Re-wrote tens of thousands of cache tokens per run; up to 54x OpenCode OpenCode 1.17.18 ~7,000 tokens Byte-identical prefix each run; cached once, read back cheaply OpenCode's request prefix was byte-identical in every captured run, so it paid to cache its payload once per session and read it back for pennies. Claude Code re-wrote large amounts of prompt-cache tokens mid-session, run after run — and because cache writes bill at a premium, one usage dashboard climbs while the other stays flat. Where it piles on in real setups The harness floor is only the start. The benchmark added variables one at a time: A production repository's 72KB instruction file added an average of ~20,000 tokens to every request. Five modest MCP servers added 5,000–7,000 more. By th

2026-07-13 原文 →
AI 资讯

The Developer's Guide to Picking the Right Coding LLM at Scale

The Developer's Guide to Picking the Right Coding LLM at Scale Six months ago, I was staring at our monthly AI bill — $14,000 and climbing fast. We were using the "premium" model for everything, including trivial code completions. That night, I built a small internal benchmark to figure out which models actually earn their cost. What I learned reshaped how we think about AI tooling, vendor lock-in, and what "production-ready" really means. Here's the raw truth from my testing rig, what we shipped, and how we cut costs by 70% without touching output quality. Why I Stopped Trusting Default Recommendations Every vendor says their model is the best. Every benchmark site ranks things differently. Most "best of" lists are either sponsored or built on vibes. I needed numbers that matched my actual workflow: generating Python services, debugging JavaScript race conditions, implementing TypeScript algorithms, and reviewing Go for security. So I took ten models, threw identical prompts at them, and scored them myself. No vendor PR. No cherry-picked examples. Just the same five tasks, run the same way, scored on the same rubric. Here are the ten models I tested, with their output pricing per million tokens — because at scale, that's the metric that decides whether your AI strategy is viable or a margin killer. Model Provider Output $/M DeepSeek V4 Flash DeepSeek $0.25 DeepSeek Coder DeepSeek $0.25 Qwen3-Coder-30B Qwen $0.35 DeepSeek V4 Pro DeepSeek $0.78 DeepSeek-R1 DeepSeek $2.50 Kimi K2.5 Moonshot $3.00 GLM-5 Zhipu $1.92 Qwen3-32B Qwen $0.28 Hunyuan-Turbo Tencent $0.57 Ga-Standard GA Routing $0.20 Before you ask: yes, I tested against the originals. I also tested against Global API's unified routing layer, which lets you hit any of these through one endpoint. More on that later — it became the architectural decision that actually saved us. My Benchmark Methodology (No Marketing Fluff) I built five tasks that mirror what my engineers actually do every week. Not synthetic acad

2026-07-13 原文 →
AI 资讯

Why your integration tests pass but your message queue still double-processes in production

You know the story. You write a consumer, test it locally, deploy to staging, everything green. Then in production, the same message gets processed twice. Orders are duplicated. Notifications go out twice. Everyone blames "a race condition" and moves on. We spent the last few months formally verifying what actually happens when consumers crash with at-least-once delivery semantics. The short version: it doesn't matter which broker you use . The race condition is a property of the delivery contract, not a bug in the implementation. What we did Instead of writing more integration tests hoping to reproduce the timing, we wrote a formal specification of the consumer-broker interaction in TLA+ (the specification language by Leslie Lamport). The model checker exhaustively explores every possible interleaving of events — not just the ones you think to test. Then we took the counterexamples from the model and validated them in real running systems using Docker + Toxiproxy (network fault injection). If the math said "double-execution is possible", we confirmed it with actual containers. What we found We applied this pipeline to five different systems: Celery — ACK timeout + network blip: the model found the crash window at depth 9 (444 states). Chaos confirmed it: task executed twice. RabbitMQ — consumer stores result, drops connection before AMQP ACK. Same pattern. 108 states, depth 7. Chaos confirmed. NATS JetStream — consumer crashes after DB write, before ACK. 47 states, depth 6. Docker kill + Toxiproxy confirmed: duplicate execution. Apache Pulsar — batch consumer crashes between Process and SendAck. 21 states, depth 4. Chaos confirmed: 2 DB rows for 1 message. Kafka — consumer commits offset after processing. Crash between StoreResult and commitSync(). Same result: double execution. Every single time, the model predicted the collision and chaos confirmed it. The insight The spec is the same across all five systems. The variable names change, but the topology is identic

2026-07-13 原文 →
AI 资讯

How I Built a GeoGuessr Game for Super Mario Odyssey

I wanted to build something different from the usual fan project. Instead of recreating gameplay, I asked a different question: How well do players actually remember the worlds of Super Mario Odyssey? The result is OdysseyGuessr, a browser game inspired by GeoGuessr. Players receive a screenshot from one of the game's Kingdoms and must place a marker on the correct location. Every meter counts. The project includes: Browser-based gameplay Single-player with customizable rounds Real-time multiplayer with a 5,000 HP battle system Community-submitted locations Admin moderation tools One of the biggest challenges wasn't the gameplay—it was collecting interesting locations that were difficult but still fair. Watching players confidently choose the wrong cliff or rooftop has been surprisingly entertaining. If you're interested in browser games or Nintendo fan projects, I'd love to hear your feedback. Play here: https://odyssey-guessr.lovable.app OdysseyGuessr is an unofficial fan project and is not affiliated with Nintendo.

2026-07-13 原文 →
AI 资讯

The One DevOps Metric Every Solo Developer Ignores

What’s up everyone! Back again for my daily drop. We talk a lot about deployment frequency and lead time for changes, but if you're a solo dev or part of a small team building something like LaunchAlly , there’s one metric that rules them all: Time to Recovery (TTR) from a bad push. When you're marketing, coding, and handling support all at once, a broken main branch is a massive bottleneck. Here is my quick tip for today: Invest 20 minutes into setting up strict automated rollbacks . If a deployment fails health checks, let the system revert it instantly without your intervention. Spend lots of hours working today...happy to go to bed now:) What’s your go-to strategy for handling failed deployments on the fly?

2026-07-13 原文 →
开发者

Integrating .NET GC in your C++ application

.NET appears as large monolith where everything is working like a magic. But it is not, and even GC can be used outside of .NET runtime (obviously with caveat). When I was a kid, I always love disassemble things, to see how they works. Not always toys survive that exercise. But thanks to source control, .NET GC is safe from my hands. Hopefully lot of people like me, and will enjoy tearing down large system into pieces. submitted by /u/kant2002 [link] [留言]

2026-07-13 原文 →
AI 资讯

Passion Edition

Submission: Edu-Insight Assistant What I build I built the Edu-Insight Assistant, a tool designed for educators to bridge the gap between complex school management data and actionable insights. It allows teachers to query students performance data using natural language, turning educational evaluation into a conversation rather than a manual data-processing task. Demo 🔗 Link: Passion-challenge How I Built It I utilized Next.js for a responsive, performant frontend and hooked it up to Google Gemini 3.5 API. The core logic involves a server-side API route that takes a teacher's natural language questions, prompt Gemini to generate the necessary SQL, and execute that query against a database. This architecture makes data exploration accessible to non-technical educators. Prize Categories: - Best Use of Google AI : Leveraged Gemini 3.5 Flash for natural language-to-SQL translation and result interpretation. - Best Use of Snowflake: Designed with an extensible data layer ready for production-scale analytical workloads in Snowflake.

2026-07-12 原文 →
AI 资讯

skip에서 partition overwrite로: business_date 재처리를 Iceberg로 다시 표현하기

skip에서 partition overwrite로: business_date 재처리를 Iceberg로 다시 표현하기 이전 글에서는 같은 source_hash 가 다시 들어왔을 때 기존 successful run을 재사용하는 idempotency를 다뤘다. 하지만 재처리에는 두 종류가 있다. 1. 같은 입력이 다시 들어온 경우 -> skip이 맞다. 2. 같은 business_date의 정정 입력이 들어온 경우 -> skip하면 안 된다. -> 같은 날짜의 gold 결과를 중복 없이 교체해야 한다. manufacturing-data-platform-mini 의 B5 slice는 두 번째 문제를 아주 작게 다룬다. 전체 Spark pipeline을 만든 것이 아니다. gold_daily_metrics Iceberg table 하나를 local Spark에서 만들고, business_date partition overwrite와 snapshot evidence만 검증했다. Scenario 이미 아래 gold row가 있다. business_date=2026-06-29 plant-a / line-1 / gearbox-a units_produced=120 defect_count=3 나중에 같은 business_date=2026-06-29 에 대한 정정 source가 들어온다. 운영자가 원하는 것은 append가 아니다. 원하지 않는 상태: 2026-06-29 old row 2026-06-29 corrected row -> 같은 날짜 결과가 중복됨 원하는 상태: 2026-06-29 corrected row만 남음 2026-06-30 같은 다른 날짜 partition은 그대로 유지됨 재처리 전후 snapshot evidence가 남음 그래서 이 slice의 질문은 이렇다. 같은 business_date의 정정 source를 처리할 때, gold table에서 해당 날짜 partition만 중복 없이 교체하고, 어떤 run이 어떤 Iceberg snapshot을 만들었는지 남길 수 있는가? Decision Pressure Slice1의 CSV pipeline은 already-successful source를 안전하게 skip할 수 있다. dataset_id + business_date + source_hash 이 key가 같으면 같은 입력이다. 다시 계산해도 같은 결과이므로 기존 run을 재사용한다. 하지만 source_hash 가 달라졌다면 의미가 다르다. same business_date different source_hash 이건 retry가 아니라 correction이다. CSV run-folder 방식에서는 새 run output을 만들 수는 있지만, "현재 gold table에서 해당 날짜를 원자적으로 교체한다"는 table-level 의미가 약하다. Iceberg를 붙이는 이유는 여기 있다. source_hash -> 같은 입력인지 판단하는 idempotency key business_date partition -> 정정 시 교체할 gold table 범위 snapshot_id -> table commit의 evidence 즉 Spark/Iceberg는 도구 이름을 추가하려고 붙인 것이 아니라, 재처리 상태 전이를 더 명확히 표현하기 위해 붙였다. Options Option 장점 문제 판단 same source면 항상 재계산 단순함 retry 때 불필요한 commit이 계속 생김 제외 corrected source를 append 구현 쉬움 같은 날짜 gold row가 중복될 수 있음 제외 whole-table overwrite 단순함 다른 날짜 partition까지 지울 위험 제외 business_date partition overwrite correction 범위가 명확함 Spark/Iceberg 설정과 test가 필요 선택 MERGE/upsert 강력함 이번 skeleton에 과함 backlog 이번 구현은 DataFrameWriterV2.overwritePartitions() 를 사용했다. corrected_d

2026-07-12 原文 →
AI 资讯

wide CSV 여러 개를 EAV로 모아 gold mart 만들기

wide CSV 여러 개를 EAV로 모아 gold mart 만들기 현실의 데이터 소스는 한 가지 모양으로 오지 않는다. 같은 의미의 값도 어떤 파일에서는 생산수량 , 다른 파일에서는 units , 또 다른 파일에서는 made 로 올 수 있다. 온도도 어떤 곳은 섭씨, 어떤 곳은 화씨일 수 있다. 이걸 매번 pipeline code에 if source == ... 로 박기 시작하면 source가 늘 때마다 코드가 지저분해진다. manufacturing-data-platform-mini 의 EAV mini slice는 이 문제를 작게 다룬다. 여러 wide CSV를 mapping config로 표준 attribute에 맞춘 뒤, EAV long format으로 모으고, 다시 gold metric mart로 pivot/aggregate한다. 데이터는 모두 synthetic이고, 회사 코드·고객 데이터·실제 schema는 쓰지 않았다. 1. Scenario 서로 다른 공장/라인/벤더에서 비슷한 제조 지표 파일이 들어온다. 예: plant_a.csv: 설비ID, 생산수량, 불량수, 온도C, 압력kPa plant_b.csv: machine_id, output_units, defects, temp_f, pressure_bar vendor_d.csv: unit_name, made, scrap, deg_c, kpa 비즈니스적으로는 같은 지표를 보고 싶다. units_produced defect_count temperature_c pressure_kpa 문제는 source마다 column name과 unit이 다르다는 점이다. 2. Decision Pressure 단순 구현은 source마다 코드를 늘린다. if source == "plant_a": 생산수량을 units_produced로 읽는다 if source == "plant_b": output_units를 units_produced로 읽는다 temp_f를 섭씨로 변환한다 if source == "vendor_d": made를 units_produced로 읽는다 이 방식은 작게는 빨라 보이지만 source가 늘수록 문제가 된다. 새 파일 형식마다 pipeline code를 고쳐야 한다. column mapping과 transform logic이 섞인다. unit conversion이 흩어진다. quality check가 source별로 갈라진다. gold mart grain을 설명하기 어려워진다. 그래서 mapping은 config로 빼고, pipeline은 표준 attribute를 처리하게 만들었다. 3. Options option result risk source별 hard-coded parser 처음엔 빠름 source가 늘 때 code change 반복 모든 source를 wide table 하나로 합치기 보기 쉬움 sparse/heterogeneous column 폭발 EAV long format 이종 attribute를 표준 형태로 모음 pivot/quality 설계가 필요 full mapping DSL/rules engine 유연함 mini project에는 과함 이 프로젝트의 선택은 단순한 JSON mapping + EAV long + gold pivot이다. 4. Decision 각 source는 JSON config로 자신의 column을 표준 attribute에 매핑한다. source column -> standard attribute output_units -> units_produced temp_f -> temperature_c with f_to_c pressure_bar -> pressure_kpa with bar_to_kpa pipeline 흐름: wide CSVs -> mapping configs -> EAV long rows -> gold entity_daily_metrics -> quality checks -> catalog/lineage EAV row의 핵심 shape: entity_id business_date attribute value v

2026-07-12 原文 →