今日已更新 329 条资讯 | 累计 40774 条内容
关于我们

标签:#trans

找到 470 篇相关文章

AI 资讯

How a Baseten Engineer Traced 7 Years of Attention Mechanism Evolution -- From GPT-2 to Kimi K3, in Runable PyTorch

Last week, a Baseten inference engineer who goes by @waterloo_intern published a technical blog post titled "22,580: From GPT-2 to Kimi K3, Explained." It hit 2.4 million views in days. He didn't write a press release. He wrote runnable PyTorch code — starting from GPT-2's attention block, stepping through every architectural change, explaining one problem and one cost per iteration. It's the best transformer lineage explanation I've seen. I devoured his post, then cross-checked the key claims against 5 original papers. Here's the full picture. The 22,580x Number In February 2019, OpenAI released GPT-2 — 124M parameters. Seven years later, Moonshot AI open-sourced Kimi K3 — 2.8T parameters. You could fit 22,580 GPT-2s inside one Kimi K3 . But this isn't a "throw more compute at it" story. It's a story about how we store, update, and retrieve memory . Starting Point: GPT-2 class Block ( nn . Module ): def forward ( self , x ): x = x + self . attn ( self . ln_1 ( x )) x = x + self . mlp ( self . ln_2 ( x )) return x Every time the model generates a new token, it recomputes Q, K, V projections for all historical tokens, then runs an O(N²) softmax attention. K and V from tokens 1 through N-1? Thrown away. Token N+1 arrives? Recompute everything. That's why KV Cache was invented. KV Cache: Store It, Don't Recompute Simple idea: cache the already-computed keys and values. For the next token, new Q only needs one dot product against the cached K. Problem solved — but a new one created. KV cache grows linearly with sequence length. At 1M tokens × d_model × layers, that's dozens of GB of VRAM. Every decoding step reads all of it from HBM. The bottleneck isn't compute. It's memory bandwidth. This is the key to understanding every improvement that follows. Linear Attention: Fixed-Size Memory Can we compress O(N²D) into O(ND²)? The idea: replace softmax with a feature map. # Standard softmax (must materialize N×N first) attention = softmax(QKᵀ / √d) × V # Linear attention (fold

2026-07-31 原文 →
AI 资讯

Presentation: Parting the Clouds: The Rise of Disaggregated Systems

Murat Demirbas discusses the shift toward disaggregated cloud database architectures driven by cloud economics. He explains how decoupling compute from storage enables elastic scaling, cost efficiency, and fault isolation. He shares how classical Paxos roles foreshadowed disaggregation, while analyzing network tradeoffs, shared-memory evolution, and self-assembling database designs. By Murat Demirbas

2026-07-30 原文 →
AI 资讯

The Ferrari Luce has at least 500 fans

Just two months after a bumpy launch earlier this year, Ferrari has reportedly already hit its 2026 sales goal for its polarizing, Jony Ive-designed Luce EV. Ahead of Ferrari announcing its quarterly earnings on Thursday, the Financial Times reports that Ferrari was aiming to sell "just under 500" Luce units this year, and reached that […]

2026-07-29 原文 →
AI 资讯

Presentation: Getting Rid of LeetCode Interviews in the World of AI

Daniel Doubrovkine explains why traditional LeetCode whiteboard interviews fail to evaluate senior engineering talent. He discusses his own experience bombing basic algorithm tests despite decades of leadership, and shares actionable frameworks for redefining the interview loop. Discover how evaluating human judgment, system design, and hands-on AI collaboration yields far better hiring signals. By Daniel Doubrovkine

2026-07-29 原文 →
AI 资讯

Presentation: The Future of Engineering: Mindsets That Matter When Code Isn’t Enough

Ben Greene discusses how software engineers can adapt and thrive in an era of rapid AI code automation. Drawing on his startup experience, he explains key mindsets like starting simple, maintaining code comprehension, attacking hard problems first, and focusing on customer impact. He shares why human empathy, agency, and practical problem-solving remain irreplaceable when code is automated. By Ben Greene

2026-07-28 原文 →