标签：#ci

Best-of-K eval selection bias: pick the highest-scoring config from K candidates on one eval set and that observed score is biased up. It reports the expected maximum of K noisy estimates, which beats the field mean whenever K exceeds one. The bias appears even when all K configs are truly equal, grows with K, and shrinks with n. Here is the version that bites you. Your bake-off ran a batch of prompts against one eval set, the top one came out ahead, and you shipped it. In production it does worse. That drop reads like bad luck, or drift, or a bad week. It is none of those. It is a number you could have computed before you shipped, and it gets larger the more candidates you tried. I ran a small script to make the gap concrete. Eight configs, one hundred eval items, and here is the catch: I made all eight configs truly identical , every one a fair coin at 50%. There is no real best. Nothing to tune. Then I let selection pick a winner anyway: config 0: 47/100 = 47.0% config 1: 50/100 = 50.0% config 2: 52/100 = 52.0% config 3: 52/100 = 52.0% config 4: 50/100 = 50.0% config 5: 50/100 = 50.0% config 6: 54/100 = 54.0% <- selected winner (argmax) config 7: 44/100 = 44.0% config 6: 54.0% (k=54 n=100 SE=4.98) config 2: 52.0% (k=52 n=100 SE=5.00) RANK: INDISTINGUISHABLE - gap 2.00 pp against 7.06 pooled SE = 0.28 SE < 2.0. Ranking "config 6" above "config 2" is NOT allowed. Held-out the winner on a fresh 100 items: 48/100 = 48.0%. Config 6 wins the bake-off at 54.0%. Then I asked the same eval-guard I use in the McNemar and rule-of-three pieces to rank config 6 against the runner-up. It refused: the gap is 0.28 SE, far under the two-SE bar, so INDISTINGUISHABLE . The ranker would not call config 6 the best. Selection did. And on a fresh held-out set the 54.0% falls back to 48.0%, toward the true 50% it was always going to be. TL;DR Picking the best of K configs by observed pass rate reports the expected maximum of K noisy estimates. The max of K exceeds the field mean, strict

The cyclospora diarrhea outbreak isn't an isolated incident. It's part of a pattern of leafy greens wreaking digestive havoc.

Several federal agencies are contributing toward the interdisciplinary research efforts.

A closely watched social media addiction lawsuit that had been set to go to trial next week has been dropped after the plaintiff voluntarily dismissed his claims against Meta, leaving none of the major tech companies facing trial in the case.

As access to Anthropic’s and OpenAI’s frontier models becomes more restricted, Chinese labs are pitching their open-source alternatives as stable, accessible, and increasingly capable.

Introduction: Strategic Entry into Cybersecurity The cybersecurity domain operates as a dynamically evolving ecosystem, characterized by the rapid emergence of specialized disciplines that outpace the ability of newcomers to systematically map them. From web security to cloud infrastructure, each subdomain demands a distinct integration of technical proficiency and strategic foresight. For entrants, this duality presents both opportunity and risk. While the diversity of career paths is expansive, it concurrently induces a decision paralysis —a condition where the proliferation of options dilutes focus and impedes progression. Consider the scenario of a novice equipped with foundational competencies in Linux, Python, and network fundamentals, now confronted with a spectrum of specializations: web security, binary exploitation, malware analysis, SOC operations, and cloud security. Each pathway entails a unique learning curve and industry relevance. The critical risk lies not in selecting an inherently "incorrect" path but in the suboptimal allocation of time within a field where technological obsolescence outpaces learning cycles. Cloud security exemplifies this dynamic. The transition to cloud-native architectures has introduced a critical stress point in cybersecurity frameworks. Traditional perimeter defenses, such as firewalls and VPNs, are increasingly inadequate for distributed systems. Misconfigurations in platforms like AWS or Azure—often stemming from human error or incomplete automation scripts —account for over 80% of cloud breaches (IBM Cloud Security Index, 2023). This is not a theoretical vulnerability but a causal mechanism : misconfiguration (internal process) → breach (impact) → data exfiltration (observable effect) . In contrast, niche domains like binary exploitation, while foundational for understanding low-level vulnerabilities, exhibit a diminishing practical application. Modern software increasingly leverages memory-safe languages (e.g., Rust, G

Yope, a fast-growing social app focused on private groups of friends and family, has raised $12.3 million in seed funding. Instead of chasing creators and algorithmic feeds, the startup is betting that the future of social networking lies in small, private communities powered by messaging, photo sharing, and AI features designed to strengthen real-world relationships.

I wired an AI code reviewer into CI, felt clever, and then looked at the bill after a busy week of PRs. Every push, every commit, sending full diffs to a big frontier model. It added up faster than I expected, and most of what it was reviewing was formatting changes and README edits that did not need a genius reading them. So I rebuilt it as two tiers, and the cost dropped hard without losing the catches that mattered. The idea is simple. Do not send everything to the expensive model. Use a cheap model as a bouncer that decides what is even worth escalating, and reserve the big model (with caching turned on) for the files that could actually hurt you. Tier one: a cheap triage pass The first tier looks at the diff and answers one question: is anything here risky enough to justify a real review? That is a classification task, and classification does not need a frontier model. I run this tier on a small local model. On my own machine that is Ollama with qwen2.5-coder, but in CI it can be a self-hosted small model or the cheapest cloud tier your provider offers. The job is triage, not judgment. The triage prompt is deliberately narrow. I do not ask it to review. I ask it to route: You are a triage filter for code review. For the diff below, output JSON only: { "risk": "low" | "high", "reason": " <one short phrase > " } Mark "high" if the diff touches: auth, access control, money/token math, cryptography, SQL or shell string building, deserialization, file paths, network calls, or anything that changes who can do what. Mark "low" for formatting, comments, docs, tests, renames, and config bumps. DIFF: <diff here > A small model is perfectly good at "does this touch auth or money." When it says low, CI posts a one-line "no high-risk changes detected" and stops. No expensive call. In practice that shortcuts the majority of PRs, because most PRs really are docs, tests, and plumbing. Tier two: the big model, but only on the risky files, with caching When tier one says high, t

TL;DR: Use per-test coverage data to build a reverse map (file → tests that touch it). Git diff + map lookup = run only relevant tests. 43min → 4min CI time. The Problem Everyone Has Your test suite runs for 45 minutes. You change one file. Should you: Run all 2,000 tests? (Safe but slow) Run tests with matching filenames? (Fast but broken) Manually tag tests? (Gets stale immediately) We tried all three. They all failed. The Insight: Stop Guessing, Start Measuring What if we just measured what each test actually executes? Coverage tools have done this for years: { "src/MyComponent.js" : { "statements" : { "0" : 5 , "1" : 12 , "2" : 0 } } } Key insight: Coverage tools report per test run, not per test file. All we needed: save coverage after each individual test. The Architecture (5 Steps) Step 1: Instrument the Code // Build config (webpack/vite/rollup/etc) export default { plugins : [ process . env . COLLECT_COVERAGE && coveragePlugin ({ include : ' src/**/* ' , exclude : [ ' **/*.test.* ' ] }) ] } Works with: Istanbul (JS), coverage.py (Python), SimpleCov (Ruby), JaCoCo (Java) Step 2: Collect Coverage Per Test // Test framework afterEach hook afterEach ( async () => { const coverage = getCoverageData (); saveCoverage ( `coverage- ${ testName } .json` , coverage ); }); Step 3: Build the Reverse Map const map = {} ; for (const coverageFile of allCoverageFiles()) { const testName = extractTestName(coverageFile); const coverage = JSON.parse(read(coverageFile)); map [ testName ] = [] ; for (const [ sourceFile , data ] of Object.entries(coverage)) { if (wasExecuted(data)) { map [ testName ] .push(sourceFile); } } } Example output: { "tests/user-profile.test.js" : [ "src/components/Profile.jsx" , "src/components/Card.jsx" , "src/utils/formatters.js" ] } The magic: The test told us what it executed. No parsing, no guessing. Step 4: Detect Tests CHANGED = $( git diff origin/main...HEAD --name-only ) TESTS = $( lookupTestsInMap " $CHANGED " ) npm test $TESTS Step 5: Auto-Up

A mirror that moves mid-photon could release a shower of new photons.

Sign up with a username.

WhatsApp on Apple CarPlay and Android Auto will let users hear and reply to messages, make calls, view their call history, and access favorite contacts hands-free and right from their car's screen.

Cut the noise. Keep the stack traces. CI logs are mostly package installs. The failure is a few dozen lines buried under thousands of Downloading… lines. Scrolling for the real stack trace is a tax paid on every red build. logsnip is a zero-dependency Python CLI that extracts those failure regions and collapses the rest. Install pip install git+https://github.com/SybilGambleyyu/logsnip.git # or the whole toolkit: curl -fsSL https://raw.githubusercontent.com/SybilGambleyyu/devkit/main/install.sh | bash One-liners # Last failure from a GitHub Actions run gh run view --log-failed | logsnip --last # Headlines only logsnip ci-full.log --summary # Safe to paste into an AI assistant gh run view --log-failed | logsnip --last | redactx What it matches Built-in patterns cover pytest E lines and AssertionError , npm ERR! , rustc error[E…] , TypeScript error TS… , GitHub Actions ##[error] , make failures, and common exit-code messages. Stack frames after a hit are pulled in automatically. Design No network, no config files, no dependencies — pipe-friendly --check exits 1 when error-like lines appear (CI gate) --json for machines; --summary for humans in a hurry Pairs with redactx before anything leaves your machine Source: github.com/SybilGambleyyu/logsnip · MIT

Guidelines, not rules. Here's the difference — and why it matters. What is SOLID? SOLID is a set of software design guidelines — not hard rules, but principles that guide how we organize our code. The goal is simple: as your codebase grows and your team scales, things should get easier to change, not harder. SOLID is what makes that possible. Five principles. One goal. Let's walk through each one with real code. S — Single Responsibility Principle A class, function, or method should have one and only one reason to change. The Violation class Bird : def __init__ ( self , name : str , bird_type : str ): self . name = name self . bird_type = bird_type def make_sound ( self ): # two jobs — deciding the type AND making the sound if self . bird_type == " parrot " : print ( " Squawk! " ) elif self . bird_type == " eagle " : print ( " Screech! " ) elif self . bird_type == " owl " : print ( " Hoot! " ) else : print ( " ... " ) make_sound() has two responsibilities — deciding which bird type it is AND making the sound. That's two reasons to change. Add a new bird? Touch make_sound() . Change how sounds work? Touch make_sound() again. Two different reasons, one method. SRP violated. The Fix from abc import ABC , abstractmethod class Bird ( ABC ): def __init__ ( self , name : str ): self . name = name @abstractmethod def make_sound ( self ): pass class Parrot ( Bird ): def make_sound ( self ): print ( " Squawk! " ) class Eagle ( Bird ): def make_sound ( self ): print ( " Screech! " ) class Owl ( Bird ): def make_sound ( self ): print ( " Hoot! " ) # Usage birds = [ Parrot ( " Polly " ), Eagle ( " Sam " ), Owl ( " Oliver " )] for bird in birds : bird . make_sound () Now each class has one responsibility. Parrot.make_sound() only changes if parrots change how they sound. Nothing else touches it. O — Open/Closed Principle A class should be open for extension but closed for modification. SRP and OCP go hand in hand. When you fixed SRP in the Bird example above — you also fixed OCP.

Jake Mannix discusses moving AI agents past chaotic "1970s BASIC" architectures. He shares how implementing an intermediate protocol layer allows engineering leaders to build versioned, encapsulated "virtual tools." This design enables interface mapping, dynamic schema projection, and runtime taint tracking to proactively eliminate data exfiltration risks without slowing velocity. By Jake Mannix

Harrowing story of an identity theft victim. Yes, the person made a mistake—they gave the scammer a two-factor authentication code that allowed the scammer to take over their email address. But the real story here is how, for many of us, the security of most of our accounts hangs on the security of our email accounts.

In the face of backlash to concerns the AI boom will increase consumer electricity bills, the largest utility companies and data center developers in the US are now promising to do something about it. The Wall Street Journal reports that nearly 200 organizations have signed President Donald Trump's "rate payer protection pledge" that's meant to […]

Structural engineers evaluate ASCE 7 Chapter 2 load combinations on nearly every design. The factors themselves are not hard — the failure mode is a spreadsheet cell that someone “improved” six months ago, or a notebook that only checks one combination. loadcomb is a small, dependency-free Python library and CLI that evaluates the basic LRFD and ASD combination sets on scalar load effects you already have from analysis. pip install loadcomb loadcomb --D 120 --L 80 --S 40 --W 55 --method LRFD from loadcomb import LoadCase , governing g = governing ( LoadCase ( D = 120 , L = 80 , S = 40 , W = 55 ), method = " LRFD " ) print ( g . id , g . value , g . formula ) What it handles Basic LRFD and ASD combinations from ASCE 7 Chapter 2 (Lr or S or R) resolved to the roof variable with largest absolute effect Automatic ± envelope for wind and earthquake Governing combination by absolute value What it is not Not FEA, not member design, not every exception in the standard. Confirm the ASCE 7 edition and local amendments for your project. Links pip install loadcomb GitHub PyPI Blog: sybilgambleyyu.github.io/posts/loadcomb.html MIT licensed.

The rule of three for evals says zero failures in N runs is a count, not a rate. With 0 failures in N independent runs, the exact 95% upper bound on the true failure rate is 1 - 0.05^(1/N) , which 3/N approximates. After 100 clean runs you still cannot rule out a 2.95% rate, about 1 in 34. Here is the reading that bites you. Your eval harness runs the agent 100 times, prints "0 failures," and the tile goes green. Someone screenshots it into the launch thread. The unspoken translation is "the failure rate is zero." It is not what the data says. I wrote a small script to make the gap concrete, so I ran a real gate over 200 deterministic agent runs first, counted honestly, and got the dashboard everyone trusts: gate: spend<=budget over 200 deterministic agent runs observed failures: 0 (distinct scenarios: 200) naive point rate : 0.00% binomial SE: 0.00 pp naive 95% CI : [0.00%, 0.00%] <- zero width: false certainty Look at the standard error. For a zero count the binomial SE is sqrt(0*1/200) , which is exactly 0, so the naive 95% interval collapses to [0.00%, 0.00%] . A zero-width confidence interval. The math is telling you it is completely certain, from 200 samples, that the true rate is precisely zero. That is obviously wrong, and it is the exact shape of every "all green" board I have ever trusted too much. TL;DR "0 failures in N runs" is an observed count, not a rate. The naive binomial SE of a zero count is 0, which is why a green board looks like proof and isn't. The honest number is the one-sided upper bound. With 0 failures in N runs, the 95% upper confidence limit on the true failure rate is 1 - 0.05^(1/N) . The rule of three, 3/N , approximates it and rounds the risk slightly up. At N=30 the bound is 9.50% (about 1 in 11). At N=100 it is 2.95% (1 in 34). At N=1000 it is 0.30% (1 in 334). Zero failures in 30 runs is compatible with a 1-in-11 true failure rate. To rule out a 0.1% rate at 95% you need about 2995 clean runs, not 50. "We ran it fifty times" and "

The cybersecurity-focused models, including GPT-5.6 Sol, broke out of a testing sandbox, exploited a zero-day, and gained access to the open internet to pull off the attack.