标签：#Performance

Environment And Situation Control room of an apartment Number of installed product : 3 (PC-based NVR, dual-LAN supported) Remote support : X (I actually went to the site and diagnosed) Reported Issue In viewer, when user changes play speed while playing back the recorded data, it randomly plays the data in hyper speed(almost 30x~60x) For example: 4x play means 4 seconds in video per a second. But in the site, it played 30~60 seconds per a seconds, showing the video stutturing. Diagnosis Checked the overall environment. System(CPU / RAM usage), network environment(bandwidth), resoulution, stream configurations, etc. -> Nothing suspicious. Some of the installed cameras had unusual fps and gop values Normally, fps and gop values are set to be equal(for exmaple, if fps is 30 then gop is also 30 so that iframe can appear every second) But the cameras' set up values were fps 15, gop 60(iframe per 4 seconds) Assumption Somehow the viewer keeps failing to find iframe to play. And it's maybe because iframe appears with a long gap. Quick note: iframe is kind of a key-frame. Since the viewer starts decoding from an iframe, it's necessary when it comes to playback. What I Tried Set all the cameras' gop value to 15(same as fps) Result Ran a test with data before changing the gop values and after. During interval before changing the gop, the issue occurred almost every time I tried. But after chaning the gop, the issue no longer occurred. Concolusion The issue was triggered by large GOP value (GOP 60 with FPS 15). With only one iframe every four seconds, the viewer sometimes failed to find an appropriate iframe after changing the playback speed, causing abnormal playback behavior. According to the viewer developer, this is likely related to the viewer's iframe searching logic, which is still under investigation. Keep This In Mind Check camera settings(especially gop and fps) first when it comes to playback issue. Always check before/after data to confirm assumption.

As developers, data engineers, or analysts, we’ve all been there: you download a massive database export, a logging stack dump, or a transaction archive, only to find it's a multi-gigabyte JSON file. You try to import it into a spreadsheet or run it through a standard online converter, and boom—your browser tab freezes, crashes, or shows the dreaded "Out of Memory" screen. Even worse, if you try to use standard cloud-based online tools, you might have to wait for a 500MB upload to complete, only to hit a rigid file-size cap or, worse, compromise sensitive data privacy by uploading corporate logs or database records to a third-party server. In this guide, we will explore: Why large JSON files crash standard parsers (the V8 heap limit problem). How streaming architectures solve this by reading data chunk-by-chunk. NDJSON (JSON Lines) vs. JSON Arrays and how to stream them. A browser-native, 100% offline tool to convert large JSON to CSV instantly: Parsify's Large JSON Stream Converter . How to implement your own basic browser-based JSON streaming parser in JavaScript. 1. The Anatomy of a Memory Crash (Why JSON.parse Fails) If you are using JavaScript or Node.js, the simplest way to read and parse a JSON file is to load the file into memory and run JSON.parse(). const fs = require ( ' fs ' ); // Naive approach: Will crash on a 1GB+ file fs . readFile ( ' database-dump.json ' , ' utf8 ' , ( err , data ) => { if ( err ) throw err ; // POINT OF FAILURE: V8 Heap Out of Memory const records = JSON . parse ( data ); records . forEach ( record => { // Process record... }); }); This works fine for small config files. But once your JSON file reaches 100MB, 500MB, or 1GB+, this approach is guaranteed to trigger a fatal crash: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory Why does this happen? The String Duplication Overhead: When you load a 1GB file into memory, you first allocate ~1GB of RAM for the raw text string. The

Slack has outlined how its AI serving infrastructure evolved through four distinct phases, moving from a self-managed Amazon SageMaker deployment to a multi-cloud architecture spanning AWS Bedrock and Google Cloud Vertex AI. By Matt Foster

We have all been there: a meeting starts, and suddenly your phone rings. I built Muffle to automate silent profiles, but the biggest hurdle wasn't the UI—it was making sure the app didn't destroy the user's battery while monitoring GPS coordinates. The Trap of Continuous Location Updates Early prototypes used LocationManager with frequent updates. This is the fastest way to get your app uninstalled. Keeping the GPS radio active in the background forces the device to wake the CPU constantly, leading to significant battery drain. To solve this, I moved away from active polling and shifted to the GeofencingClient API. Leveraging GeofencingClient for Passive Monitoring Instead of calculating distance from a point every few seconds, I transitioned to system-level geofencing. By defining circular regions around locations like the office or a mosque, the OS handles the monitoring at the hardware abstraction layer. kotlin val geofencingRequest = GeofencingRequest.Builder() .setInitialTrigger(GeofencingRequest.INITIAL_TRIGGER_ENTER) .addGeofences(geofenceList) .build() This approach allows the OS to do the heavy lifting. The app stays in a dormant state until the location provider signals a transition. The kernel only wakes the app when the device enters or exits the radius. The Trade-off: Precision vs. Power Using GeofencingClient means accepting a slightly slower trigger time compared to raw GPS polling. Sometimes, there is a delay of a few seconds as the device wakes from a deep sleep state. For a utility like Muffle, this is a fair trade-off. Users prefer their phone to silence five seconds after entering a building rather than finding their battery dead by noon. To mitigate the delay, I combined geofencing with a secondary intent service that performs a final check once the geofence trigger hits, ensuring that we aren't just reacting to a momentary GPS jitter. Final Thoughts By offloading the monitoring to the platform's native geofencing API, I was able to keep Muffle

A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the Photon vectorized engine on top to squeeze out maximum performance. Table of Contents Why Query Optimization Matters The Catalyst Optimizer Pipeline Stage 1: Parsing — From SQL to Unresolved Logical Plan Stage 2: Analysis — Binding to the Catalog Stage 3: Logical Optimization — Rule-Based Rewrites Stage 4: Physical Planning — Strategies and Cost Models Adaptive Query Execution (AQE) The Photon Engine Reading Explain Plans Tuning Reference Table References Why Query Optimization Matters A Spark query written by a human and a Spark query executed by the engine are often very different things. The gap between them — the optimization — is what separates a job that runs in 3 minutes from one that runs in 3 hours on identical hardware. Databricks compounds Spark's native Catalyst optimizer with two additional layers: Adaptive Query Execution (AQE) — re-optimizes the query at runtime using actual statistics collected mid-job Photon — a C++ vectorized execution engine that replaces the JVM-based Spark executor for eligible operators Understanding all three lets you write queries that cooperate with the engine rather than fight it. The Catalyst Optimizer Pipeline Catalyst is Spark's rule-based and cost-based query optimizer. Every query — whether written in SQL, DataFrame API, or Dataset API — passes through the same four-stage pipeline before a single byte of data is read. Stage 1: Parsing — From SQL to Unresolved Logical Plan # ── Catalyst Stage 1: Parsing ───────────────────────────────────────────────── # Spark uses ANTLR4 to parse SQL into an Abstract Syntax Tree (AST). # At this point column names are NOT validated — the plan is "unresolved". from pyspark.sql import SparkSession spark = SparkSession . builder . appName ( " catalyst-demo " ). getOrCreate () # Both of these produce identical internal representations df_api = ( spark .

Performance work has a reputation for being glamorous — the heroic "we cut latency by 80%" story. Most days it's not that. Most days it's a janitorial pass: you go looking for the queries you're firing without realizing it, and you quietly delete them. That was today. One sustained sweep across an app and the package that backs it, chasing the same theme everywhere: stop asking the database for things you don't use. Let me walk through the patterns, because they generalize to any Laravel app of a certain age. First, make the invisible visible You can't fix N+1s you can't see. The first move was wiring up an N+1 detector in the local/dev environment only — beyondcode/laravel-query-detector . It hooks into the request lifecycle, watches your Eloquent relationship loads, and screams (in the console, or as an exception if you want it strict) when it spots the classic loop-and-lazy-load pattern. The "dev-only" part matters. You never want a query detector running in production — it adds overhead and it's a developer aid, not a runtime guard. So it goes in behind an environment check, registered only when the app isn't in production: public function register (): void { if ( $this -> app -> environment ( 'local' , 'testing' )) { $this -> app -> register ( \BeyondCode\QueryDetector\QueryDetectorServiceProvider :: class ); } } Think of it like a smoke detector you only arm while you're cooking. It's noisy by design — that's the point. The noise is a to-do list. Eager loads you don't actually use are just N+1s wearing a disguise Here's the counterintuitive one. We're all trained to fix N+1s by adding with() . But the opposite bug is just as common and almost never gets caught: you eager-load a relationship, and then... never touch it in the view. Index screens are the worst offenders. Someone builds a listing, eager-loads creator and approver so the table can show names, then a redesign drops those columns — but the with(['creator', 'approver']) stays. Now every page load hyd

Data-Oriented Design in C#: Why Objects Are Slowing You Down In my previous article, we talked about starving the Garbage Collector by moving away from heap-allocated class types and leaning heavily into struct , Span<T> , and ArrayPool<T> . That’s a critical first step, but it only solves half the problem. You’ve stopped the GC from pausing your app, but you might still be leaving massive amounts of CPU performance on the table. Why? Because of how your data is structured. It’s time to talk about Data-Oriented Design (DoD) . The Object-Oriented Trap We are taught from day one to model our code after the real world. If you are building a social network graph, you might write something like this: public class UserNode { public int Id { get ; set ; } public string Name { get ; set ; } public List < Edge > Connections { get ; set ; } } public class Edge { public UserNode Target { get ; set ; } public int Weight { get ; set ; } } This makes perfect logical sense. A user has connections, and those connections point to other users. But modern CPUs don't care about your logical models. A CPU only cares about reading data from memory into its L1/L2 caches as fast as possible. When a CPU reads a byte from RAM, it doesn't just read that one byte; it pulls a whole 64-byte "cache line" under the assumption that you will probably want the neighboring bytes next. When you loop through a List<UserNode> , traversing from object to object, you are jumping randomly across the heap. The CPU pulls a cache line, reads your data, and then has to go fetch a completely different block of RAM for the next node. This is called pointer chasing , and the resulting cache misses are devastating to performance. Enter Data-Oriented Design: Struct of Arrays (SoA) Data-Oriented Design says: Stop modeling the real world. Model the data the way the hardware wants to consume it. Instead of an Array of Structs (AoS) (or an array of objects), we invert the architecture to a Struct of Arrays (SoA) . If we

Payment data pipelines fail in ways that ruin a payments engineer’s week, and the failures rhyme. The dashboards froze. Fraud scores arrived after the transaction had already cleared. Settlement reports came in stale. Nobody slept. The frustrating part is that the same data architecture had run fine for years. So, what changed? The honest answer is that batch thinking does not survive contact with real-time payments. A lot of banks built their data foundations in an era when nightly jobs were good enough. Load the warehouse overnight, run the reports in the morning, move on. That rhythm worked when money moved slowly. It does not work when a customer expects an instant confirmation and a fraud engine has milliseconds to make a call. Here is where things crack. Real-time payment rails push a constant stream of events instead of a tidy nightly dump. Your pipeline now has to ingest, transform, and serve data while transactions are still happening. Add ISO 20022 into the mix and the pressure climbs. ISO 20022 messages are rich. They carry far more structured detail than the old formats, which is wonderful for analytics and miserable for a pipeline that was never designed to parse that much context at speed. This is not a fringe concern either. Swift reported that by the time its MT/ISO 20022 coexistence period closed in November 2025, around 80% of daily traffic was already running on the ISO 20022 format, with more than 3.1 million of these messages exchanged every day. The rich-data era is the default now, not the roadmap. Then there is the fraud-scoring window. Fraud models need fresh features. Account behaviour over the last few minutes, velocity checks, device signals. If your pipeline takes thirty seconds to surface that data, the fraud decision is already too late. You are essentially detecting fraud after the loss. That gap between when data is created and when it becomes usable is the silent killer in most payment systems. And the cost of getting it wrong runs

csv-pipe: read and write CSV in TypeScript, several times faster than papaparse Myroslav Martsin Myroslav Martsin Myroslav Martsin Follow Jun 22 csv-pipe: read and write CSV in TypeScript, several times faster than papaparse # javascript # typescript # webdev # node 1 reaction Add Comment 2 min read

আমার colleague এখন প্ল্যান দেখতে পারছে। Scan types বুঝতে পারছে। Join types বুঝতে পারছে। Estimate আর actual এর gap দেখতে পারছে। BUFFERS ও দেখছে। কিন্তু সে প্রশ্ন করল। এসব দেখে কি করব? Step by step কোন পথে যাব? আমি বললাম। পাঁচটা step আছে। অর্ডার অনুযায়ী। পর্ব ২ এ আমি বলেছিলাম scan types, join types, estimate আর actual এর gap। BUFFERS কি। এবার আসি সমাধান এ। Diagnostic Workflow আপনার কাছে একটা slow query এসেছে। কিভাবে debug করবেন? এই পাঁচটা প্রশ্ন করুন অর্ডার অনুযায়ী। ৯০% slow query প্রথম বা দ্বিতীয় ধাপেই solve হয়ে যায়। ১. Deepest Seq Scan দেখুন Table বড় কি না? Filter selective কি না? Missing index থাকলে add করুন। আজই শুরু করুন যখন একটা Seq Scan দেখবেন big table এ, প্রথমে WHERE clause টা check করুন। Selective কি না? ৫% এর কম row return হওয়ার কথা? যদি তাই হয়, index missing। CREATE INDEX idx_name ON table(column) run করুন। ২. Join types দেখুন কোনো Nested Loop আছে কিন্তু দুই পাশেই বড় table? Hash Join force করুন বা ডান পাশে index add করুন। আজই শুরু করুন Nested Loop দেখলে ডান পাশের table এ index check করুন। যদি না থাকে, create করুন। Index থাকা সত্ত্বেও planner Nested Loop use করছে? SET enable_nestloop = off temporarily disable করে দেখুন। Hash Join আসবে কি না। ৩. Row estimates দেখুন Estimate vs actual ১০x এর বেশি difference? ANALYZE table দিন বা predicate rewrite করুন। আজই শুরু করুন rows=1 estimate কিন্তু rows=100000 actual দেখলে ANALYZE tablename run করুন। Statistics refresh হবে। তারপর plan আবার দেখুন। যদি তাও না আসে, WHERE clause rewrite করুন। Function call থাকলে remove করুন। Type mismatch থাকলে fix করুন। ৪. BUFFERS add করুন কোনো node এ অনেক disk reads? Caching investigate করুন। আজই শুরু করুন EXPLAIN (ANALYZE, BUFFERS) run করে দেখুন shared read high কোথায়। সেই node টাই bottleneck। Index add করলে reads কমবে। Pre-warm cache করতে পারেন। Data pre-load করতে পারেন। ৫. Sorts আর hashes দেখুন কোনো spill-to-disk আছে? work_mem raise করুন বা sort eliminate করুন। আজই শুরু করুন Plan এ external merge Disk: 421MB দেখলে spill-to-disk হয়েছে। SET work_mem = '256MB' temporarily rais

In the earlier posts of this series, we looked at practical query tuning tips and how to read and interpret query plans . A recurring theme in both was: "add an index here." But "add an index" is a bit like saying "use the right tool" — the interesting part is which one. PostgreSQL ships with several index types, each tuned for a different kind of data and query. Picking the wrong one means PostgreSQL quietly ignores your index and goes back to a sequential scan. In this post, we'll walk through the main index types, when each shines, and the special index variations (composite, partial, covering, expression) that often matter more than the type itself. Setting the Scene: Schema and Sample Data We'll reuse the same schema from the previous posts, with one small addition — a metadata JSONB column and a tags array on orders , so we can explore the more exotic index types. CREATE TABLE customers ( id SERIAL PRIMARY KEY , customer_name VARCHAR ( 255 ), email VARCHAR ( 255 ), created_at TIMESTAMPTZ DEFAULT NOW () ); CREATE TABLE orders ( id SERIAL PRIMARY KEY , customer_id INT REFERENCES customers ( id ), order_date TIMESTAMPTZ DEFAULT NOW (), total_amount NUMERIC ( 10 , 2 ), status VARCHAR ( 20 ), tags TEXT [], metadata JSONB ); -- Insert sample customers INSERT INTO customers ( customer_name , email ) SELECT 'Customer ' || i , 'customer' || i || '@example.com' FROM generate_series ( 1 , 1000000 ) AS s ( i ); -- Insert sample orders INSERT INTO orders ( customer_id , order_date , total_amount , status , tags , metadata ) SELECT ( RANDOM () * 1000000 ):: INT , NOW () - interval '1 day' * ( RANDOM () * 365 ):: int , ( RANDOM () * 500 + 20 ), ( ARRAY [ 'pending' , 'shipped' , 'delivered' , 'cancelled' ])[ FLOOR ( RANDOM () * 4 + 1 )], ARRAY [( ARRAY [ 'gift' , 'priority' , 'fragile' , 'bulk' ])[ FLOOR ( RANDOM () * 4 + 1 )]], jsonb_build_object ( 'channel' , ( ARRAY [ 'web' , 'mobile' , 'store' ])[ FLOOR ( RANDOM () * 3 + 1 )]) FROM generate_series ( 1 , 1000000 ) AS s ( i

Hey everyone, I wanted to share an open-source library I’ve been developing to solve a persistent issue in frontend file ingestion: handling large-file uploads efficiently without blocking the main thread, consuming excessive client-side memory, or introducing heavy npm dependencies. The core architecture leverages Fetch Duplex streams combined with Web Streams API to achieve constant memory usage during large file transfers. For browsers lacking full duplex stream support (such as Safari), it seamlessly switches to an automated chunked XHR fallback at runtime. ⚙️ Core Architecture & Features Constant Memory Footprint: Streams large chunks sequentially using Fetch duplex streaming where supported. Intelligent Runtime Fallback: Detects capabilities instantly and falls back to a robust, chunked XMLHttpRequest pipeline to ensure cross-browser compatibility (including Safari). Resilient Lifecycle Management: Built-in hooks for pause, resume, manual abort, and automated chunk-level retries with a configurable exponential backoff algorithm. Zero Dependencies & Tree-shakeable: Written entirely in vanilla TypeScript with no external runtime dependencies (npm install u/universal-uploader/core). The architecture is highly modular, ensuring that unused upload strategies are completely tree-shaken during compilation. React Primitive Included: Ships with a declarative React hook that maps the entire upload lifecycle to state primitives without causing redundant re-renders. 🛠️ Why Existing Solutions Didn't Fit Most mainstream uploading libraries either rely on heavy multi-part form encodings that require buffering files entirely into browser memory, or pull in heavy polyfill architectures that bloating the initial bundle size. I designed this to isolate the transport layer logic via a composition-based approach, separating the stream controller from the network client. To ensure deterministic behavior, the codebase is fully covered by 127 integration/unit tests validating network

I run Convertify , a free image converter built on Rust and libvips. Last week I decided to stop guessing about format performance and actually measure it. I took 50 real images (26 PNGs, 24 iPhone HEIC photos), ran 17 conversions through the production pipeline, and recorded every file size and encode time. Some results confirmed what everyone says. Others did not. The three results that surprised me 1. Converting HEIC to JPG makes files 14% bigger , not smaller. This one hurt. "Convert iPhone photos to JPG" is probably the most common advice on the internet. But HEIC wraps the HEVC codec, which compresses roughly 2x better than JPEG. Going from a better codec to a worse one means the file grows. Every time. If you actually want smaller iPhone photos: HEIC to WebP saves 43%, HEIC to AVIF saves 57%. 2. AVIF encodes 7x slower than WebP for 10% more compression. AVIF Q63: 55 KB, 1.30s per image. WebP Q80: 61 KB, 0.19s per image. That is a 10% size difference for a 7x speed penalty. For a single hero image, nobody cares. For a batch pipeline processing thousands of product photos, that is the difference between 3 minutes and 21 minutes. 3. PNG at 600 DPI is smaller than PNG at 300 DPI when rasterizing PDFs. This was the weirdest one. I was benchmarking PDF-to-image and noticed PNG output shrank from 2,221 KB at 300 DPI to 1,660 KB at 600 DPI. I spent an hour convinced I had a bug. Turns out it is a real property of PNG encoding. Higher DPI renders smoother gradients between adjacent pixels, and PNG's prediction filters (Paeth, sub, up) compress smooth gradients dramatically better than the sharp edges you get at lower resolutions. Not a bug. Just PNG being PNG. The quick reference table Conversion Size change Speed JPG to WebP Q80 -64% 0.19s JPG to AVIF Q63 -68% 1.30s PNG to WebP Q80 -92% 0.21s PNG to JPG Q85 -86% 0.07s HEIC to JPG Q85 +14% 1.90s HEIC to WebP Q80 -43% 5.64s HEIC to AVIF Q63 -57% 14.52s WebP to JPG Q85 +60% 0.09s AVIF to JPG Q85 +80% 0.15s What I actual

I was debugging an internal project management application built with SvelteKit and a Rust API. Locally, navigation felt almost instant. On the VPS, opening the Tickets, Timeline, and OpenSpec docs pages felt noticeably slower. Clicking a ticket also took too long before the preview panel became useful. My first assumption was infrastructure: Maybe the VPS was underpowered. Maybe PostgreSQL queries were slow. Maybe the reverse proxy added latency. Maybe SvelteKit SSR was taking too long. The measurements pointed somewhere else. The Baseline I started with the feature list endpoint used by both Tickets and Timeline. For a project with 52 tickets: Metric Result API response time ~4 ms Response size 353,956 bytes Number of tickets 52 The API was not slow. But it was returning around 354 KB for a list of only 52 items. The SvelteKit route payload showed the same pattern: Route Data payload Tickets 349,857 bytes Timeline 354,731 bytes This explained why local testing was misleading. On localhost, transferring and parsing a few hundred kilobytes is easy to miss. Once the app runs behind a VPS, reverse proxy, TLS, and a real network connection, the payload becomes much more visible. What Was Inside the Payload? I broke down the feature response by field. The descriptions alone accounted for: 296,177 bytes That was more than 80% of the complete response. The list endpoint was returning something similar to this for every ticket: interface FeatureListItem { id : string ; title : string ; status : string ; priority : string ; storyPoints : number | null ; dueDate : string | null ; description : string | null ; checkoutCommand : string | null ; openSpecCommand : string | null ; } The problem was not that these fields were useless. They were useful on the ticket detail panel. They were not useful when rendering the initial list. Timeline was even more wasteful. It used ticket status, dates, dependencies, and assignees, but still downloaded every full Markdown description. The D

Imagine a user trying to upload a 100MB video or a high-resolution photo to your app. If you use the standard Rails file upload, that file travels from the user's browser to your Rails server, and then your server sends it to S3 or Google Cloud. This is a terrible way to do it. While that 100MB file is transferring, your Rails worker (Puma) is frozen. It can't handle other users. If three people upload large files at once, your whole app will stop responding. In 2026, the professional way to handle this is Direct Uploads . With Direct Uploads, the file goes directly from the user's browser to your cloud storage (S3, R2, etc.). Your Rails server only handles a tiny bit of metadata. It is faster for the user and much safer for your server. Here is how to set it up in Rails 8. STEP 1: Configure Your Storage First, make sure you aren't using the local disk for production. You need a cloud provider like AWS S3 or Cloudflare R2. In your config/storage.yml : amazon : service : S3 access_key_id : <%= ENV['AWS_ACCESS_KEY_ID'] %> secret_access_key : <%= ENV['AWS_SECRET_ACCESS_KEY'] %> region : us-east-1 bucket : my-app-uploads # Crucial for Direct Uploads! public : true Note: You must configure CORS in your S3/R2 dashboard to allow requests from your domain. If you don't do this, the browser will block the upload. STEP 2: The Rails Form Rails makes the backend part incredibly easy. You just add one attribute to your file field: direct_upload: true . <!-- app/views/users/_form.html.erb --> <%= form_with ( model: user ) do | f | %> <div class= "field" > <%= f . label :avatar %> <%= f . file_field :avatar , direct_upload: true %> </div> <%= f . submit "Save Profile" %> <% end %> When you add direct_upload: true , Rails automatically includes a JavaScript library that handles the "handshake" with S3. STEP 3: Adding a Progress Bar (The UX Win) Direct uploads can take a few seconds. If nothing happens on the screen, the user will think your app is broken. We can use the built-in Ac

Logging has a habit of ending up in the places you care about most. It starts as a few lines for visibility. Then those lines appear in request handling, market-data processing, matching loops, telemetry pipelines, and other code where predictable latency matters. At that point, a log statement is no longer just observability. It is work running on the same thread you are trying to keep fast. A line like this can look harmless: LOG_INFO ( logger , "order_id={} price={}" , order_id , price ); The important question is what happens before the caller continues . Does it evaluate expensive arguments? Format text? Copy buffers? Allocate? Contend with other producer threads? Wait for queue space? For many applications, those costs are acceptable. For latency-sensitive systems, they are part of the latency budget . spdlog is one of the best-known C++ logging libraries and a strong general-purpose choice. It is mature, easy to use, and has a broad feature set. Quill was designed for a narrower problem: How little work can a C++ logger leave on the caller thread while still producing rich, human-readable logs? That is the lens for this comparison. The interesting difference is not which library has more features. It is where each library chooses to spend work. At a Glance Area spdlog async Quill User-message formatting Producer thread Backend thread Producer handoff Shared thread-pool queue Per-thread SPSC queue Arguments for runtime-disabled levels Evaluated if the level was not compiled out Skipped by the macro-level runtime check Native synchronous mode Yes No Backend workers Configurable thread pool Single backend worker Primary focus General-purpose flexibility Low producer-side latency These differences do not make one library universally better. They make each library better suited to different workloads. Async Logging Is Not One Design "Async logging" often means "file I/O happens on another thread." That is useful, but it is not enough to describe the cost paid by t

C++ source code is written in order. That does not mean the processor executes it in order. This is the first correction. It is also the one many performance discussions manage to avoid. Modern high-performance cores use out-of-order execution . They accept a sequential instruction stream, break it into internal operations, rename registers, place work into scheduling structures, execute ready operations early, and then retire the results in program order. The machine preserves the visible behavior of sequential execution. Internally, it is not taking attendance line by line. For ordinary software, this is mostly invisible. For C++ intended to run in tens of nanoseconds, it is not invisible. At that scale, performance is not just about the number of instructions. It is about whether those instructions can be scheduled in parallel or whether the program quietly built a dependency chain and then acted surprised. The processor is a dependency scheduler Out-of-order execution exists because in-order pipelines waste time. If an older instruction stalls, an in-order processor must often wait even if later instructions are independent and ready. That is a poor use of hardware. The chip has execution units available. The instruction stream has more work. Dynamic scheduling fixes part of this problem. The processor tracks which operations have their inputs ready. When an operation is ready and an execution unit is available, it can issue. Older operations may still be waiting. Later operations may run first. The final architectural state is still committed in order, so the program behaves correctly. Tomasulo’s algorithm is the classic model for this idea. It used reservation stations and register renaming to allow instructions to execute when their operands became available rather than strictly when they appeared in the original program (Tomasulo, 1967). Later superscalar processors extended the same general approach with speculation and reorder buffers, but the central idea

TL;DR — UUID v7 trie 13× plus vite que v4 en simulation B-tree (1M entrées), expose une empreinte mémoire identique, mais révèle son timestamp d'émission. UUID v4 reste le choix "zéro réflexion" pour les identifiants isolés. Le reste de cet article vous donnera les données pour décider. Introduction Les UUIDs sont omniprésents dans les systèmes modernes : clés primaires de bases de données, identifiants de sessions, tokens de traçabilité. Pourtant, le choix de la version impacte directement les performances en écriture et en lecture, la fragmentation des index, et — dans certains contextes — la confidentialité des données. UUID v4 (RFC 4122, 2005) est aujourd'hui la version par défaut de presque tous les ORM et frameworks. UUID v7 (RFC 9562, 2024) est son successeur moderne, conçu pour corriger son principal défaut : le désordre lexicographique. Dans cet article, nous allons mettre les deux en face avec des benchmarks réels sur des volumes de 100 000 à 10 millions d'UUIDs , analyser leur structure bit par bit, et vous donner une grille de décision claire. Structure interne : ce que contiennent ces 128 bits UUID v4 xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx Bits Contenu 0–47 Aléatoire 48–51 Version (0100 = 4) 52–63 Aléatoire 64–65 Variant (10) 66–127 Aléatoire 122 bits d'entropie pure. Aucune information temporelle. Chaque UUID est statistiquement indépendant des autres. UUID v7 019ed5c8-2a2f-7974-91f2-6ba1f313dcfa └──────────────┘ 48 bits = timestamp Unix en millisecondes Bits Contenu 0–47 Timestamp Unix (ms) 48–51 Version (0111 = 7) 52–63 Aléatoire (sub-ms ou compteur) 64–65 Variant (10) 66–127 Aléatoire 74 bits d'entropie + 48 bits de temps. Naturellement monotone : deux UUIDs générés dans la même milliseconde sont toujours distincts et ordonnés de façon cohérente. Lecture du timestamp (Python) : import uuid6 u = uuid6 . uuid7 () b = u . bytes ts_ms = int . from_bytes ( b [: 6 ], ' big ' ) # → 1781703125553 (ms depuis epoch Unix) Depuis la sortie de nos benchmarks : [00

As a backend developer, I sometimes help companies evaluate candidates by reviewing their recorded technical interviews. However, over time, I’ve noticed a deeply ingrained misconception. When discussing memory management or data streaming, many developers explicitly state: "Iterators in Python are inherently eager. If you want true lazy loading or lazy evaluation, you have to use generators and the yield keyword." This misconception is common. Many popular bootcamps and online courses introduce lazy evaluation exclusively through generators . Custom class-based iterators are usually skipped or dismissed as boilerplate-heavy OOP theory rarely used in production Python. This confusion is further reinforced by two common educational simplifications : The List vs. Generator Expression Analogy: Beginners are taught that square brackets [...] (list comprehensions) are eager and take up memory, while parentheses (...) (generator expressions) are lazy. This often creates a false binary mental model: "generators = lazy, everything else = eager." Standard "Textbook" Examples: When courses demonstrate a custom iterator, they usually write a basic class that accepts an already fully loaded list in its __init__ and simply increments an index in __next__ . While this is valid for in-memory data, it leads developers to assume that custom iterators inherently require loading all data upfront. In reality, generators are a specialized language feature designed to implement the iterator protocol automatically . They comply with the exact same interface ( __iter__ and __next__ ). A generator is lazy not because of some magical property of the yield keyword, but simply because it adheres to this underlying contract. To show that custom iterators can be lazy without using any generators or yield keywords, I’ve put together a lightweight and reproducible benchmark. 🧪 The Experiment: Proving Lazy Loading with Custom Iterators Suppose we need to read a database export file ( test_users_db.

A bot on Polymarket quietly extracted $32k in near risk-free profits by sniping “Will Company XYZ Beat Earnings?” markets. It waits for the official release, then instantly buys the winning side. Many limit orders from retail traders remain uncancelled, creating a post-announcement arbitrage window. Two developers decided to challenge it. Here’s what they learned while trying to build a faster version. Infrastructure Choices Location : Polymarket’s CLOB runs in AWS eu-west-2 (London). They deployed from Ireland (eu-west-1, Dublin) — the closest realistic option without IP tricks. UK IPs are blocked. Language : Rust for type safety and speed. The author notes you can achieve competitive latency in Python if you strip unnecessary network calls. Key Warning : Avoid the official Polymarket SDKs for ultra-low latency. They include helpful but slow pre-trade checks. Build lean custom clients. The Data Feed Challenge (The Real Bottleneck) The critical edge is getting earnings announcements faster than competitors. Source Performance Verdict Scraping Newswires Too slow Failed Benzinga Low-Latency Slower than manual clicking Failed Paid ultrafast feed ~500ms after release Still too slow EDGAR Consistently slower than newswires Backup only Even at 500ms, the order book was already swept by faster bots. The top players are likely using extremely expensive dedicated feeds or custom setups. Technical Lessons Learned Network > Code Most latency lives in the network round-trip, not in language choice. Optimize transport first. Custom Execution Layer Skip heavy SDK abstractions. Direct signed orders with minimal validation. Post-Event Sniping Logic Monitor newswire feeds aggressively Parse EPS vs. estimate instantly Place aggressive limit/market orders on the winning side Handle cases with ambiguity (multiple interpretations of “beat”) Reality Check They made some wins during EPS ambiguity or when faster bots hit size limits, but never won on pure speed against the leader. Why This