标签：#ev

Disclosure: the numbers below are from Tessera, which I work on. The reasoning applies to any proxy that sits in a session path. Every vendor page in this category says something like "scales to thousands of users". It is a useless number, because a user who is not connected costs nothing. What costs something is a session that is open right now. So here is the arithmetic instead, with the method, so you can check it against whatever you are evaluating. The unit The controller does not know how many engineers you employ and does not care how many targets are registered. It knows how many sessions are open. The planning rule that has held up for us: on a normal working day, 10–20% of a team is connected at once. A 200-person engineering organisation is 20–40 concurrent sessions, not Size for your own observed peak, but if you are estimating from scratch, start there. This matters because the difference between the two numbers is the difference between a 512 MB VM and an argument about whether you need a cluster. Memory A proxied session is mostly buffers. In our case: about 256 KB of copy buffers, plus 6 to 10 goroutines at roughly 8 KB of stack each. Call it 320 KB per session. The base Go process is about 30 MB. So 200 concurrent sessions is 200 × 320 KB ≈ 64 MB of live data, plus 30 MB base, ≈ 94 MB. Except that is not what RSS will show you, and this is the part people get wrong when they size Go services. Go does not hand memory back to the operating system promptly, and at the default GOGC=100 the collector lets the heap grow to roughly twice the live set before collecting. So resident memory settles at about double the arithmetic. Concurrent sessions vCPU Expected resident Provision up to 50 1 ~90 MB 512 MB 50–200 2 ~190 MB 1 GB 200+ 4 ~380 MB 2 GB The gap between the last two columns is headroom for spikes, not a hidden cost. A controller serving 200 sessions really does use a couple of hundred megabytes. Our Helm chart ships requests: 256Mi and limits: 1Gi ,

Disclosure: I work on Tessera, which is one of the proxy-shaped tools. Both shapes are legitimate and I try to be fair to the other one below. Most comparisons of access-control tools start with feature tables. That is the wrong end. The decision that actually determines whether a rollout finishes is the deployment shape, and there are only two. Shape one: agents and certificates You run an internal certificate authority. Hosts are configured to trust it. Users get certificates that expire in a few hours. For Kubernetes, an agent runs inside the cluster and brokers access from there. What this buys you is genuinely good. Expiry does revocation automatically, which removes the human step that fails. The credential on the user's laptop is worthless tomorrow. The model scales well because the CA does not sit in the data path — once the certificate is issued, the user talks to the target directly, so there is no proxy to size and no bandwidth to plan. What it costs is that you have to change production before you get anything. sshd_config gets rewritten across the estate to add TrustedUserCAKeys . An agent gets deployed into every cluster. In some setups the tool's binary is copied onto hosts. None of that is technically hard. It is organisationally hard. You need a change window, sign-off from whoever owns those hosts, and a rollback plan — for a project whose entire benefit is "nothing bad will happen later". That conversation is where access-control rollouts stall, and it stalls most reliably in exactly the organisations that need the tool most: the ones where nobody is quite sure who owns which box. The other cost is that the CA private key becomes the most sensitive object your company owns, and now you operate a CA. Shape two: a proxy The credential stays on a controller. The user authenticates to the controller. The controller opens its own connection to the target, authenticates with the real credential, and relays. The target sees a normal connection from a nor

Every infrastructure post-mortem contains the same paragraph, and it is never the one anyone expected to write. The initial access was not sophisticated. It was a credential that existed, that worked, and that nobody had a reason to look at — because it had been legitimately issued months earlier, for a reason that had since ended. That is standing access. It is not a vulnerability, so no scan finds it. It is not a misconfiguration, so no posture report flags it. It is the residue of a hundred reasonable decisions: a key added for a migration, a database password shared during an outage, a kubeconfig sent to a contractor who did good work and left on good terms. It stays invisible to the risk register because a risk register asks what could go wrong, and standing access is the record of things that already went right. The distribution problem The mechanism is worth being precise about, because it explains why the usual fixes only partly work. Infrastructure access is almost always handed out rather than granted . A key is copied to a host. A kubeconfig is copied to a laptop. A password is copied into a password manager and then, at two in the morning, into a chat window. Once a credential has been copied, the organisation has permanently lost the ability to list its copies. There is no query that returns the answer. Revocation stops being an operation and becomes an investigation — carried out by people, at the exact moment the person who knew where everything was has left. This is why the honest test of an offboarding process is not "did we remove their access". It is: can we show a third party, at any point in the future, that access ended when we say it ended? Most organisations that pass the first test fail the second, and usually find out during an audit or a due-diligence review, which are the two worst moments to find out. Why the obvious fixes fall short Configuration management as the source of truth is a real improvement. It makes access declarative and pu

Nobody chooses .NET. Not as a student, at least. You hear "backend" and the room splits into Spring...

To add the AWS AppConfig Agent Lambda extension to an AWS Lambda function, you can open the documentation, scroll through a table of ARNs, find the one that matches your AWS Region and architecture, copy it, and paste it into your template. However, a few months later, AWS publishes a new version and now your deployment is silently using an older one. There’s a better approach that uses public parameters in AWS Systems Manager Parameter Store (Parameter Store). What are public parameters in Parameter Store? Many AWS services use Parameter Store to publish read-only public parameters with names that start with aws/service/{service-name} . These public parameters contain up-to-date metadata about AWS services. You've probably seen them used for AMI lookups for fetching the latest Amazon Linux AMI ID without hardcoding it. The same mechanism is available for Lambda layer ARNs, ECS-optimized AMIs, and other resources that AWS updates regularly. The key idea is instead of looking up a value in documentation and pasting it into your code, you query Parameter Store at deploy time and get the current value. The only IAM permission that's required is ssm:GetParameter . The parameters are public and readable from any AWS account. The problem with hardcoded layer ARNs The AWS AppConfig Agent Lambda extension is distributed as a Lambda layer. To attach it, you need the layer's ARN, which includes a version number at the end: arn:aws:lambda:us-east-1:027255383542:layer:AWS-AppConfig-Extension:128 That version number changes every time AWS releases an update. If you hardcode it, you get a working deployment, but you also get silent drift. A few months from now, you'll be running an older version without realizing it. Imagine a team that's managing dozens of functions in multiple AWS Regions, and you can see how this can become a maintenance problem. Someone has to regularly check the documentation, update the ARN, and redeploy. It's not difficult work, but it's the kind of thing

Since 2021, our team at Autor has shipped over 50 AI products across healthcare, fintech, logistics, and SaaS. Some of them are running in production right now, handling thousands of automated calls per month. Others failed spectacularly — and those are the ones that taught us the most. Where This Comes From I started Autor in Toronto as a one-person AI development shop. The original thesis was simple: companies needed custom AI but couldn't hire fast enough to build it themselves. Four years and 50+ products later, we're a senior-only studio with a production voice AI platform (Loquent) serving healthcare and dental clients 24/7. Along the way, we've impacted over 5 million users, helped clients raise more than $10 million in funding, and shipped across 10+ countries. This isn't a highlight reel. This is the unvarnished list of things I got wrong, figured out the hard way, or wish someone had told me before I wrote my first line of production AI code. 1. Your First AI Product Should Be Boring Our first few products were ambitious. Multi-modal pipelines, complex reasoning chains, novel architectures. Most of them took twice as long as estimated and required constant babysitting in production. The products that actually made money and kept clients happy? A straightforward document classifier. A simple intent router. A basic FAQ bot with good fallback logic. I used to think "boring" meant "not innovative." Now I know boring means "reliable enough that I don't get paged at 3am." Our most successful product, Loquent, handles healthcare scheduling calls. It's not doing anything architecturally exotic. It picks up the phone, understands what the caller needs, books or reschedules an appointment, and hangs up. The magic isn't in the model — it's in the 200+ edge cases we've handled around it. If you're building your first AI product, pick the most boring version of your idea and ship that. You can add complexity later. You cannot add reliability later. 2. Prompt Engineerin

Originally published on Medium It's 2:51am. The incident is resolved. Error rate is back to zero, the rollback worked, and your on-call pager has finally gone quiet. Now you have to write the postmortem. If you've been in SRE or DevOps for any length of time, you know this feeling. You're exhausted, your brain is running on adrenaline fumes, and somewhere in the back of your mind you know that what you write in the next hour is going to be read by engineers, product managers, and probably a VP or two. It needs to be clear, blameless, specific, and actionable. Most of us write it badly. Not because we're bad at our jobs — because we're human beings who just spent two hours firefighting and now we're staring at a blank document at 3am trying to remember the exact sequence of events. There's a better way. The Problem With How We Write Postmortems The standard postmortem template is a solved problem. Every company has one. Timeline, root cause, contributing factors, action items — we all know the structure. The hard part isn't the structure. It's the writing. Specifically: Reconstructing the timeline from a chaotic Slack thread where half the messages are noise Writing the root cause narrative in plain language when your brain is still in technical mode Generating action items that are actually specific and assignable instead of vague gestures toward improvement Translating all of it into an executive summary that a non-technical VP can understand without losing the technical accuracy Each of these is a hard writing task under normal circumstances. At 2am after an incident they're brutal. What Changed for Me I started treating postmortem writing like any other repetitive engineering task: I built a system for it. Specifically, I built a set of AI prompts designed for the exact scenarios SREs face. Not generic "write me a postmortem" prompts — structured prompts that work with the raw material you actually have in front of you at the end of an incident. The key insight w

Today SparkLLM releases and open-sources two on-device general models: Spark X2.5-4B and Spark X2.5-1.7B . Both natively support a context window of up to 1,000,000 tokens — as far as we know, the only on-device models to do so. Why 1M context on-device In real work, you rarely hand a model a single question — you hand it a whole after-sales manual, a set of meeting materials, a batch of project docs, or an entire code repository. On-device models used to chop long content into pieces and ask about each separately, which loses context and drops information. Spark X2.5-4B and 1.7B natively support up to a 1M-token context window, trained on hundreds-of-billions-of-tokens of high-quality long-document data, so they can take in and reason over far more information in a single task — and keep the full picture across a continuous, multi-step interaction. Not just answering — doing the work Long context decides whether the model can see everything; agent + tool-use decides whether it can act on it. Office (with Loomy): upload a sales spreadsheet and ask for an analysis plus a bilingual department report — X2.5-4B writes a script to aggregate the data, extracts key metrics and trends, generates a ~3,000-word Chinese report, produces an English version in the same structure, and validates content, structure and layout end to end. Code: on algorithm implementation, completion and generation, X2.5-4B rivals cloud models 2–3× its size . It plugs into open harnesses like DeepSeek Harness, OpenCode, Codex and Pi for local dev and automation — with low latency, offline use, and code kept on-device. Smart home: on the Domux smart-home test set, X2.5-1.7B reaches 90.3% end-to-end command accuracy at 0.85s average latency. Robotics: both sizes suit continuous perception-and-execution on-robot or on edge devices — operation control, target tracking, navigation decisions — with less dependence on the cloud. Domestic compute, open deployment Both models were trained end to end on a ful

The Problem: Testing Across Regions Is Hard If you're a developer, you've probably faced...

A raw sold-listings search is not automatically a usable comp set. Search for a phone and you may also get cases, chargers, broken screens, empty boxes, and nearby models. Search for a camera lens and you may get caps, adapters, or a different focal length. If those rows go directly into a median, the result can describe the search noise instead of the product. I wanted a larger measurement than a single convenient example, so I ran a fixed 100-product study through CompSniper, the sold-price API I own. The goal was not to prove that an automated classifier is always correct. The goal was narrower: Measure what the production relevance cleaner removed and how the product-level median changed on one predeclared sample. The protocol I selected the products before making the first request: 20 smartphones and tablets 20 gaming and computing products 20 cameras and lenses 20 audio and music products 20 collectibles and luxury products Every search used the same settings: Marketplace: ebay.com Sold window: 2026-06-02 through 2026-08-31 Page: 1 Requested rows: 240 Sort: ended recently Condition: any Relevance cleaning: enabled Each relevance-enabled response contained the raw sample count and raw median captured before classification, followed by the cleaned rows and deterministic price summary from the same fetched page. That meant one production request per product, not separate raw and cleaned fetches. All 100 requests succeeded with unique request IDs. The headline results Across the study: 19,220 priced raw rows were parsed 11,942 priced rows remained after cleaning 7,278 rows were classified out The weighted removal rate was 37.87% 34 of 100 product medians changed by at least 10% 15 of 100 changed by at least 25% 11 of 100 changed by at least 50% The direction was not always upward: 73 medians increased 21 medians decreased 6 medians stayed unchanged That is important. The cleaner is not instructed to raise prices. It tries to retain listings for the requested produ

Three parts into this series, and if you'd asked me to define the word sitting underneath every...

It's the lightest Bentley in 85 years.

The Slate Truck is affordable because you have to 'bring your own tech'.

Let me describe a moment you might recognize. You had an idea for an app. You didn't know how to...

Sooner or later you hit the same small problem: two services, on two machines, want to touch the same thing at the same moment — append to a shared file, update a row nobody is fencing, call an API that tolerates one caller at a time. One of them has to wait. The usual answers feel heavier than the problem. Put a service in front and serialize everything through it — now you are building a queue, and then a second queue to hand results back, because you no longer know the outcome at the moment you asked. Cache the resource in Redis and lock there — fine until the resource does not fit in memory, and you have inherited Redlock's ordering guarantees (there are none) and its debates. I wanted the lock as its own primitive: lock a key, do the work, unlock the key. Nothing else. That is Locking-Center — a single binary, one dependency, no config file, no consensus layer to operate. This post is about the three ideas that made it small enough to be worth trusting. 1. One channel per key — and the queue comes for free Every key gets a Go channel with a buffer of exactly one: type Channel struct { key string mutexChan chan bool // buffered, capacity 1 } func NewChannel ( key string ) * Channel { return & Channel { key : key , mutexChan : make ( chan bool , 1 )} } Sending into it acquires the lock. Receiving from it releases : c . mutexChan <- true // acquire — blocks if someone already holds the key // ... critical section ... <- c . mutexChan // release The buffer of one is the whole trick. The first send fills the buffer and returns immediately: that caller holds the key. The second send has nowhere to go, so it blocks — and so does the third, and the fourth. The blocked senders are the wait queue. When the holder releases (a receive frees the slot), the runtime wakes the next blocked sender. And it wakes them in order. The Go runtime keeps a FIFO wait queue behind every channel, so callers are served roughly in arrival order rather than whoever happens to reschedule firs

When you're building a multi-tenant app, there's one bug category that's worse than any other: a user seeing someone else's data. Not a crash, not a broken button — a genuine privacy failure. I recently built a Next.js + Supabase + Stripe starter kit, and before I'd call the database layer "done," I wanted to actually prove the security held, not just assume it did because the code looked right. The setup Supabase's Row Level Security (RLS) lets you write policies directly in Postgres that filter rows based on who's asking — the database itself becomes the security boundary, not just your application code. That's powerful, but it also means a single wrong policy (or a missing one) silently exposes everything. Here's the policy pattern I used for an owner-scoped table: create policy "projects: owner reads" on public . projects for select to authenticated using (( select auth . uid ()) = owner_id ); Simple enough. But "simple enough" is exactly the kind of thing worth verifying rather than trusting. The actual test Rather than just trusting the policy, I ran an impersonation test directly in the SQL editor: begin ; set local role authenticated ; set local request . jwt . claims = '{"sub":"<a-real-user-id>"}' ; select id , name from public . projects ; rollback ; This temporarily pretends to be a specific user (inside a transaction that touches nothing) and asks: what can this user actually see? Run it with the real owner's ID — you get their rows. Run it with a fake or different user's ID — you should get zero rows back, not an error. That's the key detail: RLS filters rows out silently rather than throwing a permission error, so "nothing happened" is actually the success signal, not a bug. I did the same check for writes — attempting to delete another user's row from a different account's session, confirming it returns 0 rows affected rather than either succeeding or throwing. Why this matters more than it seems It's easy to write an RLS policy that looks right and s

Most teams don't notice their CI/CD pipeline is broken — they just notice that deploys "feel slow" and shrug it off as normal. It isn't. A pipeline that takes 25 minutes to ship a one-line copy change isn't a fact of life, it's a symptom. Here are the mistakes we see most often when reviewing pipelines — roughly in order of how much time they silently burn. 1. Running the full test suite on every single change If a developer fixes a typo in a README and the pipeline still runs the entire integration suite, database migrations, and end-to-end tests, you're paying full price for a change that touched nothing critical. Fix: split your pipeline into stages based on what actually changed. Path-based triggers (only run frontend tests if frontend files changed) and a fast "smoke test" tier before the full suite can cut average pipeline time dramatically without sacrificing safety. 2. No caching between builds Reinstalling every dependency from scratch on every run is one of the most common — and most fixable — sources of wasted time. Package managers, build artifacts, and Docker layers are all cacheable, and most CI platforms support this natively. Fix: cache dependency directories keyed by lockfile hash, and structure Dockerfiles so rarely-changing layers (base image, dependencies) come before frequently-changing ones (application code). 3. Sequential steps that don't need to be sequential Linting, unit tests, and security scans are often run one after another when they have no dependency on each other. That's pure wasted wall-clock time. Fix: parallelize independent jobs. Most CI systems support fan-out/fan-in patterns — run lint, test, and scan simultaneously, then gate the deploy on all three passing. 4. Environments that drift from production A pipeline that passes in staging and fails in production usually means the environments aren't actually equivalent — different env vars, different resource limits, different service versions. Teams respond by adding more manual

If you ever needed a quick list of Docker commands, here you go.. 1. Check that Docker is installed docker --version Shows the installed Docker version. 2. Run your first container docker run hello-world Pulls the official test image (if needed) and runs it. You should see a “Hello from Docker!” message. 3. See running containers docker ps Lists containers that are currently running. Use docker ps -a to also show stopped ones. 4. See downloaded images docker images Shows every image on your machine (name, tag, size, ID). 5. Stop a running container docker stop CONTAINER_ID Gracefully stops a container. Get the ID from docker ps . 6. Remove a stopped container docker rm CONTAINER_ID Deletes a container that is already stopped. 7. Force stop and remove docker rm -f CONTAINER_ID Force-stops the container (if it’s still running) and removes it in one step.

In this post I'll show how to merge PDF files entirely in the browser using PDF.js and pdf-lib — no server, no file upload, no backend. Everything runs on the user's machine, which is great for privacy and for keeping hosting costs at zero (it's just a static site). Why process PDFs on the client? Most "free" PDF websites quietly upload your documents to their server, which: Exposes private/sensitive files to third parties Imposes size limits Often slaps a watermark on the output Requires you to trust their storage If you handle PDFs with client-side JavaScript (WebAssembly / WASM + PDF.js), none of that happens. The user's file never leaves their device, and you don't need a backend at all — so it's cheap and private. Caveats pdf-lib works well with standard PDFs; heavily encrypted or unusual documents may need extra handling. Very large PDFs are memory-hungry since everything is client-side, but for typical documents it's fast and free. Some complex PDFs with unusual fonts can lose fidelity — test on your own files first. Try it I packaged this approach (plus split, compress, rotate, unlock, image-to-PDF) into a free no-upload tool: https://yourutilityhub.com/pdf/merge-pdf The whole project is open source: https://github.com/Jalal-khn/utilityhub- If you have questions about the architecture or want a deeper dive on any part, ask away. The basic idea Read the input file with FileReader Parse it with pdf-lib (a pure-JS PDF library) Copy the source pages into a new document Save the merged PDF and trigger a download Here's the core function: js import { PDFDocument } from "pdf-lib"; async function mergePdfs(files) { const merged = await PDFDocument.create(); for (const file of files) { const bytes = await file.arrayBuffer(); const src = await PDFDocument.load(bytes, { ignoreEncryption: true }); const pages = await merged.copyPages(src, src.getPageIndices()); pages.forEach((page) => merged.addPage(page)); } const out = await merged.save(); return new Blob([out], { typ

In mission-critical infrastructure, latency isn't just a metric—it's the difference between a resolved incident and a catastrophic outage. Whether you are managing an SRE team handling cluster failures or a teleassistance platform routing domestic SOS alerts, the core engineering challenge remains identical: getting a human's attention in milliseconds without administrative friction. This technical breakdown explores how we architected a high-availability voice routing engine using Cloudflare Workers and Twilio, bridging the gap between hardware teleassistance and DevOps incident workflows. The Dual-Use Architecture: From Teleassistance to SRE Paging Our platform core serves two distinct but structurally identical needs: Senior Safe: A Chilean domestic teleassistance product where an SOS trigger must reach a family guardian instantly. DevOps On-Call: An infrastructure alert triggered via Grafana or UptimeRobot webhooks that must wake up an engineer at 3 a.m. The blast radius differs (a household vs. a production database), but the technical path is identical. To solve this at scale without charging steep "per-seat" licensing models that penalize growing squads, we built the entire pipeline on serverless isolates. Bypassing Cold Starts with Edge Ingest When an emergency happens, you cannot afford to wait for a virtual machine or container to boot. The public ingest pipeline lives directly on Cloudflare Workers ( api.wakeupdev.com ). Because V8 isolates are kept warm globally across the edge network, there is zero Lambda-style cold start penalty on the first page. The ingest contract is minimal: Authentication: Handled via an x-api-key header. Payload: Raw text or JSON (capped at 4,000 characters). Execution: Credits are consumed atomically in a global Postgres layer before the voice cascade is scheduled. An HTTP 202 Accepted status code guarantees that the credit is validated and the call flow is in flight. Solving the Voicemail Problem: True Human Acknowledgement A