标签：#devops

Introduction As applications grow, traditional relational databases such as MySQL may struggle with analytical workloads involving millions of records and complex aggregations. While MySQL excels at Online Transaction Processing (OLTP), ClickHouse® is purpose-built for Online Analytical Processing (OLAP), enabling lightning-fast analytical queries on massive datasets. Migrating data from MySQL to ClickHouse® allows organizations to build high-performance reporting systems, dashboards, and real-time analytics without impacting transactional workloads. In this guide, you'll learn several approaches to migrate data from MySQL to ClickHouse®, along with their advantages, limitations, and ideal use cases. Why Migrate from MySQL to ClickHouse®? MySQL and ClickHouse® are designed for different workloads. Feature MySQL ClickHouse® Storage Model Row-based Columnar Best For Transactions (OLTP) Analytics (OLAP) Query Speed Fast for row lookups Extremely fast for large scans Aggregation Performance Moderate Extremely fast Scalability Primarily Vertical Optimized for analytical scaling Typical Use Cases Applications and transactional systems Reporting, dashboards, and analytics Migrating from MySQL to ClickHouse® makes sense when: Analytical queries are becoming slow in MySQL. You need real-time dashboards over large datasets. Reporting queries are impacting your production database. You regularly process millions or billions of rows. Migration Architecture MySQL │ ▼ Export / Synchronization │ ▼ Data Transformation │ ▼ ClickHouse® │ ▼ Dashboards / Analytics Migration Methods There are multiple ways to migrate data depending on your requirements. Method 1: CSV Export and Import (Recommended for Beginners) This is the simplest approach for performing a one-time migration of historical data. Step 1: Export Data from MySQL Run the following command inside MySQL: SELECT * INTO OUTFILE '/tmp/employees.csv' FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY ' \n ' FROM employ

Hey everyone 👋 If you run batch jobs, data pipelines, or any kind of AI and ML training on Kubernetes, you have probably hit this wall. Kubernetes is fantastic at deciding WHERE a pod should run, but it is surprisingly clueless about WHEN a job should start. 😅 You submit ten jobs, the cluster fills up, and the rest just sit there as Pending. No real queue, no priority, no fairness between teams. One noisy team can eat all your expensive nodes while everyone else waits. 🥲 That is exactly the gap Kueue fills, and today I want to walk you through it with a pile of hands on examples you can run on any cluster, even your homelab. 🏡 👉 Key takeaway up front: Kueue is a job level manager that holds your jobs in a real queue and only admits them when there is enough quota to actually run them. 🧪 Everything in this guide was tested against Kueue v0.18.1 using the v1beta2 API. I pinned every command and manifest to that version so you do not get surprised by API drift. 📋 What we will cover ✅ Why Kubernetes needs a queue ✅ The building blocks in plain language ✅ Installing Kueue ✅ Setting up quota with a ResourceFlavor, a ClusterQueue, and a LocalQueue ✅ Submitting a Job and watching it get queued and admitted ✅ Priority based admission ✅ Partial admission and elastic jobs ✅ Multiple resource flavors for x86 and arm ✅ Fair sharing between teams with cohorts ✅ Dedicated quota with a shared fallback ✅ Queueing a plain Pod ✅ Why this matters a lot for GPUs and your cloud bill 🤔 Why Kubernetes needs a queue Native Kubernetes scheduling is pod centric. The scheduler looks at one pod at a time and tries to place it. That works great for long running services. Batch workloads are different. They have a beginning and an end, they often need a fixed chunk of capacity, and they compete with other teams for the same nodes. Without a queueing layer you get: ✅ Jobs that fail or stay Pending when resources are tight ✅ No quota governance, so one team can starve the others ✅ No admission prio

When something breaks in Linux, experienced engineers don’t guess. They check the logs. 👉 Logs are the “black box recorder” of a Linux system. They tell you: what happened when it happened why it failed If you can read logs properly, you can debug almost anything. What Are Logs? Logs are records of system and application activity. Linux constantly records: System events Errors User activity Application behavior Linux constantly records: Where are Logs Stored? Most Linux logs are stored inside: /var/log Check logs directory: cd /var/log ls This is the first place DevOps engineers check during system issues. Important Log Files Log File Purpose Command to View /var/log/syslog General system messages tail /var/log/syslog /var/log/auth.log Login attempts & authentication tail /var/log/auth.log /var/log/kern.log Kernel & hardware messages dmesg or tail /var/log/kern.log /var/log/nginx/error.log Web server errors (Nginx) tail /var/log/nginx/error.log /var/log/dmesg Boot and hardware logs dmesg /var/log/apache2/ -> Apache logs These logs help you identify system, security, and application-level issues. View Logs Using cat cat /var/log/syslog Good for small files. Using less less /var/log/syslog Useful keys:: Space → Next page b → Previous page q → Quit 👉 Best for large log files. Using tail tail /var/log/syslog Show last 10 lines. Real-Time Monitoring (tail -f) tail -f /var/log/syslog 👉 -f = follow live updates This is one of the most-used debugging commands in production servers. Stop with: Ctrl + C Searching Logs with grep grep error /var/log/syslog Case-insensitive: grep -i failed /var/log/auth.log Show latest matching errors: grep error /var/log/syslog | tail -n 50 👉 Essential for filtering huge logs quickly. Boot & Hardware Logs (dmesg) dmesg Shows: Boot messages Hardware detection Kernel events Useful for startup and hardware troubleshooting. Modern Log System: journalctl Modern Linux systems use systemd logs . journalctl Recent errors: journalctl -xe Specific servic

Let’s be honest: as developers, our relationship with our office chairs is... complicated. We start the day sitting upright like productivity gurus, but four hours into a debugging session, we’ve morphed into a human pretzel. This "gamer lean" isn't just a meme; it leads to chronic back pain and decreased focus. In this tutorial, we are going to build a real-time posture tracking system using MediaPipe Pose and Computer Vision to save your spine. By leveraging AI productivity tools and the power of cross-platform Electron desktop apps , we will create a silent guardian that watches your form and pings you the moment you start slouching. If you've been looking for a practical way to dive into MediaPipe and Node.js integration, you're in the right place. For those looking for more production-ready patterns and advanced AI implementations, I highly recommend checking out the deep dives at WellAlly Blog . 🏗 The Architecture The system works by capturing frames from your webcam, processing them through a pre-trained neural network to identify body landmarks, and then applying some basic trigonometry to determine if your posture is healthy. graph TD A[Webcam Stream] --> B[MediaPipe Pose Engine] B --> C[Extract 33 Keypoints] C --> D{Geometry Engine} D -->|Angle > Threshold| E[Slouch Detected] D -->|Angle < Threshold| F[Good Posture] E --> G[Electron Main Process] G --> H[System Notification 🔔] F --> I[Wait 5s] I --> A 🛠 Prerequisites To follow along, you'll need: Node.js (v16+) MediaPipe (The pose solution) OpenCV.js (For frame manipulation) Electron (For the desktop shell) 🚀 Step 1: Setting Up the Pose Engine MediaPipe provides a "Pose" model that gives us 33 landmarks in 3D space. For posture correction, we specifically care about the Ears (7, 8) , Shoulders (11, 12) , and Hips (23, 24) . The Math: Calculating the "Slouch" We measure the angle between the Ear, the Shoulder, and a vertical axis. If your ear moves too far forward relative to your shoulder, that's "Forward

trpc has a scheduled workflow called "Lock Issues & PRs." Its own scorecard shows it failing on almost every run. It is still scheduled, still running, still red. trpc ships excellent software, which is exactly the point: if a project this careful has a workflow that has been red for ages, the rest of us almost certainly do too. It is not a one-off. drizzle-orm has one ("Unpublish release"). cal.com has one ("PR Update"). I scanned 35 popular open-source repos and the same thing kept turning up: a scheduled workflow that fails on nearly every run, quietly, for a long time. Why nobody notices GitHub does email you when a scheduled workflow fails. So how do these survive? Two reasons. First, those emails are routine. You get them for flaky reruns and transient blips too, so you filter them out. Second, a workflow that is always red stops reading as a signal. It is just how that row looks now. I did exactly this on my own project. GitHub emailed me that a workflow had failed. The next day it emailed again. I saw it, told myself I would fix it tomorrow, and promptly forgot. It was my nightly database backup, quietly broken the whole time, and I only caught it when a failure-rate number crept up where I would notice. An always-red workflow is not free It burns minutes every run to produce nothing but a red X. Worse, it trains you to ignore the failure that actually matters: the day a real one lands in the same inbox you have learned to skim past. How to find yours Open your Actions tab and look at the scheduled workflows, the cron-triggered ones nobody watches. If the last several runs are all red, you found one. From the CLI: gh run list --workflow = "Lock Issues & PRs" --status = failure What to do about it Two honest options: fix it, or if the workflow is genuinely abandoned, turn it off. Do not leave it scheduled and red. gh workflow disable "Lock Issues & PRs" Or drop the schedule trigger from the workflow file if it should not run on a timer at all. A disabled work

Two days ago GitHub emailed me to say one of my workflows had failed. The next day it emailed me again. I saw it, told myself I would fix it tomorrow, and promptly forgot. It was my nightly database backup, quietly broken the whole time, and I only caught it because a failure-rate number nudged up. A failed run at least gets you an email. A slow run gets you nothing. GitHub never pings you when CI quietly takes twice as long, runs the whole suite twice per PR, or rebuilds dependencies from scratch every time. That waste compounds where no one looks. Here are the usual culprits, each with the exact fix. When I scanned 35 popular open-source repos, not one had a fully clean config. 32 of 35 had no concurrency control, 33 of 35 had no job timeouts, and 22 of 35 ran the full suite twice on every PR. If projects this polished leave minutes on the table, the rest of us definitely do. Your suite runs twice on every PR Trigger a workflow on both push and pull_request and, for a branch in the same repo, opening a PR fires both. You just paid for two identical runs. This one is pure waste and it can roughly halve your PR-related minutes. Trigger on pull_request , and keep push for your default branch: on : push : branches : [ main ] pull_request : Old runs don't cancel when you push again Push a fix 30 seconds after the first push and, with no concurrency group, both runs go to completion. The first is dead weight, and it is holding a slot in your queue while it finishes. This hides even when you do have a group: astro has a concurrency group on one workflow but left off cancel-in-progress , which our scan estimates leaves roughly 1,850 minutes a month on the table. Add a group keyed on the branch, with cancel-in-progress , so a new push supersedes the old run: concurrency : group : ${{ github.workflow }}-${{ github.ref }} cancel-in-progress : true Every run reinstalls dependencies from scratch No cache means every run re-downloads and rebuilds your dependencies. On a typical

Force nginx to TLS 1.3 An nginx server is accepting an old TLS version, and the exam wants it locked to TLS one point three. The config lives in a ConfigMap. The catch is that editing the ConfigMap alone changes nothing. Let's do it the way the CKA expects. 🎥 Watch the video: https://www.youtube.com/watch?v=rx-77YBw99w This is a CKA Workloads & Scheduling walkthrough. Every command below is real output from a live cluster, and you can reproduce the whole thing yourself (scripts at the end). The scenario An nginx-static Deployment serves HTTPS, and its server config comes from a ConfigMap named nginx-config. Right now it allows both TLS one point two and one point three. Your task is to allow only TLS one point three, then make nginx actually use the change, so that a TLS one point two request fails. nginx-static serves HTTPS from the nginx-config ConfigMap It currently allows TLS 1.2 AND 1.3 Restrict ssl_protocols to TLS 1.3 only A TLS 1.2 request to the Service must then fail How nginx, ConfigMaps, and rolling restarts fit together Two ideas drive this. First, ssl_protocols is an allow list; leave only TLSv1.3 and nginx rejects any older handshake. Second, a ConfigMap mounted into a pod updates the file on disk, but nginx only reads ssl_protocols when it starts. So you must roll the Deployment, with kubectl rollout restart, for the new value to take effect. Inspect the current state Start by seeing what is running and what the config says. The nginx-static Deployment, its Service on port four forty three, and the nginx-config ConfigMap are all here. Grep the rendered ConfigMap for the ssl_protocols line: it lists TLSv1.2 and TLSv1.3, so old clients still get in. $ kubectl -n nginx-static get deploy,svc,configmap NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-static 1/1 1 1 17h deployment.apps/tester 1/1 1 1 17h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-static ClusterIP 10.96.13.162 <none> 443/TCP 17h NAME DATA AGE configmap/kube-root-ca.

OpenAI is investigating issues with Codex usage limits Tibo wrote that the Codex team spent Sunday in a war room, digging through logs and looking for anything that could have caused faster usage drain for some users. As the investigation continues, OpenAI has issued a full reset of Codex usage limits for everyone. The funny part: this week at OpenAI is called RESET week. In US corporate culture, that usually means a lighter week to slow down, clear the calendar a bit, and recharge.

Originally published on bckinfo.com Redis with Docker Compose: Persistence, Security, and Production-Ready Configuration Table of Contents Why Redis Containers Lose Data RDB vs AOF: Choosing a Persistence Strategy Basic Setup with Docker Compose Full Persistence Configuration Securing Redis in Docker Setting Resource Limits Redis in a Multi-Service Stack Backing Up and Restoring a Redis Volume Common Issues and Quick Fixes Closing Notes Redis is one of the most common services to run in Docker — it's fast to spin up, lightweight, and perfect for caching, session storage, and queues. But that same simplicity hides a trap: by default, Redis in Docker stores everything in memory, and the moment a container is removed, all of that data disappears . This guide walks through setting up Redis with Docker Compose the right way — covering persistence, authentication, resource limits, and the health checks you need before putting it anywhere near production. Why Redis Containers Lose Data A Docker container is meant to be disposable. That's a feature for stateless services, but it's a liability for a database like Redis. If you start a plain Redis container without a mounted volume, here's what happens: The container writes its dataset only inside its own writable layer. docker compose down or docker rm removes that layer entirely. The next time the container starts, Redis initializes with an empty dataset. This single oversight accounts for a large share of "we lost our session data" incidents in small teams running Redis in containers for the first time. The fix is straightforward once you understand the two persistence mechanisms Redis offers. RDB vs AOF: Choosing a Persistence Strategy Redis supports two persistence models, and production setups typically combine both: RDB (Redis Database snapshots) Point-in-time snapshots of the dataset, saved at intervals you define. Fast to restore, but you can lose any writes that happened after the last snapshot. AOF (Append Only Fil

Last month, I had a problem with a popular mobile banking app in Southeast Asia. Nothing exotic. A transaction didn't go through, and my support ticket had been sitting untouched for two weeks. So I opened the app's chatbot. It greeted me warmly, asked how it could help, and then couldn't do a single useful thing. It couldn't look up my transaction. It couldn't check the status of my ticket. It couldn't tell me why my issue was unresolved. It could answer FAQ questions, and that was it. I called the hotline instead. Spent an hour navigating prompts, got bounced between menus, and every path ended the same way: "Please contact our chatbot or check your existing ticket." The system was built for deflection, not resolution. The ticket that nobody had touched for fourteen days. I gave up. And somewhere in that company's dashboard, my interaction counted as a successful AI chatbot deflection. The uncomfortable part: if you shipped a deflection-optimized bot this quarter, a customer somewhere is living this exact loop right now. Your dashboard is calling it a win. The Deflection Metric Everyone Loves (and Nobody Questions) Deflection rate measures the percentage of customer contacts handled without a human agent. It's cheap to track, easy to celebrate, and it maps directly to cost savings. Industry benchmarks citing McKinsey's 2026 service operations data put AI resolutions at $0.62 per ticket versus $7.40 for human agents. That's a 12x cost difference. Of course executives love this number. But deflection doesn't measure whether the customer's problem got solved. It measures whether the customer stopped asking. Those are very different things. This is Goodhart's Law applied to customer experience: when a measure becomes a target, it ceases to be a good measure. Deflection is cheap and easy to optimize. Resolution is hard and expensive to track. So companies optimize the proxy and stop looking at the goal. Gartner data, as reported by Forbes , confirms the gap: only 14% o

Over the past year, one concept has fundamentally changed how I think about AI applications. Not larger language models. Not better prompts. Not even AI agents. It's Model Context Protocol (MCP) . For a long time, most AI applications lived inside a closed environment. They could generate text, answer questions, or write code, but they couldn't easily interact with external systems. MCP changes that. It provides a standardized way for AI models to communicate with tools, databases, APIs, and applications. Instead of building custom integrations for every project, developers can expose capabilities through MCP servers. After experimenting with different workflows, these are five MCP servers that have had the biggest impact on how I build AI applications. 1. GitHub MCP Server If you're building software with AI, GitHub integration is one of the most valuable capabilities you can add. Imagine asking an AI assistant to: Read a repository Review pull requests Search issues Create commits Open new issues Inspect project structure Instead of manually copying files into ChatGPT, the AI can interact directly with your repository. For developers, this dramatically improves productivity. Typical workflow: Developer Request ↓ GitHub MCP Server ↓ Repository ↓ LLM ↓ Action or Response This is far more scalable than copying snippets of code into prompts. 2. Filesystem MCP Server Almost every AI workflow eventually needs access to local files. Examples include: Reading documentation Editing Markdown Creating reports Refactoring code Updating configuration files Without an MCP server, these tasks often require multiple manual steps. With a Filesystem MCP server, an AI application can safely interact with project directories. For example: Read: /docs/api.md Update: /src/routes.py Create: /reports/summary.md This makes AI assistants feel much more like development partners. 3. PostgreSQL MCP Server One limitation of traditional chatbots is that they don't know your data. Connecting an

I use my SSH manager every day. I also use a separate monitoring tool every day. For a long time I just accepted that these were two different things. Then one day I was SSH'd into a server that was behaving weird. I wanted to check if it was CPU or memory, but I had to open a different app, find the server in there, and wait for the dashboard to load. It took maybe 15 seconds. Not a huge deal. But it broke my flow every single time. I already had an SSH connection open to that server. Why was I opening a second thing just to see what was happening to it? That's what pushed me to build server monitoring directly into Termique, the SSH manager I've been working on. The interesting part: reusing the existing SSH connection SSH connections aren't just for terminals. The protocol supports multiple channels over a single TCP connection. You can have a terminal session running in one channel while sending short exec commands through another channel on the same connection. That's how the monitoring feature works. When you open the metrics panel for a server, Termique creates a separate exec channel on the existing SSH connection and polls /proc/stat for CPU, /proc/meminfo for RAM, and /proc/loadavg for system load. Short-lived commands, called on an interval, over the connection you already have open. No second SSH handshake. No separate auth. Just another channel on the same pipe. The tradeoff: you do need an agent I want to be upfront about this. The monitoring feature requires a small agent installed on each server. It's not agentless. I considered going agentless, relying entirely on /proc reads through exec channels. That works fine on most Linux servers. But the agent makes it easier to handle edge cases properly and opens the door for future features like alerts and longer history retention. Without it, I'd be fighting a lot of fragile shell parsing. If you're managing Linux servers, it's a one-command install. Non-Linux systems aren't supported yet. That's a real l

When Logs and Metrics Aren't Enough You have great dashboards. Your log aggregation is solid. But when a user reports "the checkout page is slow," you still spend 30 minutes jumping between services trying to find the bottleneck. That's the gap distributed tracing fills. What Tracing Actually Shows You A trace is a complete picture of a single request as it flows through your system: User Request → API Gateway → Auth Service → Product Service → DB → Cache → Response 5ms 12ms 45ms 120ms 3ms ^ This is your bottleneck Without tracing, you'd see: API Gateway: latency looks fine Auth Service: latency looks fine Product Service: latency is HIGH but why? With tracing, you see the exact DB query inside Product Service that's taking 120ms. Getting Started with OpenTelemetry OpenTelemetry is the standard. Here's a minimal setup: # Python example with Flask from opentelemetry import trace from opentelemetry.instrumentation.flask import FlaskInstrumentor from opentelemetry.instrumentation.requests import RequestsInstrumentor from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter # Setup provider = TracerProvider () provider . add_span_processor ( BatchSpanProcessor ( OTLPSpanExporter ( endpoint = " http://otel-collector:4317 " )) ) trace . set_tracer_provider ( provider ) # Auto-instrument everything FlaskInstrumentor (). instrument_app ( app ) RequestsInstrumentor (). instrument () SQLAlchemyInstrumentor (). instrument ( engine = db . engine ) That's it. Three auto-instrumentations cover 80% of what you need. Custom Spans for the Other 20% Auto-instrumentation gives you HTTP calls and DB queries. Add custom spans for business logic: tracer = trace . get_tracer ( __name__ ) def process_order ( order ): with tracer . start_as_current_span ( " process_order " ) as sp

Aprenda a substituir o velho Cron por systemd timers para ganhar logs nativos, controle de dependências e maior robustez.

I am committed to understand how systems actually work. I'm working on a multi-node lab to follow the complete path of a request from Python APIs to Linux processes, through Docker containers, networking and observability. The idea is simple: build a system that observes another system to understand the abstraction layers behind modern infrastructure. This project is about learning by building, experimenting and understanding what happens under the hood. Link: [ https://github.com/daniloprandi/devops-network-automation-lab ] DevOps #Linux #Python #Docker #Networking #Observability #Infrastructure

When I started thinking about real-time alerts for my SaaS, my first instinct was Slack. Familiar,...

In the previous article on hosting a Next.js app on a VPS , I'd left the deployment pipeline as a rough sketch: four lines to say "it ships to production on its own when you push." That's the piece I want to open up here, because it's what separates a VPS you fuss over by hand from infrastructure you can forget about. There's a stubborn myth that CI/CD is a big-company thing, with a dedicated DevOps team and six-figure tooling. Not true. The pipeline that deploys this portfolio fits in two YAML files, you can read it in five minutes, and it gives me back exactly the comfort I liked about Vercel: I push to master , I go grab a coffee, the app is live when I'm back. The one thing I gained along the way is knowing precisely what happens between the git push and the running container. Four steps, in this order Deployment is a chain. On every push to master , GitHub Actions runs lint, security scan, image build, and deploy. What matters is the needs : as long as a step fails, the following ones don't start. A critical vulnerability caught by the scan, and the image never gets built. At all. jobs : lint : # ESLint runs-on : ubuntu-latest # ... security : # Trivy scan (reusable workflow) uses : ./.github/workflows/security.yml build-push : # build the Docker image → push to GHCR needs : [ lint , security ] # ... deploy : # SSH to the VPS → docker compose pull && up -d needs : [ build-push ] # ... Lint first, because it's the cheapest step and there's no point building an image if ESLint is already screaming. The scan next, as a barrier. Then the build, which produces the Docker image and pushes it to GHCR, GitHub's container registry (private, in my case). And finally the deploy, which connects over SSH to the VPS, pulls the new image and restarts the container. Four links, each blocking the next. That's the whole secret. The security scan is in the path, not in a review "for later" This is the one I won't budge on. Dependency security, in a lot of projects, is a Dependabo

Vercel m'a longtemps convenu. Tu pousses ton code, trente secondes plus tard c'est en ligne avec un certificat valide, un CDN et des previews par branche. Pour démarrer un projet, je ne connais rien de plus confortable. Le problème arrive après, quand le projet vit. La facture grimpe avec le trafic et les fonctions serverless, certaines fonctionnalités propriétaires deviennent compliquées à reproduire ailleurs, et tu finis par ne plus vraiment savoir où ni comment ton app tourne. C'est un excellent point de départ, et un piège dès qu'on veut maîtriser son coût et son infra. Pour ce portfolio comme pour plusieurs projets clients, j'ai pris le chemin inverse. Un VPS à quelques euros par mois, une image Docker, un reverse proxy, un pipeline maison. L'idée n'est pas de revenir à l'âge de pierre du déploiement par FTP : je garde le « git push et c'est en ligne », mais sur une machine que je contrôle de bout en bout. Voici comment c'est câblé, et les deux ou trois endroits où je me suis fait avoir. L'image Docker : tout repose sur le mode standalone La pièce qui change tout, c'est output: "standalone" dans next.config.ts . Au build, Next trace exactement les fichiers nécessaires au runtime et les recopie dans .next/standalone/ . On passe d'une image d'environ 1 Go à environ 200 Mo. Sans ça, tu traînes tout node_modules dans ton conteneur de prod pour rien. Le Dockerfile est multi-stage : une étape pour installer les dépendances, une pour builder, une dernière qui ne garde que le strict nécessaire. # deps : installe les dépendances (cache Docker optimal) FROM node:22-alpine AS deps WORKDIR /app COPY package.json yarn.lock .yarnrc.yml ./ RUN corepack enable && yarn install --immutable # builder : build l'app FROM node:22-alpine AS builder WORKDIR /app COPY --from=deps /app/node_modules ./node_modules COPY . . RUN corepack enable && yarn build # runner : image finale, non-root FROM node:22-alpine AS runner WORKDIR /app ENV NODE_ENV=production RUN addgroup -g 1001 nodejs && a

Legacy ETL modernization is often described as a conversion exercise: Informatica mapping in. Snowflake SQL out. That framing is incomplete. A real migration is not only about translating expressions. It is about preserving transformation intent, identifying what is missing, documenting assumptions, validating target behavior, and ensuring that someone is accountable for decisions before generated artifacts are released. I have been building a prototype called Data Engineering Copilot around that idea. The latest capability starts from an Informatica PowerCenter XML export and produces a governed Snowflake migration delivery packet. The workflow is: Informatica PowerCenter XML ↓ Metadata and Lineage Extraction ↓ Canonical Metadata Model ↓ Snowflake Artifact Generation ↓ Validation and Migration Risk Assessment ↓ Human Review and Approval ↓ Governed Release Package The problem with simple code conversion An Informatica mapping can contain far more than a direct field-to-field relationship. A typical mapping may include: source definitions and target definitions source qualifiers and filters expression transformations reusable transformations lookups constants and default values mapping parameters target load order connector-level lineage update strategy or sequence-generation behavior target fields with no visible incoming connector A generator that only reads source and target columns may produce SQL that looks valid but does not preserve the original delivery intent. That is risky. For example, imagine a target field that has no visible source column. It may still be populated through: a constant such as 'SOURCE_A' a default such as 'XNA' a surrogate-key lookup a runtime parameter a load timestamp a sequence generator a business decision that was never documented in the mapping If the tool silently inserts NULL , the SQL may compile while the migration is functionally wrong. The prototype approach The Data Engineering Copilot prototype accepts two starting points:

Email authentication has been "solved" on paper for years. SPF, DKIM, and DMARC are old standards, every deliverability guide repeats them, and Google and Yahoo made DMARC effectively mandatory for bulk senders in 2024. So I expected the top of the web to be in good shape. In June 2026 I ran SPF, DKIM, DMARC, and MTA-STS checks across the Tranco top 10,000 domains, using public resolvers (1.1.1.1 and 8.8.8.8) and the same checks my own tool runs. The records are public DNS, so anyone can reproduce this. The picture is worse than the "solved problem" framing suggests, and the interesting part is not adoption, it is where people stop. A third of the top 10k still have no DMARC 3,318 of the 9,937 domains that resolved (33.4%) publish no DMARC record at all. These are not obscure sites, they are the most-visited domains on the web. Without DMARC a receiver has no published instruction for what to do when SPF and DKIM fail, and you get none of the aggregate reporting that tells you who is sending as you. It does get better at the very top. Among the top 1,000 domains, 28.4% have no DMARC, versus 34% across the rest of the 10k. Better, not good. The real problem is p=none, not missing records This is the number that actually matters. Of the 6,619 domains that do publish DMARC, only 46.5% are at p=reject . About a quarter (26%) are still sitting at p=none . p=none is monitor-only. It asks receivers to report what they see and to enforce nothing. It is the correct first step: publish p=none , collect aggregate reports, fix the sources that should be passing, then tighten the policy. The trouble is that p=none is also where most deployments quietly stop. The reports start arriving, nobody reads them, and the domain sits unprotected behind a policy that does nothing while looking like progress. Moving from p=none to p=reject is the step that turns DMARC from a dashboard into a defense, and it is the step most people never finish. I wrote up the safe way to make that move , si