标签：#disc

If you’ve ever tried to reconcile a night of sleep from an Oura Ring , a morning run from a Garmin watch, and active minutes from an Apple Watch , you know the "Dirty Data" struggle is real. Each platform has its own schema, its own definition of "active calories," and its own idiosyncratic export format. In the world of Data Engineering , this is a classic multi-source integration problem. But you don't need a massive Snowflake cluster to solve it. Today, we’re building a high-performance, serverless data pipeline to clean and normalize wearable data using DuckDB , dbt , and GitHub Actions . By leveraging a modern Serverless Data Pipeline and DuckDB's lightning-fast processing, we can turn a mess of CSVs into a structured Parquet -based personal data warehouse. The Architecture: From Chaos to Clarity Before we dive into the code, let’s look at how the data flows from your wearables to a clean, queryable state. graph TD A[Oura JSON] -->|Python Ingestion| D[(DuckDB Raw)] B[Garmin CSV] -->|Python Ingestion| D C[Apple Health XML] -->|Python Ingestion| D D --> E{dbt Models} E -->|Cleaning| F[stg_models] E -->|Normalization| G[int_health_metrics] G -->|Final Output| H[Gold Layer: Parquet Files] H --> I[Visualization / BI] subgraph GitHub Actions D E F G H end Prerequisites To follow along, you'll need: DuckDB : The "SQLite for OLAP" that makes local analytical processing insanely fast. dbt-duckdb : The adapter that lets dbt talk to DuckDB. GitHub Actions : Our free "orchestrator." Tech Stack : DuckDB, dbt, Python, Parquet. Step 1: The Ingestion Layer (Python + DuckDB) The first hurdle is getting disparate files (JSON, CSV, XML) into a unified storage format. DuckDB is magical here because it can query these files directly. We'll use a simple Python script to load these into a local .duckdb file. import duckdb def ingest_raw_data (): # Initialize the database con = duckdb . connect ( ' health_data.duckdb ' ) # Ingest Garmin CSV con . execute ( """ CREATE TABLE raw_garmin

🚀 Technical Briefing: This tutorial is part of our deep-dive series on Agentic Workflows at Gate of AI . For the full technical breakdown, interactive code sandbox, and the native Arabic translation, visit the original article here . An evaluation-first guide to deciding whether GPT-4o mini fine-tuning is justified for a narrowly defined language task. This article uses the available research context rather than assuming unverified API capabilities, model snapshots, pricing, or deployment features. GPT-4o Mini Fine-Tuning: Start With Evidence, Not an Upload Fine-tuning is often presented as the next step after prompt engineering, but the available evidence does not support treating it as an automatic upgrade. Before preparing a dataset or committing to a training workflow, define the task, establish a baseline, select measures that reflect the real objective, and decide what result would justify changing the system. The verified research context is especially relevant for text transformation. A TREC 2024 Plain Language Adaptation of Biomedical Abstracts study evaluated prompt engineering, a two-AI-agent approach, and fine-tuning with OpenAI GPT-4o and GPT-4o mini models. Its objective was to simplify biomedical abstracts for a K-8 audience, approximately 13- to 14-year-old students. The study used qualitative assessments for simplicity, accuracy, completeness, and brevity on 5-point Likert scales, together with readability measures including Flesch-Kincaid grade level and the SMOG Index. Its results are a useful warning against simplistic claims. Prompt engineering with GPT-4o mini and the two-agent approach showed stronger qualitative performance in that evaluation. Fine-tuned models excelled in accuracy and completeness, but were less simple. The paper also reported that GPT-4o mini prompt engineering outperformed the evaluated iterative two-agent and GPT-4o fine-tuning approaches on its qualitative results. That is not a universal verdict on fine-tuning. It is ev

Within minutes of this post going live, this laptop goes back to the company. Work is handed over....

Welcome to this week's Top 7, where the DEV editorial team handpicks their favorite posts from the previous week (Saturday-Friday). Congrats to all the authors that made it onto the list 👏 The Next Evolution of Software Developers Shifts junior training and trust Roberto B. Roberto B. Roberto B. Follow Aug 12 The Next Evolution of Software Developers # ai # learning # mentorship # software 62 reactions 29 comments 7 min read @robertobutti argues that AI isn't replacing developers so much as raising the abstraction layer they work at, the same way high-level languages, frameworks, and cloud platforms did before it. Drawing on their own shifting workday, they make the case that as implementation gets cheaper, ownership of architecture, trade-offs, and outcomes becomes the real differentiator. The End of Undetectable AI Text? Claude’s New Watermark Explained Separating AI provenance from detection myths Sylwia Laskowska Sylwia Laskowska Sylwia Laskowska Follow Aug 11 The End of Undetectable AI Text? Claude’s New Watermark Explained # news # ai # llm 138 reactions 93 comments 6 min read @sylwia-lask cuts through the panic around Anthropic signing the EU AI Act transparency code, separating what is actually confirmed about Claude's new text watermark from the myths that piled up within hours. They also flag the question nobody has answered yet: how do you statistically watermark generated code, where the model has far fewer equivalent tokens to choose from? I Stopped Trusting AI Agents With Tools. So I Built a Gatekeeper. 83 real agents tested zero mocks Debashish Ghosal Debashish Ghosal Debashish Ghosal Follow Aug 13 I Stopped Trusting AI Agents With Tools. So I Built a Gatekeeper. # ai # agents # security # gatekeeper 43 reactions 48 comments 14 min read @debashish_ghosal built a contextual permission engine that sits between an agent and its tools, replacing binary allow-or-deny with four decisions and a deterministic policy the LLM can't talk its way past. After fiel

The Linux kernel, Fedora, and LLVM now require an “Assisted-by” tag on patches created with the help...

WLOADCTL is built as a distributed scheduling platform composed of multiple cooperating processes. Communication between different nodes, such as: Server ↔ Agent Server ↔ Client is handled through TCP/IP socket communication. However, communication between components on the same node relies heavily on Linux Inter-Process Communication (IPC) mechanisms, including: Message Queues Shared Memory Semaphores In some environments, the default Linux IPC configuration may not be sufficient for high-volume scheduling workloads. When this happens, WLOADCTL may encounter message queue-related errors or communication bottlenecks. This article explains how to: Check current IPC limits Increase message queue capacity Inspect IPC resource usage Remove unused IPC resources Understanding Current IPC Limits Before making any changes, it is important to inspect the current IPC configuration. Use: ipcs -l This command displays the system-wide limits for IPC resources, including: Maximum number of semaphore sets Maximum number of semaphores Maximum message queue size Maximum shared memory limits Pay special attention to the Message Limits section. Example: ------ Messages Limits -------- max queues system wide max size of message (bytes) default max size of queue (bytes) If the value of: default max size of queue (bytes) is around: 16384 the queue capacity may be too small for larger scheduling environments. Increasing Message Queue Capacity If the current limits are low, we recommend adjusting the Linux kernel IPC parameters. As the root user, edit: /etc/sysctl.conf and add the following settings: kernel.msgmni=1600 kernel.msgmax=8192 kernel.msgmnb=1638400 Parameter descriptions: Parameter Description Typical Default Recommended msgmni Maximum number of message queues 16 1600 msgmax Maximum size of a single message (bytes) 8192 8192 msgmnb Maximum capacity of a message queue (bytes) 16384 1638400 In WLOADCTL, a typical internal message is approximately: 512 bytes After modifying the con

Well, first real job and I still haven't gotten used to waking up for a 6:30 AM shift... I've been...

So a bit of context, I've been doing testing for V.E.L.O.C.I.T.Y. Drone and initially, I just copied over a binary, but I wanted it to be a bit easier to setup, so I thought I'd make an installer for it, so it can register as a system tray app. So naturally, I looked up what's the best installer and up popped Inno Setup 7. So I used it and it worked fine I guess, then I saw they apparently charge $155 for individuals, up to $1195 for unlimited users and that locks you to a version, if you want a new version, you need to buy a new license... So per my usual, I built my own. It's smaller (tool), faster and completely cross-platform, using zstd with adjustable compression ratio, dependency checking, bundling, CI/CD updating, Delta-Updating, adding MSI compliance for managed deployments too and a few more nice to have features. I'm releasing it under Apache 2.0, so it's actually free and completely open-source, use it commercially, start the next Microsoft and release a billion copies using it, you don't owe me a penny. My reason for creating it, is like so many other times, I found that the industry gatekeeps actually making money out of software, tooling should be free, so the real products can be made, without any hidden fees. So my question to everyone is, do you need an installer? And have you been burnt in the past by hidden costs from 'open-source' releases that charge you once you hit a revenue floor?

Meme Monday! Today's cover image comes from the last thread . DEV is an inclusive space! Humor in poor taste will be downvoted by mods.

You heard me. Alright, that was mean lol. Though based on the title, you probably already knew the...

Every programmer wants to improve. We all dream about becoming the person who can look at a...

"our work got checked by an external reviewer." That's what Fable said after I brought Gemini...

I recently learned something about building products that I probably should have understood much earlier: People liking your product doesn't necessarily mean they need it. I built a platform called Rizzzler, an open-source profile/link-in-bio platform. The idea was pretty simple. I'd seen people using platforms where they could put a link in their social media bio and create a small personal page. I thought I could build my own version — something simple, fast, customizable, and a little more fun. So I built it. And because I wanted people to be able to trust what they were using, I made the project open source too. I spent a lot of time building the actual product. There are profiles, customization, coins, notifications, milestones, community chat, and other small systems intended to make the platform feel less like a static link page and more like something people could actually interact with. At that point, I thought: "Okay, now I just need people to find it." That turned out to be the easy part. Then I started promoting it. I submitted Rizzzler to places like Product Hunt, SaaSFrame, and other platforms where people discover new products. And for a few days, things actually looked pretty good. I started getting visitors. At one point, the traffic was above the 25th percentile for the category I was looking at in GA4. People were visiting. Some people signed up. And I started getting feedback like: "Good UI." "This is good." "Someone finally made link-in-bio profiles look cool." Those comments felt great. They also gave me a slightly dangerous impression: Maybe I've built something people actually want. Then the traffic stopped. Not gradually. It just became cold again. The initial spike from launching and posting about the product disappeared, and there wasn't enough organic interest to keep bringing people back. That was the part I didn't expect. The product wasn't necessarily bad. This is something I've been thinking about a lot. I don't think the main problem

We’ve all been there: staring at a delicious plate of pasta, trying to figure out if it's 400 or 800 calories. Manual tracking is a chore, and standard apps often fail at portion estimation. But what if we could combine Computer Vision , Multimodal LLMs , and Vector Databases to build an automated nutritionist? In this tutorial, we are building a state-of-the-art Multimodal Food Estimation Pipeline . By leveraging the Segment Anything Model (SAM) for precise boundary detection and GPT-4o Vision for contextual analysis, we can bridge the gap between "looking at a photo" and "calculating nutritional density." Whether you're interested in AI-driven wellness , FastAPI development , or Multimodal RAG , this guide covers the full stack. The Architecture 🏗️ The pipeline follows a sophisticated "Identify -> Analyze -> Match" flow. We don't just ask GPT-4o "what is this?"; we use SAM to isolate food items first to ensure the LLM focuses on the right pixels. graph TD A[User Uploads Image] --> B{SAM Model} B -->|Segmentation| C[Isolated Food Patches] C --> D[GPT-4o Vision API] D -->|Item + Volume Est.| E[Embedding Generation] E --> F[PostgreSQL + pgvector] F -->|RAG Retrieval| G[Verified Nutritional Data] G --> H[Final Response: Calories & Macros] Prerequisites 🛠️ Before we dive in, make sure you have the following ready: Python 3.10+ OpenAI API Key (for GPT-4o) PyTorch (for SAM) PostgreSQL with the pgvector extension enabled FastAPI for the backend Step 1: Precise Segmentation with SAM 🎯 The biggest challenge in food AI is overlapping items. Using Meta’s Segment Anything Model (SAM) , we can extract the exact mask of a food item, which helps in calculating the relative "area" occupied on the plate. import torch from segment_anything import sam_model_registry , SamPredictor import cv2 # Load SAM model sam_checkpoint = " sam_vit_h_4b8939.pth " model_type = " vit_h " sam = sam_model_registry [ model_type ]( checkpoint = sam_checkpoint ) predictor = SamPredictor ( sam ) def get_f

Intro AI Avatar is a free app where your VRoid (VRM) avatar cheers you with all its might .🤗 It lives in your VS Code sidebar (reacts to Claude Code / GitHub Copilot) or browser side panel (reacts to ChatGPT / Claude). Animations and speech bubbles all run without AI too. This time I have three main topics. 🤝Catbot Integration 🏢AI Office 😺Cat Mode Let's see how they are! Catbot Integration I was asked to collaborate with my DEV Community friend @annavi11arrea1 Catbot . Catbot is A galactic robot cat you can talk to from any device — and a harness that lets you switch between (or combine) all of your AI models. https://github.com/AnnaVi11arrea1/catbot I was happy about this offer because I loved Anna's creativity and cool designs. I added the features below to AI Avatar to integrate Catbot. Launch Cat button: With this button, AI Avatar can run Catbot. Catbot with button: This makes Catbot stay beside AI Avatar. Cat Boss button: This changes the AI Office boss from a VRM avatar to Catbot. Cat Mode Many people feel that animals are healing and soothing. It is close to the AI Avatar concept of cheering people up. So I decided to add Cat Mode . I added the features below to make it look like a cat. Cat-like text, "Meow/Purrr" in English and "にゃ～" in Japanese Cat emojis Cat pose animations A new avatar with cat ears and cat whiskers. To tell the truth, the hardest part of making this mode was adding whiskers to the avatar using Blender . I can do basic things in Blender, but it is too difficult for me, even with the help of AI, just to add whiskers. It would be more fun if I added other animal modes too. AI Office AI Avatar displayed only one avatar. I thought it could do more things if it displayed several avatars at once. So I added AI Office mode. Two avatars are displayed and talk and move around when idle, and they also make a communication animation when using AI or clicking. I made one avatar a boss and one a worker. The hard part of making this mode was the timin

When many people are first learning how to code, they find it difficult and when they ask, "How can I get better at coding?" they are usually told, "With time and practise it will get easier." . But instead of using so much time and effort to get better at coding using hard programming languages, what if coding could get better for you instead of you getting better at coding ? Well, this is the reason that inspired me to make a new programming language called Sofya . Sofya is designed to be so simple (even simpler than Python ) so that anyone can find programming easy and fun. But to prove my point, let us use an example. Let us say that we want to make a program that will show us all the numbers from 1 to 20 . Let us compare how this program will look like in Python and Sofya . The Python Program for number in range ( 1 , 21 ): print ( number ) The Sofya Program Variable Number is 0 Do this { Increase Variable[Number] by 1 Write Variable[Number] on the screen } Until Variable[Number] = 20 From this example, we can see that the Sofya program is easier than the Python program, for a beginner in programming, for the following reasons: Sofya uses simpler commands than Python: It is easier for a beginner in programming to remember the command Do this...Until Variable[Number] = 20 , which is used for making a loop, as compared to the command for number in range(1, 21): . Sofya's syntax is closer to English as compared to Python's syntax: When we are making a loop variable in Sofya, we simply say Variable Number is 0 rather than saying number in range(1, 21) in Python. The Sofya program can easily be understood by anyone even if it is the first time that they are seeing it as compared to Python: A beginner in programming can easily tell that in the line where we say Increase Variable[Number] by 1 , that we are increasing the value of the variable called 'Number' by 1 as compared to the line number in range(1, 21) in Python. If you would like to try out Sofya for yourself

In mid-July, my Google clicks in my home market (Israel) dropped by almost half. Buyer-intent queries that used to bring steady leads just evaporated from Search Console. While I was staring at GSC dashboards trying to figure out what broke, I finally did the thing I should have done months earlier: I stopped looking at dashboards and started reading raw server logs. What I found there was a parallel universe. Google Search was sending me less than ever — but AI systems were reading my site constantly . Not "someday this will matter" constantly. Right-now constantly: an AI assistant was fetching a page of mine roughly every 26 minutes, around the clock, because a real human had just asked it a question. So I built a small log analyzer and let it run. Here's what 34 days of complete Caddy logs from a small business site (about 70 real human visitors a day) actually look like. The numbers All counts are HTTP 200 responses only (more on why below), over 34 days: Bot Requests Per day What it is bingbot 5,444 158.2 Bing's index — which feeds ChatGPT ChatGPT-User 1,388 40.3 Live fetch while a human asks ChatGPT Googlebot 1,233 35.8 Classic Google crawl GPTBot 547 15.9 OpenAI training crawler Claude-User 519 15.1 Live fetch while a human asks Claude OAI-SearchBot 281 8.2 ChatGPT search indexing Applebot 268 7.8 Apple (Siri / Apple Intelligence) ClaudeBot 214 6.2 Anthropic training crawler Amazonbot 136 4.0 Amazon (Alexa & co.) PerplexityBot 103 3.0 Perplexity indexing Three things in that table genuinely surprised me. ChatGPT-User outreads Googlebot. 40.3 fetches a day versus 35.8. This isn't a crawler building an index for later — ChatGPT-User is the user-agent OpenAI sends when a human is mid-conversation and ChatGPT decides to pull a live page to answer them. On my site, that now happens more often than Googlebot visits. For a tiny business site in a niche market, I did not expect that. Bing crawls 4.4x harder than Google. 158 requests a day versus 36. Nobody optimizes

A known-bad regex is useful evidence, but putting it directly in the test process can hang the runner before the timeout assertion fires. The boundary I am using: run each adversarial case in a fresh worker thread or child process let the parent own a hard timeout and terminate the child keep semantic-parity fixtures separate from timing guards require the safer replacement to pass both suites record the timeout class and bounded elapsed time as evidence Browser workers have the same trap: startup time should not consume the execution budget, and output limits matter alongside time limits. Disclosure: I maintain MonoTools. I recently tightened its browser-local Regex Tester around a 300 ms post-startup Worker budget, named groups, replacement previews, and regression cases: try the bounded tester What does your team treat as a deterministic CI failure receipt for ReDoS: an exit code, a timeout class, an elapsed-time range, or something else?

Hey everyone! 🌸 So, recently I was thinking about something and thought I’d share it with you all. 😄...

Anthropic signed the EU AI Act's Code of Practice on Transparency of AI-Generated Content, and...