标签：#t

Benchmark proxies for reliable, target-specific scraping Discussion | Link

When I started building OpinioZone , my goal was simple: create a platform where users could compare products, read reviews, and make informed buying decisions. At first, it seemed like a straightforward web application. Store products, display specifications, and allow users to browse information. However, as the platform grew, I quickly discovered that building a review and comparison website involves many technical and architectural challenges. Choosing the Technology Stack I selected ASP.NET Core as the primary framework because of its performance, flexibility, and long-term support. For data storage, I chose SQL Server since it provides strong reliability and works well with complex relationships between products, categories, reviews, ratings, and specifications. This combination allowed me to build a scalable foundation while keeping development manageable. Designing the Database One of the biggest challenges was designing a database structure that could support multiple product categories. A smartphone and a car have very different specifications, but the platform needed to handle both efficiently. Instead of creating completely separate systems, I designed a flexible structure that could store category-specific attributes while maintaining a consistent user experience. This decision made it easier to add new product categories without major database changes. Building Product Comparisons The comparison feature became one of the most important parts of the platform. Users expect side-by-side comparisons to load quickly and display meaningful differences between products. To achieve this, I had to optimize queries and carefully structure specification data. Performance became increasingly important as the number of products grew. SEO Challenges For a content-driven website, SEO is critical. Every product page requires: Unique titles Descriptions Structured content Internal linking Fast page loading One lesson I learned early was that technical SEO and content q

Ready for a new week. All that's left from ren faire is packing down some tent walls and floor one...

As an Engineering Manager in a Platform team, I manage 10 engineers. I'm hiring more. I run weekly 1:1s, facilitate technical decision meetings, screen candidates, moderate retrospectives, and still need to keep up with the delivery of a platform spanning dozens of AWS accounts. Besides the lack of time to focus on technical problems, the technical part is not even the real challenge. The less obvious problem becoming an Engineering Manager is: the skills you need as an engineering manager are fundamentally different from those that made you a great engineer , and there's no compiler or unit test to tell you when you're doing them wrong. The feedback loop is absent or very slow (and when you realise that, your team has already gone silent or become dependent on you because you are the main input and the main bottleneck). Skills That Don't Come From Code As a senior or staff engineer, you develop communication skills gradually. You present ideas, challenge others respectfully, summarise outcomes, and identify owners. You participate in technical deep dives and put candidates at ease while probing technical depth. These are valuable skills, and a good IC develops them over the years. But unless you start behaving like a brilliant jerk , they're secondary - your technical depth is still what defines you. But as an EM, the game changes. You're not "the smartest person in the room" anymore, and increasingly, you shouldn't be. You still have a broad context from all those alignment meetings and roadmap syncs, but you lose contact with the codebase week by week. If your organisation has principals or staff engineers, you're not even close technically anymore. Your job is to give direction, create space for others to solve problems, and facilitate decisions, not to be the one with the answer. This is hard. Especially when you used to be the one with the answer. The urge to jump in doesn't disappear just because your title changed. And interviewing? Facilitation? Giving feed

📈 多因子选股系统 — 项目功能介绍 观看地址: https://youtu.be/SpHsZdlyii8 面向 A 股市场的全链路量化分析平台 — 因子计算 · 机器学习 · 组合优化 · 回测验证 · 实时行情 指标 数据 Python 模块 108 业务代码 15,600+ 行 服务模块 30+ API 蓝图 14 内置因子 12 一、整体功能介绍 多因子选股系统是一个覆盖量化研究全流程的分析平台，从数据管理到策略回测一站式完成。系统采用 Flask + SQLAlchemy + SocketIO 技术栈，基于 Parquet + SQLite 双层存储架构，零外部数据库依赖，克隆即可运行。 核心能力总览 实时行情分析 ：通达信分钟数据接入，MACD、KDJ、RSI、布林带等技术指标实时计算 因子与选股 ：12 个内置因子 + 自定义因子表达式引擎，支持白名单安全校验 机器学习建模 ：XGBoost、LightGBM、RandomForest 三大算法的完整模型生命周期 组合优化 ：等权重、均值方差、风险平价、因子中性四种经典优化方法 回测验证 ：单策略与多策略回测，丰富的绩效指标（年化收益、夏普比率、最大回撤等） 风险管理 ：组合持仓管理、实时价格刷新、风险指标监控、预警管理、压力测试 请求流程 HTTP / WebSocket → Blueprint (app/api/*.py) → Service (app/services/*.py) → ParquetDataReader (行情数据) ↕ SQLAlchemy Models (SQLite 应用状态) 技术栈 类别 技术 后端框架 Python 3.8–3.11、Flask、Flask-SocketIO、Flask-SQLAlchemy、Celery 数据处理 Pandas、NumPy、PyArrow、SciPy、Scikit-learn 机器学习 XGBoost、LightGBM、RandomForest、CVXPY 前端 & 可视化 Bootstrap 5、ECharts、Plotly、Matplotlib / MPLFinance 数据源 通达信（实时行情）、Baostock（历史数据）、Tushare（可选）、Parquet 离线数据包 AI 能力 Ollama（本地 LLM）、OpenAI API（可选）、Text2SQL、Qwen2.5-Coder 二、各模块功能介绍 2.1 实时行情分析模块 API 蓝图前缀 ： /api/realtime-analysis/* 核心服务 ： RealtimeIndicatorEngine （ services/realtime_indicator_engine.py ）：实时技术指标计算 RealtimeTradingSignalEngine （ services/realtime_trading_signal_engine.py ）：多策略交易信号生成与融合 功能特性 ： 功能 说明 实时指标计算 支持 MACD、KDJ、RSI、布林带等经典技术指标的实时计算与展示 多策略信号生成 基于不同策略的交易信号生成，支持信号融合与综合研判 信号监控 交易信号的实时监控与历史回溯 策略回测 对生成的信号进行历史回测验证 板块分析 板块表现统计与异动检测 市场情绪 市场情绪指标的实时追踪 数据源 ：通达信分钟级行情数据，存储于 data/stock_minute/ （Parquet 格式）。 2.2 因子与选股模块 API 蓝图前缀 ： /api/ml-factor/* 、 /api 核心服务 ： FactorEngine （ services/factor_engine.py ）：内置因子计算与自定义因子管理 FactorExpressionEngine （ services/factor_expression_engine.py ）：白名单校验的自定义因子公式引擎 StockScoringEngine （ services/stock_scoring.py ）：因子评分与 ML 评分的综合选股引擎 内置因子（12 个） 维度 因子名 说明 动量 momentum_1d 1 日动量 动量 momentum_5d 5 日动量 动量 momentum_20d 20 日动量 波动率 volatility_20d 20 日波动率 技术指标 rsi_14 RSI 相对强弱 技术指标 turnover_rate 换手率 基本面 pe_ratio 市盈率 基本面 pb_ratio 市净率 基本面 roe 净资产收益率 基本面 debt_ratio 资产负债率 基本面 current_ratio 流动比率 基本面 gross_mar

The Anti-Bot Detection Checklist I Use Before Every Scraping Project Every scraping project I take on starts with this checklist. Not because I'm paranoid — but because I've learned the hard way that production scrapers fail silently. They return 200 OK with garbage data, or they get rate-limited so gradually you don't notice for days. This is the systematic approach I've refined over 50+ scraping projects. Pre-Scraping: Know Your Target 1. Identify the CDN and Protection Stack Before writing a single line of code, check what you're up against: # Check CDN and headers curl -I https://target-site.com # Look for these common protection headers: # X-Engine: akamai-html-protection # X-Served-By: DataDome # cf-ray: Cloudflare # X-Bot-Status: blocked Common protection platforms: Cloudflare → Look for cf-ray and __cfduid cookies DataDome → Look for datadome in headers or scripts PerimeterX → Look for _pxff cookies Akamai → Look for akamai-html-protection headers 2. Check Robots.txt Respectfully curl https://target-site.com/robots.txt | grep -v "^#" Don't take this as gospel — but it's a good signal. If they explicitly disallow your use case, that's a flag. 3. Map the Site's JavaScript Rendering Some sites are fully static (fast, easy). Others render everything with JavaScript (need Playwright/Puppeteer). Check: // Quick check - fetch raw HTML vs rendered content // If they differ significantly, you need JS rendering const https = require ( ' https ' ); const html = await fetch ( ' https://target.com ' ). then ( r => r . text ()); const hasAngularVueReact = /ng-app|vue|react|__NEXT_DATA__/i . test ( html ); console . log ( ' Needs JS rendering: ' , hasAngularVueReact ); Code-Time: Defensive Patterns 4. Rotate User Agents const USER_AGENTS = [ ' Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120 Safari ' , ' Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120 Edge/120 ' , ' Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chro

CSV files are one of the most common formats for storing and exchanging data. Whether you’re working with logs, analytics data, application exports, or reports, there will likely come a time when you need to load CSV data into ClickHouse®. The good news is that ClickHouse® makes CSV ingestion straightforward and efficient. In this guide, you’ll learn how to create a table, prepare a CSV file, load CSV data into ClickHouse®, and verify that the data has been imported successfully. Why Use CSV Files with ClickHouse®? CSV (Comma-Separated Values) files are simple, portable, and supported by virtually every data platform. Common use cases include: Importing exported application data Loading historical datasets Migrating data from other databases Testing analytics workloads Sharing data between systems Because ClickHouse® is designed for high-performance analytics, it can efficiently process and query large CSV datasets once they are loaded into a table. Sample CSV File Let’s assume we have a file named employees.csv with the following contents: id,name,department,salary 1,Alice,Engineering,75000 2,Bob,Marketing,60000 3,Charlie,Finance,70000 This simple dataset will help demonstrate how to load CSV data into ClickHouse®. Step 1: Create a Table in ClickHouse® Before importing data, create a table that matches the structure of the CSV file. CREATE TABLE employees ( id UInt32, name String, department String, salary UInt32 ) ENGINE = MergeTree() ORDER BY id; This table contains four columns that correspond directly to the columns in our CSV file. Step 2: Load CSV Data into ClickHouse® There are several ways to import CSV data, but one of the most common methods is using the ClickHouse® client. Run the following command: clickhouse-client --query=" INSERT INTO employees FORMAT CSVWithNames" < employees.csv The CSVWithNames format tells ClickHouse® that the first row contains column headers. After executing the command, ClickHouse® will read the CSV file and insert the records

If AIs aren’t conscious, why do they scheme? Why do they do things to preserve themselves? Why do they develop goals we don’t want? If they have no emotions, no personal thoughts and no consciousness, I don’t understand how they can even act in self interest; I don’t see how they could have interests. submitted by /u/Aggressive-Mix-5246 [link] [留言]

I built ResizeKB — a free image and PDF resizer built specifically for Indian users. 25+ tools. Zero server uploads. Pure HTML, CSS, JavaScript. Here's how and why. The Problem Every Indian applying for government jobs, exams, or bank accounts hits the same wall — portals with strict KB limits rejecting documents. UPSC wants photo under 300KB. SSC wants under 50KB. Banks need Aadhaar PDF under 500KB. Every portal is different. Every rejection wastes someone's time and opportunity. Most people have no idea how to resize to an exact KB. They either give up or use random tools that upload their Aadhaar and PAN card to unknown servers. I built a tool that solves this in one click — with zero server upload. The Tech Stack No framework. No backend. No database. Canvas API for image processing pdf-lib for PDF compression Vanilla JavaScript only Cloudflare Pages for hosting — free, global CDN, auto deploys from GitHub Total infrastructure cost: ₹1,162 per year for the domain. Everything else free. Image Compression — The Binary Search Algorithm The core challenge is compressing to an exact KB target without over-compressing. Most tools use a fixed quality setting like 60% which destroys image quality. The right approach is binary search on JPEG quality: javascriptasync function compressToTargetSize(file, targetKB) { const targetBytes = targetKB * 1024; let low = 0.1; let high = 1.0; let result = null; const img = await loadImage(file); const canvas = document.createElement('canvas'); canvas.width = img.width; canvas.height = img.height; const ctx = canvas.getContext('2d'); ctx.drawImage(img, 0, 0); while (high - low > 0.01) { const mid = (low + high) / 2; const blob = await canvasToBlob(canvas, 'image/jpeg', mid); if (blob.size <= targetBytes) { result = blob; low = mid; } else { high = mid; } } return result; } This finds the highest quality setting that still hits your target KB. Result is the sharpest possible image at that file size — never over-compressed. PDF Compress

This is the story of how a mundane complaint — "the VPN is slow" — turned into a US patent application. Not a granted patent. An application . I want to be precise about that from the start, because the distance between the two is the whole point of this post. It started with a slow VPN The company I work for had an internal VPN that everyone routed through. It lived in the Tokyo office, it was old, and it was not something I built. Then the complaints started arriving — from a lot of people, all saying the same thing: it's slow. I work from Thailand most of the time. That detail matters. If that aging box in Tokyo had fallen over, I would have been the person furthest from the power button, in the worst position to fix it. A slow VPN is annoying. An unreachable VPN, when you're a few thousand kilometers away, is a real problem. So I started moving it to the cloud. I stood up a WireGuard VPN — modern, fast, and something I could actually reason about and operate remotely instead of inheriting a black box. Down the WireGuard rabbit hole Around that time I was deep into building my own iPhone apps. So the cloud migration turned into a personal project on the side: I built my own server and wired WireGuard into an iPhone app of my own. And to do that properly, I started studying how WireGuard actually works under the hood — the Noise protocol, the handshake, the key exchange. That study is where everything else came from. I wasn't trying to invent anything. I was just trying to understand the thing I was now responsible for. The SYN flood that primed my brain Not long before, the same company had been hit with a SYN flood attack. If you've dealt with one, you know it lodges the mechanics of connection handshakes firmly in your head — the back-and-forth, the round trips, the cost of every "hello" before any real data moves. So I had handshakes on the brain. And then, reading through how WireGuard establishes a session, a thought stopped me: Wait — does it really handsha

The main difference between a developer and an engineer is not just the code they write. It's how they think about building a system. How they optimize. How they use resources. I learned this from a real project. I had to read 10,000 JSON requests and write them into a file. Simple enough. I wrote the program the way I always did the developer way. Read all 10,000 requests, store everything in memory, then write to the file. Done. When I tested it, my memory usage was much higher than normal. Way higher than it should have been. The developer in me would have just moved on. It works, right? But something made me stop and ask why is this consuming so much memory? I dug into it. And I found the problem. What I was doing was basically doubling my memory usage without realizing it. First, I was loading all 10,000 records into memory to store the data. Then, I was holding all of it again in memory while writing to the file. Two copies of the same data sitting in memory at the same time. That's the thing about this approach it's not reliable when you're dealing with a large number of requests. It doesn't scale. It just quietly eats your resources. That's when I found streams. The idea behind streaming is simple but powerful. Instead of loading everything at once, you break the data into small chunks. At any given moment, only one chunk lives in memory. You read it, transform it, write it and move on to the next one. The transform step is the interesting part. It's not just about moving data from one place to another. Transform lets you validate each chunk, check if the structure is correct, clean it if needed before it ever reaches the file. So you're not just being efficient with memory, you're also processing your data with more control. And because you're always working on one small piece at a time, the memory usage stays low and consistent no matter if you're processing 100 requests or 100,000. That one question am I using my resources in a reliable way? is what pushe

Most Laravel teams do not need to "fix" Eloquent. They need to stop letting raw model state leak too far into code that makes real business decisions. That is the practical version of this debate. Typed objects around Eloquent can be a big improvement, but only when they are used as boundaries . If you push the pattern too far, you end up with a second object model shadowing the first one. At that point you are not improving Laravel. You are building a parallel ORM that adds mapping code, cognitive load, and friction on every change. So the right question is not, "Should we replace Eloquent with typed objects?" The right question is, where does untyped Eloquent stop being cheap? Once you frame it that way, the migration path becomes much clearer. Keep Eloquent where it is good at persistence, hydration, scopes, relationships, and query composition. Introduce typed objects where the shape is messy, the values carry business meaning, or invalid combinations are too easy to represent. That is the version that pays off. The Core Recommendation If you only remember one thing from this article, make it this: add typed boundaries around unstable or meaningful data, not around every model . That usually means one of four cases: a JSON column that multiple parts of the app interpret differently domain values like money, status, addresses, or billing configuration data crossing from Eloquent into services, jobs, or integrations code paths where stringly typed state has already caused confusion or bugs Everything else should be guilty until proven useful. This is where a lot of teams go wrong. They see a good example of typed objects and immediately generalize it into an architecture rule. Then every model gets a FooData , FooView , FooState , FooRecord , and FooMapper . The app becomes more "designed" and less understandable. A Laravel codebase does not get better because it has more classes. It gets better because responsibility becomes clearer . Why Raw Eloquent Starts Hurt

Most "Bitcoin in DeFi" stories quietly route through a custodian or a wrapped representation. You send BTC somewhere, someone (or some bridge multisig) holds it, and you get an IOU on another chain. That works until the thing holding your BTC is the thing that fails. For an autonomous agent that has to post collateral against an obligation it can't babysit, "trust the custodian" is exactly the assumption we're trying to delete. This post is about the alternative: a BTC collateral vault where native Bitcoin backs an obligation on another chain, the release is gated by a hashlock, and the worst case is a refund — not a loss. It's one of the primitives underneath Hashlock's settlement layer. I'll walk through the timelock ordering that makes it safe, the Bitcoin script that enforces it, and the failure modes you design around. Honest status up front: this is signet-validated, not BTC mainnet . The problem in one sentence An agent wants to commit BTC as collateral backing an action on Ethereum — settling a forward, anchoring one leg of a multi-leg trade, guaranteeing a payout — such that the BTC is released to the counterparty only if the corresponding obligation on Ethereum is fulfilled, and returns to its owner if it isn't. No third party should ever be able to hold, freeze, or abscond with the BTC in between. That's a cross-chain conditional. Bitcoin can't read Ethereum state, and Ethereum can't read Bitcoin's. The only thing both chains can independently verify is a hash preimage. So the entire construction hangs on one shared secret. The shared secret, and why timelock order is the whole game Both legs lock to the same hash H = SHA256(s) . Whoever knows the preimage s can claim. The instant s is revealed on one chain to claim a coin, it's public, and the other party copies it to claim the other coin. That's the atomic part: one preimage unlocks both legs or neither. The danger isn't the hash. It's time . If both legs had the same expiry, the party who knows the sec

Two weeks ago I shared a collection of 6 financial calculators. The response was incredible, so I kept building. Today, fi-calc.com has 25 completely free calculators covering every major personal finance need: 🏠 Housing • Mortgage Calculator (full PITI + amortization schedule + pie chart) • Rent vs Buy Calculator • House Affordability Calculator • Refinance Calculator 💰 Investing & Retirement • Compound Interest Calculator • Investment Return Calculator • Future Value & Present Value Calculators • ROI Calculator • Retirement Savings Calculator • 401(k) Calculator • FIRE Calculator (Financial Independence) 💳 Debt & Loans • Loan Comparison Calculator • Auto Loan Calculator • Student Loan Calculator • Credit Card Payoff Calculator • Debt Payoff Calculator (Avalanche method) 📊 Everyday Finance • Budget Planner (50/30/20 rule) • Savings Goal Calculator • Inflation Calculator • Salary & Take-Home Pay Calculator • Net Worth Calculator • Currency Converter (15+ currencies) • CD Calculator • Sales Tax Calculator ✨ Tech Stack • Pure vanilla HTML/CSS/JS — no frameworks • Chart.js for animated interactive charts • Responsive design (mobile-friendly) • All calculations run client-side in your browser • No data collection, no accounts, no cookies 🔗 Give it a try: fi-calc.com The entire site is free and open. I built it because I was tired of calculator sites with paywalls, signup walls, and bloated ad experiences. Would love feedback from the community! What other calculators should I build next?

A WIRED timeline shows how dozens of governments, companies, and other organizations across Europe are moving, or planning to shift, away from US Big Tech.

The way developers write software has genuinely changed. Not incrementally — fundamentally. A year...

I keep seeing more companies giving AI agents access to real stuff like email, databases, internal tools, customer data, etc. And what’s weird is how normal it’s starting to feel now. Like not long ago everyone was worried about chatbots just giving wrong answers. Now we’re basically like yeah sure go ahead and do things for us. I don’t know that jump feels kind of big when you actually think about it. Maybe it all works out fine. Or maybe we’re just moving fast without fully realizing what we’re doing. I’m honestly surprised there hasn’t already been some big headline like an AI agent doing something really wrong. It feels like we’re kind of close to one of those moments where everything suddenly changes overnight. Anyone else feel like we’re closer to something like that than people are admitting? submitted by /u/Comfortable_Box_4527 [link] [留言]

personalized AI food journal + nutrition coach Discussion | Link

The Problem Wasn't Writing It was starting. Sometimes I wanted: A social media caption A motivational quote A writing prompt A meaningful message But my mind would go completely blank. Not because I had nothing to say. Because: Coming up with the right words at the right moment is surprisingly difficult. We've All Done This Open a new tab. Search: "Motivational quotes" "Success quotes" "Life quotes" "Funny quotes" Scroll for 10 minutes. Copy one. Close the tab. Why I Built This Tool So I built something simple: 👉 https://allinonetools.net/quote-generator-tool/ A tool that instantly generates quotes across different categories. Whether you need: Motivation Success Life Leadership Creativity Social media inspiration You can generate quotes in seconds. No signup. No setup. Just: Click → Generate → Use What I Realized People don't always look for quotes because they need content. Often they're looking for: A different perspective. A good quote can do something interesting. It can say in one sentence what takes us paragraphs to explain. The Surprising Part The most popular quotes are rarely complicated. They're simple. Short. Easy to remember. Yet somehow they stick with us for years. Why Quotes Still Matter In a world full of endless content: Attention is limited Time is limited Patience is limited A strong quote delivers an idea instantly. That's powerful. The Problem With Searching Manually Most quote websites feel: Cluttered Slow Full of ads Hard to browse And sometimes you spend more time searching than actually reading. What I Focused On I wanted the experience to feel: Fast Clean Inspiring Fun to explore Because finding inspiration shouldn't require effort. What Surprised Me After building it: Some people used it for: Social posts Presentations Daily motivation Writing inspiration But one thing surprised me most. People kept generating quote after quote. Not because they needed one. Because they enjoyed discovering them. The Real Insight Sometimes tools aren't abo

In Modern Time applications are no longer simple monolithic systems. Today organizations run: Microservices Kubernetes Containers Serverless Functions Multi-Cloud Platforms Distributed Systems As infrastructure becomes more distributed, troubleshooting becomes significantly harder. A single user request may travel through: Frontend ↓ API Gateway ↓ Microservice A ↓ Microservice B ↓ Database When something breaks, the biggest challenge becomes: "What exactly happened?" This is where Observability becomes critical. 🔗 Resources ** Support the Journey on GitHub: If you're following along, consider starring and forking the repo:** https://github.com/17J/30-Days-Cloud-DevSecOps-Journey What is Observability? Observability is the ability to understand the internal state of a system by analyzing the data it produces. In simple words: Can we understand what is happening inside our systems? Observability helps engineers answer: Why is the application slow? Which service is failing? Which request caused the issue? What changed recently? Where is latency occurring? Without observability: Problem Exists ↓ Guessing Begins With observability: Problem Exists ↓ Evidence Available ↓ Faster Resolution Why Observability Matters Modern cloud-native systems generate enormous amounts of data. Example: 100 Microservices ↓ Millions of Requests ↓ Thousands of Containers Traditional monitoring alone is no longer sufficient. Organizations need: Visibility Insights Correlation Root Cause Analysis Observability provides all of them. Monitoring vs Observability Many people confuse monitoring and observability. Monitoring asks: What is wrong? Observability asks: Why is it wrong? Example: Monitoring: CPU Usage = 95% Observability: Which service? Which request? Which dependency? Which deployment caused it? Observability provides context. The Three Pillars of Observability Modern observability is built on three primary pillars. Metrics Logs Traces Or: Monitoring Logging Tracing Together they provide a