今日已更新 319 条资讯 | 累计 31147 条内容
关于我们

标签:#t

找到 14834 篇相关文章

AI 资讯

Evaluating Agents With an LLM-as-Judge Harness (Without Kidding Yourself About It)

Key Takeaways You can't unit-test a coach agent the way you test a pure function — the output is non-deterministic and "good" is a judgment call, not an assertion. An LLM-as-judge harness lets you grade a whole test set automatically against a rubric, which is the only way solo-scale eval stays sustainable. But the judge is itself a fallible model. If you don't design around its known biases — position, verbosity, self-preference, and quiet drift when the judge model updates — you build a green dashboard that means nothing. The mitigations that actually work are mechanical, not prompt-magic: shuffle order on every pairwise call, pin the judge version, keep a small human-labelled anchor set, and re-check the judge against it. The problem I actually had FamNest's coach agent generates responses to parents — check-ins, encouragement, the occasional gentle redirect. I have a growing pile of these interactions, and every time I change a prompt, swap a model, or adjust the pipeline, I need to know one thing: did I just make it better or worse? For normal code, that's what tests are for. I change something, the suite runs, red or green, done. But there's no assertEqual for "was this an empathetic, useful response to a tired parent." The output changes every run even at temperature zero-ish, and the quality bar is a human judgment, not a fixed string. Two responses can be worded completely differently and both be good. One can match my "expected output" word for word and still be worse than a version that didn't. So the honest options were: read every response by hand every time I change something (does not scale past about week two), or build a harness where a model grades the outputs against a rubric. I built the harness. Then I spent an uncomfortable amount of time learning all the ways a harness like that can lie to you. What the harness actually is At its simplest, it's a loop: def evaluate ( test_cases , coach_agent , judge ): results = [] for case in test_cases : res

2026-07-01 原文 →
AI 资讯

This will get you banned from your ChatGPT subscription

A ChatGPT subscription starts at $20 a month and is one of the cheapest ways to run inference. OpenAI has also been fairly relaxed lately about third-party agents using them , which makes the deal even better for a lot of us. But a subscription can't be used as freely as pay-per-token access , and the providers police the difference. Anthropic recently narrowed its subscriptions to first-party apps; OpenAI has its own limits. Here's what will get you banned from an OpenAI subscription. Sharing your subscription A ChatGPT subscription is strictly personal. One subscription, one user. Sharing yours breaks OpenAI's terms of service. That also covers account pooling and account rotation, where several people share the same credentials to dodge rate limits. Running it in automation Automation (CI, runners, schedulers) should run on per-token pricing, not a subscription . Once a system calls the OpenAI API with your token while you're not in the loop, the usage stops being personal. No unattended production system should run on a ChatGPT subscription. Serving other users For now, you can point an autonomous agent like OpenClaw or Hermes at your ChatGPT subscription, as long as it only talks to you. The moment that agent starts chatting with other people, or serving them in any way, it turns into a team use case , and that inference should be paid per usage. Putting it in a commercial product Same logic here. Making an LLM call authenticated with an individual ChatGPT subscription inside a product you ship breaks OpenAI's terms. That access is subsidized, and reselling it in any form isn't what it's meant for. If you've built something just for yourself and you're the only user, you're probably fine. The bottom line A ChatGPT subscription is personal . Anything that stretches past personal use can get you restricted or banned. If you're not sure your usage counts, move it to pay-as-you-go. If you want to keep the subscription for your own work and fall back to per-token pr

2026-07-01 原文 →
AI 资讯

Scankii: The First Static Security Scanner Built to Stop AI Agents from Leaking API Keys

Hey DevHunt community! 👋 I'm incredibly excited to launch Scankii! As developers, we are building more and more AI Agents using frameworks like LangChain, OpenHands, and AutoGen. The standard paradigm is giving these agents "skills" or "tools" — which are basically just Python functions combined with Natural Language instructions (prompts or docstrings). But here is the problem: Standard secret scanners (like GitLeaks or TruffleHog) are blind to AI-specific vulnerabilities. They only scan source code for hardcoded secrets. But what if your Python code securely loads an API key, and your English instructions accidentally trick the agent into printing that key to stdout? The agent framework captures that output, injects it into the LLM context window, and your secret is suddenly exposed. We call this Cross-Modal Leakage. Enter Scankii. 🛡️ Scankii solves this by analyzing the intersection of your Natural Language and your code. It uses a dual-engine pipeline (NL Semantic Analyzer + AST Syntax Analyzer) to track variable flows between your prompts and your code sinks. ✨ Core Features: Dual-Engine Scanning: Correlates English instructions with Python ASTs. Local-First & Fast: Your proprietary agent tools and code never leave your machine. CI/CD Ready: Outputs standard SARIF reports. Drop it into GitHub Actions or use it as a pre-commit hook. Framework Agnostic: Works with LangChain, AutoGen, CrewAI, MCP, or any custom python agent framework. I built Scankii to give developers peace of mind when scaling their agent toolchains. Security shouldn't be an afterthought when building autonomous systems. I would love for you to try it out on your agent repos, star the project, and leave any feedback or questions below! I'll be here all day answering them. 👇 GitHub Repository: https://github.com/ashp15205/scankii Installation: pip install scankii

2026-07-01 原文 →
AI 资讯

Your Git Commit History Is More Public Than You Think 🕵️‍♂️

In the open source world, we often say that "your code is your resume." However, what many developers forget is that every time they run a git push , they're handing over much more than just lines of code. They're delivering a permanent digital trail that can be tracked, analyzed, and used against them years later. This article is an educational guide about privacy in Git and how the metadata you generate every second can compromise your security and your professional future. The command that reveals your "naked identity" If you want to see exactly what you're sharing with the world, open your terminal in any repository and run the following command: git log --format = fuller Unlike the standard log, the fuller format breaks down the complete anatomy of your contributions. This is where most developers are shocked to see what they're actually leaking: 1. Name and Email (Direct Exposure) Git stores your user.name and user.email locally in every commit. This data: Is harvested by spam bots for marketing databases. Allows recruiters to map all your historical activity, even on projects you no longer represent. Exposes you to doxxing attacks if you contribute to controversial projects. 2. Dates and Timestamps (Time Analysis) Git not only saves the day, but also the exact second and time zone of the authorship and commit. This allows you to create an activity pattern : What time do you usually code? Are you working on personal projects during office hours? What geographical area are you actually in? 3. File Patterns and Metadata In addition to text, if you upload binaries (PDFs or images), these usually contain EXIF ​​metadata (GPS coordinates, camera model, etc.) that Git doesn't clean up by default. The Problem with "Permanent Logs" GitHub is, by design, an accountability platform, not a privacy one. Once a commit enters the public history, deleting it is extremely difficult and often pointless if the repository has already been cloned or indexed by third-party service

2026-07-01 原文 →
AI 资讯

Prepare Application Artifacts To Be Deployed To AWS | 🏗️ Build A Multi-Environment Serverless App

Exam Guide: Developer - Associate 🏗️ Domain 3: Deployment 📘 Task 1: Prepare Application Artifacts To Be Deployed To AWS Before you can deploy anything to AWS, you need to package it properly. This task covers Lambda deployment packaging (zip vs container), managing dependencies, structuring projects for multi-environment deployment, and using AWS AppConfig for runtime configuration. 📘Concepts Lambda Deployment Packaging Options Option Max Size Build Complexity Cold Start Best For Zip Package (inline editor) 3 MB (editor limit) None Fastest Simple functions, no dependencies Zip Package (upload) 50 MB compressed / 250 MB uncompressed Low Fast Most Lambda functions Zip + Lambda Layers 250 MB total (function + all layers) Medium Fast Shared dependencies across functions Container Image 10 GB Higher Slower (first invoke) ML libraries, large dependencies, custom runtimes 💡 If a scenario is about a deployment package exceeding 250 MB, the answer is container images. If it mentions sharing dependencies across multiple functions, the answer is Lambda Layers. Zip is the default for most workloads. Lambda Layers Aspect Detail What They Are Zip archives containing libraries, custom runtimes, or other dependencies Max Layers Per Function 5 Size Limit 250 MB total (function code + all layers uncompressed) Versioning Each publish creates an immutable version Sharing Can be shared across functions, accounts, or made public Path Contents extracted to /opt in the execution environment Dependency Management Strategies Strategy How It Works Pros Cons Bundle In Zip Install deps into package directory, zip together Simple, self-contained Larger package, duplicated across functions Lambda Layers Package deps as a layer, attach to functions Shared across functions, smaller deploys Layer version management, 5-layer limit Container Image Install deps in Dockerfile Full control, large deps supported Slower cold starts, ECR management sam build SAM resolves deps from requirements.txt automatic

2026-07-01 原文 →