标签：#gcp

Google just made multi-region Cloud Run significantly easier. Here is the full picture; what changed, what it means in practice, and how to build it right. Most teams discover they need multi-region architecture the hard way and sadly, during an outage. Whether you're running a global e-commerce platform, a real-time gaming API, or a financial services application, users expect your service to be available whenever they need it. There is a conversation that happens in almost every engineering team at some point. It usually starts with a post-mortem. A regional Google Cloud outage or a Cloud Run service that hit a cold start spike, or a single-region deployment that could not handle the latency demands of users spread across Lagos, Nairobi, and London simultaneously, caused enough pain that someone finally asked: why are we only deployed in one region? The answer is usually one of three things: it felt complex, it felt expensive, or no one had prioritised it yet. In July 2026, Google moved Cloud Run Service Health to General Availability and the timing was hard to miss. Six days earlier, a power cut at Google's Netherlands data centre had knocked three services offline. The GA release brings automatic cross-region failover to Cloud Run with what Google describes as a two-step setup: add a readiness probe, set minimum instances to at least 1. The load balancer does the rest. This article covers the full architecture, what Service Health is, how readiness probes underpin it, how to set up the Global Load Balancer correctly, and how to test that failover actually works. It also covers the production details. What changed: Service Health and readiness probes Before Service Health, multi-region Cloud Run required you to implement a /health endpoint in your application and configure a separate HTTPS health check at the load balancer level. This worked, but it had a significant gap. The load balancer's health check only knew whether the Cloud Run service endpoint was respon

Using Firestore as "the app's primary database" is easy at first. Writes and reads complete in one place, and onSnapshot gives you realtime push for free. But as a service grows, keeping Firestore as the SoT (source of truth) exposes some fatal constraints. Where SoT-Firestore hurts Weak transaction boundaries, missing complex queries, the cost structure, the difficulty of migrating away. The harshest one: you cannot narrow related updates with a WHERE . For the use case "update a set of documents matching a condition, consistently, in one go," Firestore is structurally weak. Redefining it as a public cache In one project, I made Cloud SQL the SoT and treated Firestore as a "read-optimized projection." Writes go through the backend (Next.js / Cloud Run), and the SoT (Cloud SQL via Data Connect) and Firestore are both updated within the same write path . Separately, a Cloud Run reconciliation job runs and checks and repairs consistency against the SoT as authoritative. This reconciliation job is enqueued at the start of the request, before the DB updates. Because it's queued first, even if the write dies halfway, the consistency check always runs afterward and repairs the state. From the client's perspective, Firestore is a rebuildable cache — in the worst case it can be rebuilt from the SoT. // Writes go through the backend. The backend updates both the SoT // (Data Connect → Cloud SQL) and Firestore. Clients never write Firestore directly. await backend . updateEntity ({ id , title }); // Changes are pushed back in realtime via Firestore's onSnapshot unsubscribe = onSnapshot ( entityRef , ( snap ) => render ( snap . data ())); Benefits and costs The benefits are clear. The SoT side (SQL) brings transactional consistency and complex queries; the read side (Firestore) brings freedom in data modeling and realtime push; and consistent syncing with external systems (OpenSearch / BigQuery, etc.) coexists naturally. Since the backend updates Firestore at write time, the f

In production we have ~1.5 TB of full-text court decisions and their vector embeddings, plus another ~550 GB of other legal data: registries, legislation, business entities, a Spanish case law corpus, EU-Lex. If we take this corpus and train an MoE model the size of DeepSeek V3, scaled to 860B parameters, on GCP — what comes out? We break down the dataset, architecture, compute cost, and the properties such a model would have on Ukrainian law. What's in the Dataset The entire corpus is what's already running in SecondLayer's production. No extra scrapes, no Common Crawl, no noise. EDRSR — the dataset core, ~1.5 TB. The Unified State Register of Court Decisions of Ukraine. 96.2 million full-text decisions (1,079 GB in PostgreSQL TOAST), 471 GB of vectors in Qdrant (voyage-3.5, 1024-dim), 28 GB of metadata (court, judge, date, case category, proceeding type, statute code). Breakdown by jurisdiction: civil 33.7M, administrative 14M+, criminal 12M+, commercial 6M+, misdemeanors 6M+. Largest annual cohort — 2024 (115 GB of TOAST text). OpenReyestr — 43 GB. Ukrainian public registries: 16.7M legal entities (EDR), ownership structures (beneficiaries, shareholders), debtors (State Enforcement Service), NAIS registries. This is the foundation for SneakyPiper — our due-diligence platform — but here it serves as raw corpus for the model. Legislation — ~40 GB. The Constitution, major codes (Civil, Criminal, Criminal Procedure, Civil Procedure, Commercial Procedure, Administrative Procedure, Labor, Tax, Customs), laws, and secondary legislation. All structurally annotated: articles, parts, clauses, revision dates with effective-date tracking. This isn't flat text: we know that Article 124 of the Constitution took effect on a specific date, carries particular references, and is cited in a precise number of decisions. Supreme Court review practices + lu_court_decisions — ~25 GB. SC plenary decisions, practice overviews, Grand Chamber rulings. This is the most valuable slice — the

The difficulty If you’re using Google Cloud, you’re probably paying for stuff you don’t need anymore—a server someone forgot to turn off, a storage disk someone forgot to delete, or an IP address someone forgot to release. "That waste is costing us money." It consumes electricity. It generates carbon emissions. Almost nobody tracks those alongside the cost. I wanted a tool that could find both problems simultaneously, without me having to think. So I made one. Live demo: https://greenops-dashboard-845589445410.us-central1.run.app/ Code: https://github.com/raghu-putta/greenops-agent How it works: 4 AI agents in a row Imagine an assembly line in which each worker has only one job: Carbon Scout scans your Google Cloud project and reports on everything that looks like it might be unused or forgotten—idle servers, unattached storage disks, and unused reserved IP addresses. GreenOps Analyzer takes that list and adds two numbers to each item: how much it’s costing you in dollars, and how much it’s costing the planet in carbon emissions. Optimization Executor takes each finding and makes it an action: stop this server, delete this disk, release this address. It shows you the plan first; it only changes if you say go. Report Generator rolls up the entire run into a downloadable report so you (or your boss or your sustainability team) have something nice to look at. Powered by: FastAPI (the web framework), Google’s Agent Development Kit (a library for building AI agents), Gemini 2.5 Pro (the AI model doing the reasoning), and Google Cloud Run (where it’s all running). The “bug” that was not a bug: The dashboard has a terminal-style live window that shows you what each agent is doing in real time, using a technique called Server-Sent Events (basically, the server streams updates to your browser one at a time instead of making you refresh). And then at some point that window just stopped showing anything. Nothing. My backend logs told me that the updates were still being sent,

It's important to monitor Dataform - jobs executed by Dataform can be the primary source of BigQuery costs in a modern data platform. Forgetting to incrementalise a table, using a table instead of a view in the wrong place or performing complex window functions on a large table can all incur large costs and long run times. Using the WorkflowInvocationAction for each job we can extract its BigQuery Job ID, then extract key metadata for each BigQuery job by querying INFORMATION_SCHEMA.JOBS_BY_PROJECT , before writing the output back to BigQuery so that it can be analysed (maybe even by transforming it in Dataform). from google.cloud import dataform_v1 from google.cloud import bigquery from datetime import datetime # ------------------------------------------------------------ # CONFIG # ------------------------------------------------------------ PROJECT_ID = " my-project " REGION = " europe-west2 " REPOSITORY_ID = " analytics " WORKFLOW_INVOCATION_ID = " 123456789 " BQ_REGION = " region-europe-west2 " OUTPUT_TABLE = " my-project.raw_dataform_monitoring.raw_dataform_bigquery_metrics " # ------------------------------------------------------------ # CLIENTS # ------------------------------------------------------------ dataform = dataform_v1 . DataformClient () bq = bigquery . Client ( project = PROJECT_ID ) repository = dataform . repository_path ( PROJECT_ID , REGION , REPOSITORY_ID ) invocation_name = f " { repository } /workflowInvocations/ { WORKFLOW_INVOCATION_ID } " # ------------------------------------------------------------ # 1. GET WORKFLOW INVOCATION ACTIONS → EXTRACT JOB IDS # ------------------------------------------------------------ job_ids = set () actions = dataform . list_workflow_invocation_actions ( parent = invocation_name ) for action in actions : # only BigQuery actions contain job metadata if hasattr ( action , " bigquery_action " ) and action . bigquery_action : if action . bigquery_action . job_id : job_ids . add ( action . bigquery_action

Building a Benchmarking Agent with A2A and MCP This tutorial aims to build and test benchmarking Agents using the A2A protocol across several mainstream Cloud providers. A Master Orchestrator Agent is exposed via MCP to allow Antigravity CLI to be used as a MCP client to co-ordinate the benchmarks. Deja Vu — What is Old is New! This paper is a re-visiting of the original benchmark series with Gemini CLI over Node, GO, and Python: Cross Language A2A Agent Benchmarking with Gemini 3 and Gemini CLI In this updated version, the Antigravity CLI is used to push Rust Agents cross-cloud and co-ordinate Mersenne Prime Calculations. Why would I need Multi-Cloud Support? And Rust? Can’t I just use Python? Most mature Agent development tools and libraries are Python based. Python allows for rapid prototyping and evaluation of approaches. Python is also an interpreted language- which has trade-offs in memory safety, and performance. Other languages like GO and Rust offer high performance and memory safe operations. With a language neutral communication protocol — the actual Agent implementation of each Agent can be coded in the most appropriate language. What is this Approach actually Benchmarking? The high level goal was to measure the actual time spent running an algorithm in the native language code inside the A2A agent. Each language had a slightly different implementation due to the language syntax. After running the algorithm- each Agent was instructed to calculate and return the elapsed time for cross cloud comparison. What is the A2A protocol? The Agent2Agent (A2A) protocol, an open communication standard for AI agents, was initially introduced by Google in April 2025. It is specifically engineered to facilitate seamless interoperability within multi-agent systems, enabling AI agents developed by diverse providers or built upon disparate AI agent frameworks to communicate and collaborate effectively. A good overview of the A2A protocol can be found here: A2A Protocol Lan