标签：#iceberg

Every time this comes up, someone credits the same setup: Athena on Iceberg is where "the code is the spec" — where you open Git and read the whole system, catalog to transforms to schema, without logging into anything. In my experience they're not wrong. What I want to argue is that they're right for mostly the wrong reason. The reason people reach for is the engine — serverless, open format, nothing to provision. But the thing that actually keeps code as the spec, when it does, is something you could have applied to almost any engine. And the thing that breaks it, when it breaks, has nothing to do with Iceberg at all. So here's the split I've landed on, for now: whether "code is the spec" holds is about 90% discipline and 10% engine . Athena/Iceberg earns that 10% honestly — but 10% is all it earns, and I keep watching people mistake it for the whole thing. (Just where my own tinkering has led — not advice.) I should say up front that I'm still in the middle of this. What the best declarative setup for an agent actually looks like — how you turn a system into a spec it can read and act on without guessing — is something I'm actively testing, not something I've settled. Read what follows as a working idea at a particular moment, written down partly so I can find out where it's wrong. To see where the 10% actually lives, it helps to notice that a stateful system always keeps two copies of itself. The spec has two copies One copy is declared : the code you wrote, the schema you committed, the transforms in dbt, the catalog in Terraform. The other is realized : the state the engine accumulates while running — statistics, physical layout, caches, maintenance history, tuning knobs someone set at 2am. "Code is the spec" is really a claim about the distance between those two. When the declared copy explains almost everything about the realized one, you can reason about the system by reading Git. When it doesn't, you can't. Some of that state is declarable — you can pin a

Original Japanese article : Iceberg REST Catalogを直接叩いて、Glue Data CatalogとS3 Tablesの違いを理解する Introduction I'm Aki, an AWS Community Builder ( @jitepengin ). Most of the time, when working with Iceberg tables, we reach for PyIceberg or Spark. I'm no exception, and honestly there were parts of the PyIceberg configuration — rest.sigv4-enabled , rest.signing-name , warehouse — that I understood only vaguely. Iceberg defines a standard called the Iceberg REST Catalog Open API specification , and AWS implements it through two separate endpoints: The AWS Glue Iceberg REST endpoint ( https://glue.<region>.amazonaws.com/iceberg ) The Amazon S3 Tables Iceberg REST endpoint ( https://s3tables.<region>.amazonaws.com/iceberg ) If two implementations follow the same spec, sending the same requests to both and comparing the results should reveal what's actually different between them. In this article, I'll bypass clients like PyIceberg entirely and hit the REST API directly to explore the differences between the two endpoints. To state the conclusion up front: Even though both implement the same Iceberg REST Catalog specification, Glue is designed as an "entry point to multiple catalogs," while S3 Tables is designed as an "entry point to a single table bucket." That difference is visible just by looking at the URL paths. I previously wrote about the relationship between S3 Tables and Glue Data Catalog in another article — worth a read alongside this one: Does Amazon S3 Tables Replace AWS Glue Data Catalog? Understanding Their Relationship What Is the Iceberg REST Catalog? The Iceberg REST Catalog is a specification that standardizes Iceberg catalog operations as an HTTP API. It's published as an OpenAPI definition (YAML), and any catalog that conforms to it can be accessed the same way from clients such as PyIceberg, Spark, and Trino. The key points of the spec are: URL paths follow a pattern like GET /v1/{prefix}/namespaces , where {prefix} is a free-form segment Clients first call

AWS recently announced Amazon S3 Annotations, a feature that lets teams attach rich, searchable context such as summaries, classifications, compliance data, or AI-generated insights directly to S3 objects. Annotations can be updated independently of the object and queried across datasets, reducing the need for separate metadata systems. By Renato Losio

Schema proliferation builds slowly and gets expensive fast. One schema per event type feels right until there are ten tables, union queries spanning all of them, and a single field rename touching every schema. Discriminator-based schema consolidation collapses that to two tables, turning multi-table unions into a single query, while new variants are additive and don't break existing consumers. By Spoorthi Basu