标签：#partition

The decision The partition function separates the array into the elements that satisfy the condition from those that do not docs . The filter function iterates over an array and applies an expression that returns matching values docs . Both scripts compute the task to split payload.records into accepted ids and rejected records with a reason, plus a retry count, by an email match. First approach The partition once approach uses the import line import * from dw::core::Arrays . %dw 2.0 import * from dw::core::Arrays output application/json var split = payload.records partition (r) -> (r.email default "") matches /.+@.+\..+/ --- { accepted: split.success map (r) -> r.id, rejected: split.failure map (r) -> { id: r.id, reason: "missing or invalid email" }, retryCount: sizeOf(split.failure) } Second approach The filter twice approach requires no import. %dw 2.0 output application/json var valid = payload.records filter ((r) -> (r.email default "") matches /.+@.+\..+/) var invalid = payload.records filter ((r) -> not ((r.email default "") matches /.+@.+\..+/)) --- { accepted: valid map (r) -> r.id, rejected: invalid map (r) -> { id: r.id, reason: "missing or invalid email" }, retryCount: sizeOf(invalid) } Same input, same output { "records" : [ { "id" : "ORD-1001" , "email" : "ana@example.com" }, { "id" : "ORD-1002" , "email" : "bad-address" }, { "id" : "ORD-1003" , "email" : "raj@example.org" }, { "id" : "ORD-1004" }, { "id" : "ORD-1005" , "email" : "mei@example.net" } ] } Both scripts print the same output for this input. { "accepted" : [ "ORD-1001" , "ORD-1003" , "ORD-1005" ], "rejected" : [ { "id" : "ORD-1002" , "reason" : "missing or invalid email" }, { "id" : "ORD-1004" , "reason" : "missing or invalid email" } ], "retryCount" : 2 } Measured Input Records Script Runs Min ms Median ms Max ms Source small 5 partition once 10 77 84 94 verified in sandbox large 50000 partition once 10 378 383 388 verified in sandbox small 5 filter twice 10 52 55.5 62 verified in sandbox

Netflix engineers introduced dynamic partition splitting for Cassandra to address wide partitions in time series workloads. The metadata-driven approach detects oversized partitions, splits them smaller units, and routes reads across child partitions. Netflix reported lower read latency from seconds to milliseconds, reduced timeouts, and improved cluster stability while maintaining transparency. By Leela Kumili

Cloudflare recently described how a slowdown in its billing pipeline was traced to contention inside the query planning stage of ClickHouse. The team profiled the bottleneck and patched ClickHouse to replace an exclusive lock with a shared lock, drop the per-query copy of the parts list, and improve part filtering. By Renato Losio