You've Seen the Pipeline. Now Meet the Matrix: The One `Vec ` Behind the 400 Shrink
How a single contiguous allocation — and a type system that won't let you feed strings to a scaler — is the real reason datarust fits in 2.3 megabytes. In the last post I showed you the whole datarust workflow: impute, scale, one-hot, train a logistic regression, evaluate, and save it as JSON — all without a Python runtime in sight. The Docker image shrank from ~900 MB to ~8 MB, and the binary was 2.3 MB. But I skimmed over something important. I kept saying "the flat memory layout" as if it were a detail. It isn't. It's the whole bet. Every scaler, every encoder, every model, every metric in datarust runs on top of one data structure. If you understand that structure — why it looks the way it does and what it refuses to let you do — the rest of the library stops being magic. So let's zoom in. Meet Matrix . Two containers, on purpose Real data is mixed. Numbers in one column, strings in the next. In Python, everything flows through one giant numpy.ndarray or a pandas.DataFrame , and the type system just... shrugs. A string column next to a float column gets coerced into object dtype. You'll find out at training time, in the form of an error message three frames deep. datarust does the opposite. It splits your data into two types at the source: use datarust :: Matrix ; use datarust :: matrix :: StrMatrix ; let numeric = Matrix :: new ( vec! [ vec! [ 3.0 , 85.0 , 24.0 ], vec! [ 12.0 , 70.0 , 31.0 ], vec! [ f64 :: NAN , 95.0 , 45.0 ], ]) ? ; let categorical = StrMatrix :: from_strings ( vec! [ vec! [ "MonthToMonth" ], vec! [ "OneYear" ], vec! [ "MonthToMonth" ], ]) ? ; Matrix is f64 only. StrMatrix is strings only. They are different types , and the compiler will refuse to compile a program that hands a string column to a scaler. Not at runtime — at compile time. In the last post I called this "putting on glasses for the first time." Let me show you what it actually buys you. The ColumnTransformer API is built on that split: ct .add_numeric ( "scaled" , vec! [ 0 , 1 ],