The LLM was better at building a solver than playing the game

I started this project because an LLM annoyed me. I gave a very strong model 322 , a small Dota 2 drafting game. The choices looked like the kind of work a computer should enjoy: repeated packs of players and heroes, visible ratings, familiarity scores, chemistry, rerolls and a simulated tournament at the end. I was disappointed by how well the LLM did. I am not a Dota expert, and I had only started watching it occasionally again during the previous six months or year. I still seemed to be doing better. The interesting engineering question was not how to write a longer prompt. It was how to replace the card-by-card language-model judgement with a deterministic policy, then test that policy without confusing improvement with luck. A stochastic benchmark needs shared randomness The browser history gave us a useful irritation and almost no reliable comparison. My earlier manual record contained 50 runs with a 14% title rate. The LLM won once in nine attempts. Putting 14% beside 11% looks temptingly quantitative, but the random offers, rejected packs and opponent fields were not preserved. The samples were small, unpaired and produced under different choices. That is not a model benchmark. It is a reason to build one. The offline solver generated every random choice from indexed tapes. Policy A and policy B received the same player offers, hero samples, field candidates and tournament randomness for a given episode. We could then compare the paired result: did the new policy win this exact episode where the old policy lost it? This is the common-random-numbers idea in a practical form. Sharing the luck removes a large amount of noise that has nothing to do with the policy change. Keep the simulator separate from the policy Before evaluating a strategy, we reproduced the game. The public client and seven data files were frozen with SHA-256 hashes. Draft legality, automatic hero allocation, chemistry, scoring and the tournament were ported into a deterministic Python engi