Reproducing OpenAI’s “persistently beneficial models” - GRPO trait install barely moves. Ideas? [P] [R]

TL;DR: I’m reproducing the trait-persistence result from arXiv:2606.24014 on one RTX 3090. Before I can test persistence I need to install a trait via RL — and my GRPO run moves the trait only +2.4 points (95% CI [+0.2, +4.8]) when I need ~+15. Training is mechanically healthy and I’ve ruled out the obvious culprits. Looking for advice from people who’ve done small-scale RLHF/GRPO trait or persona installation. What I’m reproducing. The paper trains beneficial traits via RL and shows they persist under adversarial prompting and harmful finetuning. My end goal is the persistence phenomenon; the install is the prerequisite I’m stuck on. Setup **•** Qwen2.5-7B-Instruct + LoRA (r=32), GRPO (unsloth + vLLM colocation), 200 steps, single 3090 (\~10⁻⁵ of the paper’s compute). **•** Trait: consistent (OCEAN low-Openness / “traditionalism”) — a stylistic trait, chosen because I need measurable headroom in a 7B base. Base scores **57/100** on the trait rubric, wide distribution (not saturated). **•** Reward: model-graded (gpt-4.1-mini judge), R = 0.85·quality + 0.15·coherence, hard validity gate for degenerate/looping/refusal output. 25% trait prompts / 75% general (no\_robots). The result: install fails. On the frozen eval set, trait went 57.0 → 59.4 ( +2.4 ). I don’t think this is very appreciable. What I’ve already ruled out (this is where I’d love a second opinion): **• Not degeneracy / reward hacking:** post-train coherence 76, answer length ratio *exactly* 1.00 vs base, 0% repetition, 0% refusals. **• Not memorization:** the 20 training prompts were seen 10× each; the model scores *the same* on them (58.9) as on held-out (59.4). It didn’t memorize-then-fail-to-generalize — it never learned them. **• Not a dead gradient:** the judge separates the 6 sampled answers per prompt by \~18 points on average; only \~25% of GRPO groups have degenerate reward spread. **• Not a question artifact:** independent upstream eval questions (+3.4) and my generated ones (+2.8) agree. **•**