On-policy RL for large language models faces training-inference mismatch due to separate inference and training processes; mismatch arises from engine numerics and stale asynchronous sampling weights
Read the original at arxiv.org→arXiv:2610.40945v1 Announce Type: new Abstract: On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine...
Original headline: "On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning"
Coverage timeline
- Nov 2, 05:00 UTC arXiv cs.LG lead source On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning