Policy Gradient 策略梯度的目标是，找到一个最优策略πθ∗\pi^*_{\theta}πθ∗​，让agent在与环境交互时，获得的累积回报期望J(θ)J(\theta)J(θ)最大 最直接的优化方法就是梯度上升，即沿着梯度的方向更新参数： θnew=θold+α∇θJ(θ)θ_{new}=θ_{old}+\alpha\nabla_{\theta}J(\theta) θnew​=θol

Approximating KL Divergence Verl Implementation 12345678910111213141516171819202122232425262728293031323334def kl_penalty(logprob: torch.FloatTensor, ref_logprob: torch.FloatTensor, kl_penalty) ->

Infra: https://github.com/volcengine/verl Train Data: https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k Eval Data: https://huggingface.co/datasets/HuggingFaceH4/aime_2024 https://huggin

12345def entropy_from_logits(logits: torch.Tensor): """Calculate entropy from logits.""" pd = torch.nn.functional.softmax(logits, dim=-1) entropy = torch.logsume