prompt的PPL 设prompt为x1:Tx_{1:T}x1:T​，给定LLM πθ\pi_{\theta}πθ​（也就是条件分布pθ(⋅∣⋅)p_{\theta}(\cdot|\cdot)pθ​(⋅∣⋅)） 那么其在teacher forcing下对于整个prompt的对数似然为： logpθ(x1:T)=∑t=1Tlogpθ(xt∣xt<t).logp_{\theta}(x_{1:T

Policy Gradient 策略梯度的目标是，找到一个最优策略πθ∗\pi^*_{\theta}πθ∗​，让agent在与环境交互时，获得的累积回报期望J(θ)J(\theta)J(θ)最大 最直接的优化方法就是梯度上升，即沿着梯度的方向更新参数： θnew=θold+α∇θJ(θ)θ_{new}=θ_{old}+\alpha\nabla_{\theta}J(\theta) θnew​=θol

Approximating KL Divergence Verl Implementation 12345678910111213141516171819202122232425262728293031323334def kl_penalty(logprob: torch.FloatTensor, ref_logprob: torch.FloatTensor, kl_penalty) ->

Infra: https://github.com/volcengine/verl Train Data: https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k Eval Data: https://huggingface.co/datasets/HuggingFaceH4/aime_2024 https://huggin

12345def entropy_from_logits(logits: torch.Tensor): """Calculate entropy from logits.""" pd = torch.nn.functional.softmax(logits, dim=-1) entropy = torch.logsume