Direct Desire Optimization (DPO) has been extensively adopted for choice alignment of Giant Language Fashions (LLMs) on account of its simplicity and effectiveness. Nevertheless, DPO is derived as a bandit downside by which the entire response is handled as a single arm, ignoring the significance variations between tokens, which can have an effect on optimization effectivity and make it troublesome to realize optimum outcomes. On this work, we suggest that the optimum knowledge for DPO has equal anticipated rewards for every token in profitable and dropping responses, as there isn’t a distinction in token significance. Nevertheless, for the reason that optimum dataset is unavailable in apply, we suggest utilizing the unique dataset for significance sampling to realize unbiased optimization. Accordingly, we suggest a token-level significance sampling DPO goal named TIS-DPO that assigns significance weights to every token primarily based on its reward. Impressed by earlier works, we estimate the token significance weights utilizing the distinction in prediction possibilities from a pair of contrastive LLMs. We discover three strategies to assemble these contrastive LLMs: (1) guiding the unique LLM with contrastive prompts, (2) coaching two separate LLMs utilizing profitable and dropping responses, and (3) performing ahead and reverse DPO coaching with profitable and dropping responses. Experiments present that TIS-DPO considerably outperforms numerous baseline strategies on harmlessness and helpfulness alignment and summarization duties. We additionally visualize the estimated weights, demonstrating their means to determine key token positions.
†Work carried out throughout an internship at Apple.
‡Tsinghua College
§College of Illinois at Chicago
