Improving Language Model Alignment: OpenAI's InstructGPT Models Follow User Instructions More Accurately

Welcome to the digital playground! OpenAI API has been using GPT-3 language models to assist with natural language tasks, but these models can generate untruthful, toxic, or harmful outputs due to their training on Internet text. To improve the safety and reliability of these models, OpenAI has developed InstructGPT models, which are trained with human feedback to better align with user intentions and reduce harmful outputs.

The InstructGPT models are trained using reinforcement learning from human feedback (RLHF), where labelers provide demonstrations of the desired model behavior and rank several outputs from the models. This data is used to fine-tune GPT-3, resulting in models that are much better at following instructions, generate fewer imitative falsehoods and toxic outputs, and make up facts less often. The InstructGPT models are preferred over outputs from a 175B GPT-3 model, despite having 100x fewer parameters.

To improve the safety and alignment of language models, OpenAI is exploring different approaches, such as filtering the pre-training dataset, safety-specific control tokens, or steering model generations. In the meantime, the InstructGPT models are now the default language models accessible on the OpenAI API, which have been shown to produce more appropriate outputs and make up facts less often.

However, the InstructGPT models are still far from fully aligned or safe and can generate toxic or biased outputs. To address this, OpenAI is reviewing potential applications, providing content filters for detecting unsafe completions, and monitoring for misuse. They are also conducting research into understanding the differences and disagreements between labelers' preferences to condition their models on the values of more specific populations.

OpenAI's alignment research has resulted in the development of InstructGPT models that are much better at following user intentions and reducing harmful outputs. However, there is still much work to be done to improve the safety and alignment of language models, and OpenAI will continue to push these techniques to develop language tools that are safe and helpful to humans.

Author: Nardeep Singh