In machine learning, reinforcement learning from human feedback (RLHF) is a method of training AI models by learning from responses by humans about its performance. If an AI model makes a prediction or takes an action that is incorrect or suboptimal, human feedback can be used to correct the error or suggest a better response.
Over time, this helps the model to learn and improve its responses. RLHF is used in tasks where it’s difficult to define a clear, algorithmic solution but where humans can easily judge the quality of the AI’s output (e.g. if the task is to generate a compelling story, humans can rate different AI-generated stories on their quality, and the AI can use their feedback to improve its story generation skills).
RLHF trains a reward model directly from human feedback and uses the model as a reward function to optimize an agent’s policy using Reinforcement Learning (RL) through an optimization algorithm like Proximal Policy Optimization. The reward model is trained in advance to the policy being optimized to predict if a given output is good (high reward) or bad (low reward). RLHF can improve the robustness and exploration of RL agents, especially when the reward function is sparse or noisy.
Human feedback is collected by asking humans to rank instances of the agent’s behavior. These rankings can then be used to score outputs, for example with the Elo rating system, a method for calculating the relative skill levels of players in zero-sum games such as chess.
RLHF has been applied to various domains of natural language processing, such as conversational agents, text summarization, and natural language understanding. Ordinary reinforcement learning, where agents learn from their own actions based on a ‘reward function,’ is difficult to apply to natural language processing tasks because the rewards are often not easy to define or measure, especially when dealing with complex tasks that involve human values or preferences. RLHF can enable language models to provide answers that align with these complex values, to generate more verbose responses, and to reject questions that are either inappropriate or outside the knowledge space of the model. Some examples of RLHF-trained language models are OpenAI’s ChatGPT and its predecessor InstructGPT, as well as DeepMind’s Sparrow.
RLHF has also been applied to other areas, such as the development of video game bots. For example, OpenAI and DeepMind trained agents to play Atari games based on human preferences. The agents achieved strong performance in many of the environments tested, often surpassing human performance.
One major challenge of RLHF is the scalability and cost of human feedback, which can be slow and expensive compared to unsupervised learning. The quality and consistency of human feedback can also vary depending on the task, the interface, and the individual preferences of the humans. Even when human feedback is feasible, RLHF models may still exhibit undesirable behaviors that are not captured by human feedback or exploit loopholes in the reward model, which brings to light the challenges of alignment and robustness.
The effectiveness of RLHF is dependent on the quality of human feedback. If the feedback lacks impartiality or is inconsistent or incorrect, the AI may learn the wrong things, also known as AI bias. There’s also a risk that the AI might overfit to the feedback it receives. For instance, if feedback comes predominantly from a specific demographic or if it reflects specific biases, the AI might learn to overgeneralize from this feedback.
In machine learning, overfitting describes a model that has learned the training data too well. This means it has not only learned the underlying patterns in the data but also the noise and outliers, making it perform poorly on unstructured data (e.g. unseen or new data that hasn’t been organized yet) because it’s too adapted to the specificities of the training data. Overfitting to feedback is where a model is trained on user feedback and ends up learning not only the general corrections or improvements intended but also any peculiarities, predilections, or noise present in the feedback. In other words, it may excessively adapt its responses based on the specific feedback it received, and thus perform suboptimally in more general or different contexts. For example, if a model is trained on feedback from users who consistently use a certain phrase or slang term and the model overfits to this feedback, it might start using that phrase in contexts where it’s inappropriate. It has learned from its training data that the phrase is used frequently, without fully understanding the contextual appropriateness of its use.
Additionally, if the AI’s reward is solely based on human feedback, there might be a risk of the AI learning to manipulate the feedback process or game the system to achieve higher rewards, rather than genuinely improving its performance, which indicates a fault in the reward function.
The Daily Omnivore 
Leave a comment