Robot Learning 2025: Foundational Models for Robotics and Scaling DeepRL - Robot Learning: Learning Reward Models and Using Foundational Models for Rewards

I (Glen Berseth) discuss reward functions in reinforcement learning, differentiating between extrinsic (task-specific) and intrinsic (generally helpful) rewards. He highlights the challenges of learning reward functions in complex, real-world environments with irrelevant information, using the example of an agent trying to cook fish in a cluttered kitchen. The conversation explores potential intrinsic signals that could aid learning, such as object identification, affordances, and predicting the future. Berseth introduces the concept of "nexting" and discusses a paper on the "unreal agent," which addresses the representational challenge of learning useful representations for a policy. I explain how recent foundational models can be used to help the agent get rewards. Most works use foundational models as types of data augmentation or dataset labelling to train a reward function. There is some recent work that directly uses a foundational model as a reward function but this use case is complex.