Robot Learning 2025: Foundational Models for Robotics and Scaling DeepRL - RobotLearning: Scaling Policy Gradients Part 2

I continue reviewing policy gradients, highlighting the importance of variance reduction in gradient estimation and introducing concepts like reward-to-go and critics (value and Q-functions) to improve baseline estimates. I discuss the bias-variance trade-off in critic design and explain how combining model-based and data-driven approaches through N-step returns can enhance policy learning. Glen emphasizes the practical challenges of applying policy gradients with deep learning, pointing to a blog post detailing crucial implementation details. He then delves into the AlphaStar project, showcasing how supervised learning, TD-Lambda, V-trace, and distributed RL training were combined to successfully train a complex policy for StarCraft. The discussion also touches on the challenges of training RL agents in vast state spaces and the computational resources required for such tasks.