Modern Robot Learning from Scratch - Lecture 5 - Understanding the ACT Policy from Scratch | Modern Robot Learning from Scratch

Every field has a few ideas that quietly change everything. For large language models, it was the Transformer. For modern robot learning, I would argue it is the ACT policy (Action Chunking with Transformers). The ACT policy fundamentally changed what was possible in robotics by opening the door to open-source, learnable policies that could actually run on low-cost hardware. That single shift moved robot learning out of a few elite labs and into the hands of much smaller research teams. What makes ACT truly seminal is that innovation happened on both sides of the stack. On the hardware side, costs dropped by nearly two orders of magnitude. That alone reshaped who could experiment, iterate, and deploy real robots. On the software side, there were three key breakthroughs: (1) A conditional Variational Autoencoder to model action distributions (2) A DETR-based transformer decoder for structured action prediction (3) Action Chunking, inspired by neuroscience, allowing robots to reason and act over meaningful temporal segments rather than single timesteps. Robots trained with ACT could fold clothes, insert batteries, close zip-lock bags, and handle a wide range of real-world manipulation tasks that were previously brittle or hand-engineered. This video explains the ACT Policy in detail, it build upon the previous 4 lectures very nicely. It is like the culmination of a movie with 4 parts, where each parts contributes to the final movie :) Hope you enjoy this one!