Generalist - Scaling AI in Robotics
Making general-purpose robots a reality.
We have seen what happens when the scaling hypothesis is applied to language. In 2019, AI could complete a sentence. By 2020 it was finishing a paragraph. Today, it’s working in complex codebases for hours and helping prove theorems in quantum complexity theory.
We are seeing what happens when the scaling hypothesis is applied to biology. First, AI could generate functional proteins that were of little use. Now they’re generating functional CRISPR systems completely from scratch. Soon, AI will make biology programmable.
The same trajectory is unfolding in robotics. It continues to amaze me that the transformer, when paired with enough data, works across domains. As Ilya said, “These models, they just want to learn”.
In late 2022 and early 2023 there were a series of influential robotics papers published by Google. PaLM-E was impossible to ignore. The paper showed that with the right data a generalist model beats a specialist model. It suggested that the transformer would take us to a generalized robotics foundation model.
Pete Florence and Andy Zeng were co-authors of PaLM-E and together with Andy Barry, Pete’s PhD classmate at MIT, are the co-founders of Generalist, a startup advancing the frontier of AI in robotics.
Yesterday, Generalist shared GEN-0. It is the highest-dexterity model in the world, the beginning of an era where capabilities predictably scale with data, and the first step toward realizing their mission of making general purpose-robots a reality. Along with the model they are sharing the first scaling laws in robotics, a new model architecture, and news of over 270,00 hours of real-world manipulation data from 1,000s of homes, warehouses, and workplaces.
Pete, Andy, and Andy have built a team that feels equivalent in talent and raw ambition to early OpenAI. And their progress reflects this. At OpenAI I consistently had my hair blown back by the engineering challenges that were solved in service of unblocking scale. Generalist has been no different. The stories are already legendary, including building custom hardware and infrastructure – such as laying new dedicated internet lines – to handle the scale of the data volume they’re collecting for their models.
On a personal note, it’s a joy to be reunited with my friend Evan Morikawa, part of the founding team at Generalist and my partner in building the product development org at OpenAI.
I am beyond thrilled to be involved with Andy, Andy, Pete, and the entire team at Generalist. As part of Spark’s investment I have joined the board. Generalist is well on their way to enabling a future where we all benefit from robots in our homes and work.