Yann LeCun's new venture is a contrarian bet against large language models

This kind of work is important because it's an attempt to make computers understand the real world. LLM/GenAI do not understand the real world, and the more we run up against their limitations, the more the people and systems who depend on them will try to make the world more understandable to LLMs/GenAI. This means modifying your behavior so that you are more like a machine. LeCun is trying to make machines more human.

It's a bet beyond LLM and generative AI, to embrace other techniques and areas of research.

> The world is unpredictable. If you try to build a generative model that predicts every detail of the future, it will fail. JEPA is not generative AI. It is a system that learns to represent videos really well. The key is to learn an abstract representation of the world and make predictions in that abstract space, ignoring the details you can’t predict. That’s what JEPA does. It learns the underlying rules of the world from observation, like a baby learning about gravity. This is the foundation for common sense, and it’s the key to building truly intelligent systems that can reason and plan in the real world. The most exciting work so far on this is coming from academia, not the big industrial labs stuck in the LLM world.

Someone should just build an ANN as big as currently as possible with current hardware, while still having both inference and training to be as close to real-time as possible (micro-to milli-seconds), build the self-learning using some loose equivalents of pain/pleasure feedback in actual brains, plug sensors and actuators from some sort of robot, and just see what happens.

I think anything less than that is just a parlor trick.

For more information about his methods, broadly termed energy based models, check out the deep learning course he co-taught with Alfredo Canziani at NYU. https://atcold.github.io/NYU-DLSP20/

Quantum inference. Mark my words and give it 20+ years.