descending cross-entropy to ascend entropy @PeriodicLabs || prev research @OpenAI @CarnegieMellon '23
San Francisco, CA
Joined February 2020
- Tweets 6,239
- Following 2,543
- Followers 40,357
- Likes 25,678
Pinned Tweet
Excited to finally announce what I've been up to since leaving OpenAI: autonomous science at @PeriodicLabs
Why? Because I Take the Bitter Lesson Seriously
To accelerate AI, we must enable it to hill-climb compute & energy through experimentally verifiable science 🧵
Today, @ekindogus and I are excited to introduce @periodiclabs.
Our goal is to create an AI scientist.
Science works by conjecturing how the world might be, running experiments, and learning from the results.
Intelligence is necessary, but not sufficient. New knowledge is created when ideas are found to be consistent with reality. And so, at Periodic, we are building AI scientists and the autonomous laboratories for them to operate.
Until now, scientific AI advances have come from models trained on the internet. But despite its vastness — it’s still finite (estimates are ~10T text tokens where one English word may be 1-2 tokens). And in recent years the best frontier AI models have fully exhausted it.
Researchers seek better use of this data, but as any scientist knows: though re-reading a textbook may give new insights, they eventually need to try their idea to see if it holds.
Autonomous labs are central to our strategy. They provide huge amounts of high-quality data (each experiment can produce GBs of data!) that exists nowhere else. They generate valuable negative results which are seldom published. But most importantly, they give our AI scientists the tools to act.
We’re starting in the physical sciences.
Technological progress is limited by our ability to design the physical world.
We’re starting here because experiments have high signal-to-noise and are (relatively) fast, physical simulations effectively model many systems, but more broadly, physics is a verifiable environment. AI has progressed fastest in domains with data and verifiable results - for example, in math and code. Here, nature is the RL environment.
One of our goals is to discover superconductors that work at higher temperatures than today's materials. Significant advances could help us create next-generation transportation and build power grids with minimal losses. But this is just one example — if we can automate materials design, we have the potential to accelerate Moore’s Law, space travel, and nuclear fusion.
We’re also working to deploy our solutions with industry. As an example, we're helping a semiconductor manufacturer that is facing issues with heat dissipation on their chips. We’re training custom agents for their engineers and researchers to make sense of their experimental data in order to iterate faster.
Our founding team co-created ChatGPT, DeepMind’s GNoME, OpenAI’s Operator (now Agent), the neural attention mechanism, MatterGen; have scaled autonomous physics labs; and have contributed to some of the most important materials discoveries of the last decade. We’ve come together to scale up and reimagine how science is done.
We’re fortunate to be backed by investors who share our vision, including @a16z who led our $300M round, as well as @Felicis, DST Global, NVentures (NVIDIA’s venture capital arm), @Accel and individuals including @JeffBezos , @eladgil , @ericschmidt, and @JeffDean. Their support will help us grow our team, scale our labs, and develop the first generation of AI scientists.
people often take deep learning as synonymous with backprop, but deep networks were originally trained with probabilistic energy-based methods!
found this great talk by hinton from 2012 about EBMs, boltzmann machines, and deep belief nets at the start of the deep learning era
there’s a lot of reasons to make fun of hacker house / hackathon culture
but any average 2023 agi house event had probably ~3 people who’d end up founding unicorns in the last couple yrs
still worth eventsmaxxing if new to sf
ok i'm gonna officially claim the honor of ending the @stephenwitt 2d matmul saga
thanks to my evidently goated teaching skills which he read at 1:43pm, he retracted his statements at 1:45pm
call me the yung karpathy or smth 😤😎
if i had a nickel for every time @boazbaraktcs tweeted a viral screenshot of a new york based journalist being stupid about linear algebra, i’d have two nickels
which isn’t a lot but it’s weird that it happened twice
Rohan Pandey retweeted
After a scientist tries to explain that high-dimensional matrix multiplication is beautiful in some ways, New Yorker writer doubles down -- that's interesting, but recall that matrices have TWO dimensions.
Can't make it up.
Open the schools!!!!
I agree, although I believe most matrix multiplies for AI are done in two dimensions. Anyway, I stand by my statement. Matmuls are an effective piece of mathematical machinery, but the mechanics of calculating them are headache-inducing.
Indeed, it's exactly their computationally cumbersome nature that's propelling the data center boom. That point seems beyond argument.
according to the @newyorker, matrix multiplication is as ugly as uh
*checks notes*
...the fundamental nature of reality?
surprised that no one has posted @sarahookr's hardware lottery essay since @GillVerd's TSU announcement
maybe EBMs were just waiting for the right chips to come along
hot take: you shouldn’t give white ppl shit for thinking there’s a language called “indian”
the word “hindi” comes from farsi where it literally just means indian
it’s good that non-indians know india has an indigenous national language, and we should let them call it as such
idk what the fuss is, scaling RL to 1T+ params is simple
all you need is:
1. a few thousand gpus
2. the og goat of opensource RL @vwxyzjn
3. the guy who invented the attention mechanism @DBahdanau
“when a field is getting started, it’s easy to confuse the essence of what you’re doing with the tools you’re using”
what is the essence of LLM Studies? could LLMs just be one instantiation of a broader abstraction?
please don’t answer “ml theory”
yash was instrumental in openai's 2 most successful agentive products: codex and deep research
now he's bringing that RL x Product expertise to enterprise
keep an eye on these guys!
Today, @rhythmrg, @lindensli and I are introducing @appliedcompute. We’re building Specific Intelligence for the enterprise.
Achieving SOTA today means specialization in both human and machine talent. We’ve spent the last six months working with companies like @cognition, @DoorDash, and @mercor_ai, unlocking their company knowledge to build custom agent workforces that outperform frontier models at specific tasks.
My cofounders and I all worked on different parts of this problem while at OpenAI, from Codex to o1 to the ML systems and infrastructure for RL training. Two-thirds of our team (see below!) are former founders, and everyone brings a deep technical background, from top AI researchers to Math Olympiad winners.
We’ve raised $80M from @benchmark, @sequoia, @Lux_Capital, @eladgil, @victoralazarte, and @Casspi18, and we’re hiring across engineering and research.
