Stumbled on a stat that blew my mind about AI training costs

I was reading this report from Stanford's AI Index last night, and I found out that training GPT-3 cost around $4.6 million back in 2020. But get this, training a similar sized model today runs closer to $100 million just for compute. I had no idea the gap grew that fast in just a few years. Has anyone else seen these numbers and wondered if smaller companies can even compete anymore?

2 comments

2 Comments

morgan_butler3d agoMost Upvoted

The jump from $4.6 million to $100 million in just a few years is wild, but I'm more curious about what that actually pays for. Like, are we talking purely the cost of running the computers and electricity, or does that include paying the engineers and buying the data too? Because if it's just the compute bill, that means the hardware itself got 20x more expensive, which makes me wonder if the tech is actually getting better or just more wasteful. Is part of that cost because they're running the models over and over until they get it right, or is it just that the newest chips cost that much more?

lucas_perez3d ago

$100 million is just the compute bill for one training run on those insane GPU clusters @morgan_butler, the salaries and data costs are on top of that. Honestly feels like they're just brute forcing better answers by burning cash instead of actually making the tech smarter.