You are not invited to the party

AI has been THE hot domain for the past two years - and everyone tried to position accordingly, especially VCs.

Their thesis is quite simple :

- AI went from a niche research field to mainstream market by throwing millions of dollars of training & data.

- Let's throw more dollars on it and hopefully it will get more useful

The thing is, there are levels to this shit.

• GPT2 was a proof of concept - today it can be trained in 90 minutes for $20.

• GPT3, seen as the big breakthrough - costs $5M to train in 2020.

• Latest SOTA model GPT4 costs $100M.

Do you consider GPT4 20x more efficient than GPT3 ? Obviously not.

It's sometimes even dumber & haven't enabled new use-cases (only bigger context windows & vision).

So OpenAI throws $100M into training a model, I bet they are making tons of money from inferences ?

Not at all. The only upside of inferences is to own data to make the model smaller.

You've read it right. You spend $100M, you lose money on your users just to train a smaller model after that. But let's definitely throw $1B for GPT5.

I won't even mention o1, which have been such a shitshow for OpenAI.

You could pay >30X more for o1-preview than GPT-4o. And o1-preview is ~100X slower to return a token to the user than GPT-4o.

Industry is gambling that we are so close to an inflection point that GPT5 will be magic & gives everyone superpowers. No need to be profitable if you can ask the AGI how to make money.

The next big thing in the crypto industry is decentralizing compute.

It looks great on paper, you could use all PoW GPUs that are not used anymore + gamers GPUs that idle mine between their Fortnite sessions.

I've read all the papers on the subject, read training reports from big models company & asked a lot of questions to researchers.

From first principle, I don't see how it could work. Haven't seen any paper that implemented it at scale, most of them are test training on 8 A100 workers.

We're in Q3 24 and the most meaningful implementation is on 8 workers ? While Llama 405B is open sourced and have been trained on 48 000 A100 for 14 consecutive days.

And even in this configuration, they got +400 unexpected interruption, with arguably the best team in the world for this.

You want to train a better model, using gamers shitty GPUs, allowing interruptions, tolerant to adversarial behaviors & with people spread everywhere across the globe ? Would not bet on it.

"Renting out your GPUs to anyone other than the highest bidder does not make economic sense, and Meta is publicly willing to pay more per hour for GPUs than anyone else."

I still think it's a cool experiment for a guy in their garage, just not sure if we should throw millions on it.

"Everyone just says scaling hypothesis. Everyone neglects to ask, what are we scaling?" Ilya Sutskever

Now let's talk about data, the latest grand narrative in the crypto industry.

First, let's stop pretending data is fungible. 100 GB of data from Wikipedia does not have the same value as data from a YouTube transcript, nor from Reddit shitposts.

Data isn't oil, it's sand - it's only valuable in the aggregate of millions. And all the sand has been used to make sandcastles.

There are no alpha anymore, Wikipedia has been fully scraped. Newspapers have been scraped. Books have been scraped.

Social medias have also been scraped. If Reddit licenses all their data to Google for $60M, how valuable are your Reddit DataDAO trained on 0.1% of the website ? Private data doesn't have a moat either. It's probably already used by Facebook, Google or Apple.

Anyway, lots of yapping - but my point is, AI is commoditized.

There is a gigantic gambling party happening on big tech datacenters.

And you are not invited. Neither do I.

But we can still organize our own counterparty.

Crypto has always been about compression.

And we fucking excel at this, we can do transactions for less than $0.01 on any zk-powered L2 right now.

Let's do what we are good at, and work on compression for smaller community owned models.

Most LLMs data are just fucking useless - and it won't be better in the future.

You can buy all tokens on DexScreener and get a x100 tomorrow. You won't make money with this strategy. It's the same for data, you can put everything in the world on a fucking dataset, it will improve the model but by how much ?

There are better & cheaper open-source models than GPT4 now.

We don't need to try to compete with GPT5, we can just compress GPT4 with community ownership (and decentralized training if you'd like so).

We can also innovate on smaller models.

We can make fine-tuning accessible to everyone.

We can pioneer mixture of experts architecture.

We can finance new architecture's breakthrough.

Finally, we can leverage people creativity.

I'm highly biased due to my project @merv_wtf being focused on diffusion models, but LLMs aren't the only thing that matters.

The biggest consumer project of the past few years is about voice modality.

Metaverse is also not dead, we haven't scratched the surface of possibilities around UGC + AI.

More data won't fix you ❤️