← Garden of Thoughts
July 28, 2026 ai deep-learning gradient-descent backpropagation neural-networks

Garden of Thoughts | SherryAnalytics | July 28, 2026

Your GPU Bill Isn't Paying for a Search — It's Paying for a Bartender

Sub-title: Why training a 100-billion-parameter model was never a brute-force problem.

Training a large model burns an eye-watering amount of money and compute, and that naturally suggests a picture: somewhere in a data center, a model is trying combination after combination of its parameters until one works. It isn't. It's not even in the neighborhood of possible.

The number is not just big — it's off the scale of the universe

Give every parameter just two possible values, the simplest case imaginable. A 100-billion-parameter model — the size that actually costs real money to train — has 2^100,000,000,000 possible settings. The observable universe contains roughly 10^80 atoms, total.

That's not "search would be slow." Search doesn't fit inside our universe. And you don't need anywhere near 100 billion parameters to hit that wall — a network with just 266 binary parameters already has more possible settings than there are atoms in existence. Brute force is dead on arrival for a network you could sketch on a napkin, let alone one with a hundred billion knobs.

What's actually happening is closer to tasting a cocktail

The right mental model isn't a search. It's a bartender adjusting a drink.

A bartender doesn't mix every possible ratio of ingredients and taste all of them. They pour one, taste it, and think "too bitter" — and that single taste tells them two things at once: which direction to adjust, and roughly how far. Less espresso, more cream. Taste again. A few rounds and it's dialed in.

Training a neural network runs the same loop:

The whole cycle is called gradient descent. The part that makes it work isn't tasting harder — it's that backpropagation uses calculus (specifically the chain rule) to calculate, in one pass, exactly which direction and how far to nudge every one of those 100 billion parameters simultaneously. Nothing is tried and discarded. It's solved directly.

So what's the money actually paying for

If it isn't paying for search, where does it go? It goes to repetition times scale: running that taste-and-adjust loop millions of times over trillions of tokens, and recalculating all 100 billion numbers on every single pass.

That's an extraordinarily long slog of arithmetic. It's not a hunt through possibility space.

Signoff: Training a large model is expensive because it repeats the same calculation an enormous number of times — not because it's exploring an enormous number of possibilities.

Idea, outline, and argument: mine. I used Claude to help tighten the prose.

Links