Am I pushing more than ~1 billion tokens a month, sustained, with enough concurrency to keep a batch full?

If my traffic is bursty or I'm guessing, the answer is no, and idle time will bury me. 2. Is there a non-cost reason — privacy, residency, a model nobody hosts? If yes, self-hosting may be mandatory regardless of the math, so I stop optimizing for dollars. 3. Is my time worth less than the API bill I'd save? Usually it isn't. The ops overhead is the real price, and it's denominated in the scarcest thing I have. For me, on a solo product with spiky traffic, all three lately point the same way: pay per token, keep the frontier quality, and spend my hours building the thing people actually pay for. I'll rent the H100 again the day my batch queue is full and stays full. Until then, the idle GPU was the most expensive nothing I ever bought.

When Self-Hosting an Open-Weight Model Actually Beats the API

Last month I spun up an H100 on RunPod, loaded a 70B open-weight model, and watched my own product hit it. I'd convinced myself the hosted API bill was getting silly and that owning my own inference was the grown-up move. The pod ran for nine days before I killed it.

Then I opened the billing page. I'd paid for nine days of GPU. My actual traffic had used maybe forty minutes of it. The rest was the GPU sitting at idle, fans spinning, charging me by the millisecond to do nothing.

That's the whole trap in one screenshot. The GPU doesn't care whether you're serving tokens or staring at a dashboard. It bills the same. And the per-token API I was trying to escape only charges when work actually happens.

So I sat down and did the math properly. Here's what I found, with real 2026 numbers.

The short version

Self-hosting an open-weight model beats a hosted API on cost only when you keep the GPU genuinely busy — roughly 40–60% utilization or higher, sustained, for weeks. Below that, the idle time eats every dollar you thought you were saving, and you're also paying for it with your own ops time. For most solo builders, most of the time, the per-token API wins on cost and wins harder on hassle. The crossover is real, but it lives at a traffic volume most of us don't have yet.

What renting the GPU actually costs

Let me anchor the numbers, because the whole argument turns on them. As of mid-2026, on-demand H100 80GB rental sits around $2.89/hr on RunPod and $2.99/hr on Lambda. On a marketplace like Vast.ai you can find H100 from roughly $1.47/hr, and consumer cards like the RTX 4090 go from about $0.29–0.69/hr depending on host and tier. A100 80GB lands near $1.39/hr on RunPod.

Pick the cheap-but-honest case: an H100 at $2.89/hr on a reliable host. Run it 24/7 for a month and that's about $2,080. That number doesn't move whether you serve ten requests or ten million. It's a fixed cost the moment the pod is up.

That's the figure every breakeven calculation has to clear.

Tokens per second is where the dream meets the bandwidth wall

The pitch for self-hosting is "free tokens after the GPU." But the GPU only produces tokens as fast as its memory bandwidth allows, and a 70B-class model is heavy.

Realistic throughput numbers, not marketing ones: a single H100 serving a 70B model at modest batch sizes gets you into the tens-of-tokens-per-second range per request. Spread across a small cluster you do better — a 4×A6000 vLLM benchmark shows aggregate throughput around 420–470 tokens/sec for a 70B model under batched load (Qwen2.5 hit 449 tok/s there). Smaller models on a single consumer card move faster: an RTX 4090 does roughly 104 tok/s on an 8B model and 69 tok/s on a 14B at low concurrency.

The thing to internalize: throughput climbs with batching. One H100 serving one request at a time is wildly inefficient. The same H100 serving sixteen concurrent requests produces many times the tokens for the same hourly cost. Self-hosting economics live and die on whether you have enough simultaneous traffic to keep that batch full.

The worked example

Let me make it concrete with a 70B open-weight model, because that's the size where people actually consider self-hosting to dodge frontier-API prices.

The API side. Hosted providers serve Llama 3.3 70B cheaply now. Artificial Analysis shows blended prices from about $0.12/M tokens at the bottom (DeepInfra Turbo FP8) up past $1/M at the top, with DeepInfra around $0.35/M, Fireworks near $0.70/M, and Together around $0.88/M. Call it a middle-of-road $0.40 per million tokens, input and output combined. And if you don't insist on open weights specifically, DeepSeek V4 Flash is $0.14 in / $0.28 out per million — frontier-class quality for less than the rounding error on a GPU bill.

The self-host side. That $2.89/hr H100 costs $2,080/month if it's always on. To break even against the API at $0.40/M, you'd need to push:

$2,080 ÷ $0.40 per million = 5.2 billion tokens per month.

Five billion tokens. At a realistic sustained 400 tokens/sec aggregate, a single well-batched H100 produces about 400 × 3600 × 24 × 30 ≈ 1.0 billion tokens a month if it never stops. So one H100 at full tilt, every second of every day, doesn't even clear the breakeven against a mid-priced API. You'd need the box pinned near capacity and the cheaper end of the rental market and near-zero idle to come out ahead.

Now compare against DeepSeek V4 Flash at roughly $0.21/M blended. The API just got half as expensive. The self-host case gets worse, not better, every time hosted prices drop — and they keep dropping.

The 40-minutes-of-9-days problem

My pod ran nine days. My traffic used forty minutes. That's an extreme case, but it's the normal shape of solo-builder traffic: bursty, spiky, mostly nothing.

Here's the math that actually matters. Utilization is the whole game:

At 100% utilization (GPU always batched, always busy): self-hosting can win if your volume is genuinely enormous, per the example above.
At ~50% utilization: your effective cost per token doubles, because you paid for the idle half. Now you need 2× the volume to break even.
At 5% utilization (my forty minutes): your effective cost per token is ~20× the sticker math. The API wins by a landslide and it isn't close.

Per-token APIs are pure pay-per-use. They have already solved your idle problem — by amortizing the GPU across thousands of other customers' bursts. That pooling is the actual product you're paying for. When you self-host, you take the idle risk back onto your own credit card.

You can fight idle with serverless GPU (scale-to-zero, per-second billing) or spot instances at 50–80% off. Both help. Both also reintroduce cold starts, eviction, and the exact reliability headaches you were trying to leave behind.

The costs that never show up in the calculator

The hourly rate is the honest part of the bill. The dishonest part is everything around it, and it all lands on you when you're solo:

Setup and ops time. Standing up vLLM, picking quantization, tuning batch size, wiring health checks, handling OOMs. That's hours, and you're the only engineer. At any reasonable value on your time, the first month of self-hosting is more expensive than a year of API calls.
Reliability. When a hosted API has a bad node, that's their pager. When your pod dies at 2am, that's yours. There's no failover unless you build it.
The quality gap. This is the one people skip. The best open-weight 70B is good, but the cheap frontier APIs are better on hard reasoning and code, and they cost less than your GPU. You may be paying more to ship a worse answer.
Quantization tax. The cheap throughput numbers usually assume FP8 or 4-bit quantization. That's a real quality haircut on top of the open-vs-frontier gap. Run full precision and your throughput drops and your VRAM need climbs.

None of these appear on the RunPod pricing page. All of them are real.

When self-hosting genuinely wins

I don't want to talk you out of it categorically, because there are cases where it's the right call — they're just narrower than the hype suggests:

Sustained, high, predictable volume. If you're reliably pushing billions of tokens a month with a full batch queue, the fixed GPU cost amortizes and you win. This is a real business with real traffic, not a side project.
Privacy or data-residency requirements that forbid sending tokens to a third party. Here cost isn't the deciding variable at all.
Heavy fine-tuned or niche models that no host serves, where the API option doesn't exist at any price.
A small model that fits a cheap card. An 8B on a $0.40/hr 4090 is a completely different equation than a 70B on an H100. If your task genuinely runs on a small model, self-hosting gets affordable fast — but then so does the hosted API for that same small model.

Notice the pattern: the wins are about volume, control, or constraints — not about a small builder shaving a modest API bill.

The decision rule I use now

Before renting a GPU, I ask three questions in order. If any answer is "no," I stay on the API.

Am I pushing more than ~1 billion tokens a month, sustained, with enough concurrency to keep a batch full? If my traffic is bursty or I'm guessing, the answer is no, and idle time will bury me.
Is there a non-cost reason — privacy, residency, a model nobody hosts? If yes, self-hosting may be mandatory regardless of the math, so I stop optimizing for dollars.
Is my time worth less than the API bill I'd save? Usually it isn't. The ops overhead is the real price, and it's denominated in the scarcest thing I have.

For me, on a solo product with spiky traffic, all three lately point the same way: pay per token, keep the frontier quality, and spend my hours building the thing people actually pay for. I'll rent the H100 again the day my batch queue is full and stays full. Until then, the idle GPU was the most expensive nothing I ever bought.