Self-hosted GPU or API: how I decide

Every AI product reaches the same fork sooner or later. Do you call someone else's model through an API, or do you run a model yourself on rented GPUs? I am building Bolo Media, which turns product photos and short text into video ads, so this is not an abstract question for me. Here is the framework I use for the self host AI model vs API decision, and why I think most founders answer it too early.

Self host AI model vs API is not a price question

People frame this as a cost question. It is partly that, but cost is the output of the decision, not the input.

An API gives you a model that someone else trains, hosts, scales and fixes. You pay per call. You get no idle cost, no servers to babysit, and usually the best quality available that week.

Self-hosting gives you control. You pick the model, you pin the version, you decide what runs where. In return you take on cold starts, GPU memory limits, model updates, monitoring, and the bill for every minute a machine is switched on, whether it is doing useful work or not.

So the real question is: which risks and which work do I want to own? Cost follows from that answer.

Five questions I ask before choosing

1. Does the model I need even exist as an API?

This settles more cases than anything else. If the capability you need is only available as open weights, the decision is made for you. If it is only available behind an API, same thing. The interesting cases are the ones where both exist, and even then the quality is often not equal. For Indian languages in particular, quality varies a lot between models, so "both exist" does not mean "both are usable".

2. How steady is the load?

A rented GPU costs money while it waits. An API costs money only when you call it.

If your traffic is spiky, small, or unpredictable, which describes almost every early-stage product, pay-per-call usually wins. You would be paying for a GPU to sit idle most of the day. If your load is steady and high, the maths can flip, because you are finally keeping the machine busy.

The honest answer for most new products is that you do not know your load yet. That alone is a reason to lean towards APIs at the start.

3. How much do I need to control the output?

Some work needs a specific model version that will not change under you. Some needs a fine-tuned model, a custom pipeline, or settings an API does not expose. If a vendor quietly updates a model and your outputs shift, can your product absorb that?

I wrote more about this risk in building a product on AI models you don't control. Control is the strongest argument for self-hosting, and it is a real one. It is just not free.

4. Who keeps it running at 2 a.m.?

This is the question solo founders skip. A self-hosted model is a small piece of infrastructure. It needs updates, it fails in new ways, it runs out of memory on an odd input. An API vendor has a team for that. A solo founder has themselves.

I spent years leading projects and then working as a technical project manager before starting Bolo. One thing that work teaches you is that the build is the cheap part. The running is what costs you, every week, for as long as the thing exists.

5. What does it cost to switch later?

If you design your code so the model sits behind one clean interface, switching from an API to self-hosting (or back) is a contained job. If model calls are scattered everywhere, you are locked in by your own code, not by the vendor. I try to keep this door open whichever way I go.

Measure before you migrate

My own rule, which I apply to everything with a bill attached: validate the load-bearing assumption cheaply before building on it.

For this decision, the load-bearing assumption is usually "self-hosting will be cheaper" or "this open model is good enough". Both can be tested small. Run a handful of real jobs both ways. Note the time, the output quality, and what each one actually cost. Then look at the invoice, not the pricing page, because the pricing page leaves out idle time, retries and failed runs.

A number that has not been reconciled against a real bill is an estimate. I label it as one, even in my own notes. I go into this habit in more detail in how I think about AI inference cost.

The cheapest test that can still fail is the one worth running. A single sample job will tell you more than a week of spreadsheet modelling.

Where each option tends to win

This is a rough map, not a rule:

Situation	Usually leans towards
Early product, uncertain traffic	API
Capability only exists as open weights	Self-hosted
Need a pinned, unchanging model version	Self-hosted
Need best available quality right now	API
Steady, high, predictable volume	Worth testing self-hosted
No one free to maintain infrastructure	API

Most real products end up with a mix. Some steps go to an API because it is the best tool for that step. Others run on your own GPU because nothing else does the job, or because control matters more there. That is not indecision. It is just choosing per step instead of per company.

The mistakes I try not to make

Optimising before validating. It is easy to spend days tuning a self-hosted setup around a model that was never checked against the real task. Test the model on your actual inputs first.

Comparing list prices. An API's per-call price and a GPU's per-hour price are not comparable until you know your utilisation. Idle time is the hidden line item.

Treating the choice as permanent. Models get better and prices change, often within months. A decision that was right in spring can be wrong by Diwali. Revisit it.

Ignoring your own time. For a solo founder, hours spent on infrastructure are hours not spent with customers. That cost does not show on any invoice, but it is the largest one.

Where I land

I do not have a fixed answer, and I would be wary of anyone who does. I start from the task, ask whether a usable model exists each way, test both cheaply on real inputs, read the actual bill, and choose per step. Then I come back to it when something changes.

If you are building something similar and want to see what the end product looks like, that is what we are making at Bolo Media. The plumbing underneath is just a long list of decisions like this one, each made as small and as reversible as I can manage.