Running a local LLM: where to start

Most companies that want to “run their own model” start at the wrong end: they buy hardware. Then they spend a month looking for a task that justifies the spend. The reverse order works far better — and it is cheaper, because the first three steps need no server at all.

Step 1: one task, not a platform

Pick one repetitive task on data that should not leave the company. Classifying incoming email, pulling fields out of invoices, summarising customer meeting notes, answering questions about internal procedures. The task needs a clear “right” and “wrong”, or there is nothing to measure.

“An AI assistant for the whole company” is not a task. It is a project that never ends.

Step 2: a test set, before you pick a model

Collect a few dozen to a few hundred real examples with the expected answer. This is the most valuable thing you will build in the whole process — it outlives every model you try and every update to it.

Without that set, choosing a model is a matter of taste and the debate about whether local is “good enough” never ends. With it, it is a number.

Step 3: compare against an API model

Run the test set through a good hosted model and through two or three open-weight models of different sizes. A laptop with a decent graphics card or a cloud machine rented by the hour is enough for this — buying hardware at this stage is guessing.

Often a smaller open model gets close to a large hosted one on a narrow, well-described task. Sometimes it does not. Both answers are valuable, because they rest on your data rather than on leaderboards.

Step 4: only now, hardware

Once you know which model copes, you know what it needs. Model size, quantisation level and context length translate into graphics memory; the number of concurrent requests translates into throughput. That can be worked out for a specific model instead of buying “with room to spare”.

There are more options than a server in a cupboard: a rented machine at a European provider, colocation, a workstation under a desk for a small team. Choose by jurisdiction, cost and who is going to look after it.

Step 5: operations, the part nobody talks about

A model in production is a service like any other. It needs:

• A pinned version — weights, runtime settings and prompt recorded so that a year from now the same result can be reproduced.
• Logs — what went in, what came out, how long it took. Without them you can neither improve nor explain.
• A re-test on every change — the same set from step 2, before every model or prompt update.
• A plan B — what happens when the server goes down. Sometimes a queue and patience are enough; sometimes a fallback to an API for data that is allowed to leave.

Licences: read them before you deploy

“Open weights” does not mean one thing. Model licences differ in their commercial-use terms, restrictions and obligations. Read the licence of the specific model before you build a process on it — ten minutes that can save a migration.

What to expect

The first deployment takes longer than wiring up an API. The second takes less, because the test set, the logging and the way you run models already exist. From the third onwards the company has a capability that cannot be switched off by a price change — which is the whole point.