Short answer: you almost certainly should not train a model on your company data, and you do not need to. Training is only one of four ways to make AI answer using your information, and for a company of 20 to 50 people it is usually the worst of them. The other three are cheaper, change in minutes rather than weeks, and do not go stale the moment a price list does.
This is the most common question we get from Dutch SMEs, and it is almost always asked in the wrong shape. Below: the four mechanisms in plain terms, what each actually costs at August 2026 list prices with the assumptions written out, and a decision rule keyed to where your information really lives, which in the Netherlands is usually not where the international guides assume.
What does ‘training AI on your own data’ actually mean?
Four different things get called this, and confusing them is what wastes the budget:
- Context. You paste the relevant text into the prompt. Zero engineering, instant to change, limited by how much you can paste and by whoever remembers to paste it.
- Retrieval (RAG). You keep a searchable copy of your documents. At question time the system finds the most relevant passages and puts those in the prompt. The model never memorises anything; it reads a briefing note assembled per question.
- Tool calling. You give the model permission to query the live system: your ERP, your bookkeeping package, your ticket queue. It does not read a copy of the data; it asks the system that owns it.
- Fine-tuning. You adjust the model’s weights using examples of the output you want. This is the only one that is actually training.
The rule practitioners settled on, and the one worth memorising: retrieval and tools handle knowledge that changes; fine-tuning handles behaviour that should not. Facts belong in the first two. Form, tone, and output schema belong in the third. When you use fine-tuning to inject facts, you get a model that states last year’s prices fluently and confidently, and you get to retrain every time the price list moves.
Where does your company knowledge actually live?
Almost every guide on this topic assumes your knowledge is a pile of documents, because in the American mid-market it usually is: a SharePoint drive, a wiki, a folder of PDFs. In the Dutch SME the centre of gravity sits somewhere else, and this changes the correct answer.
The Dutch back office runs on Dutch-built software. Exact Online is an accounting core that firms extend with modules and that accountancy practices are deeply anchored in; AFAS is a broader integrated suite where finance, HR, payroll and workflows sit in one system; e-Boekhouden serves the smaller end. In all three cases the thing you want AI to answer questions about — what did we invoice this customer, which orders are open, who is on which project, what is our margin on this article — is not a document. It is a row in a database with an API in front of it.
That distinction decides the architecture. A question with a factual answer recorded in a system of record is a query, not a search. Building a vector store over exported reports so a model can approximately recall a number the ERP knows exactly is the most common expensive mistake in this category. It is slower, it goes stale, and it turns an exact answer into a probable one. We have written separately about connecting AI to Exact, AFAS and e-Boekhouden and what those APIs will and will not give you.
Flemish readers should note the same logic with a different vendor mix: the principle is about where the record lives, not about which Dutch package holds it.
When is retrieval the right answer?
Retrieval earns its keep when the knowledge is genuinely written prose that a human would have to read: installation manuals, contracts and general terms, policy documents, tender responses, historical quotes, service reports written in free text. There is no row in Exact that answers “what does our warranty actually cover on this model”. There is a document, and it is 40 pages long. If you get this far, the follow-up question is which flavour of retrieval, and we have a decision framework for RAG versus GraphRAG that answers it.
Three things separate a retrieval system that survives contact with users from a demo that impressed everyone in March:
- Permissions live in the retrieval filter, not the prompt. If a salesperson may not see HR files, the search must never return them. Instructing the model not to mention them is not access control.
- Freshness has an owner. A document store that is re-indexed ‘when someone remembers’ becomes wrong quietly, which is worse than being wrong loudly.
- You have an evaluation set. Thirty real questions with agreed correct answers, written before you build. Without it, ‘it works’ means ‘the three questions we tried worked’.
What does the long-context shortcut actually cost?
Because current frontier models take roughly a million tokens of input, a tempting shortcut appears: skip retrieval, paste the entire knowledge base into every request. It works in a demo. Here is what it costs in production, with every assumption stated so you can substitute your own.
Assume a 40-person firm running an internal assistant, 15 questions per employee per working day, so roughly 13,000 questions per month. Assume the knowledge base is 800 documents averaging 1,000 tokens: 800,000 tokens in total, which does fit in a million-token window. Answers average 600 output tokens. Prices are August 2026 list prices for a mid-tier production model at roughly $2 per million input and $12 per million output, with long-context requests billed at double the standard input rate, and cached reads at ten percent of whichever input rate applies.
- Paste everything, every time: 800,000 tokens at the long-context rate of $4 per million is $3.20 per question on input alone, times 13,000 questions ≈ $41,600 per month. That is the number the demo does not show you.
- Paste everything, with prompt caching: cached reads at $0.40 per million bring the same payload to about $0.32 per question on input alone ≈ $4,160 per month, plus cache-write premiums each time the cache expires.
- Retrieve and send only what is relevant: eight passages plus instructions is roughly 7,500 input tokens, about $0.015 per question, plus $0.007 of output ≈ $290 per month, plus embedding and hosting the index.
A 140x spread between the first and third option, on identical questions and the same underlying documents. Two honest caveats. First, Anthropic-style prompt caching only pays off if the same prefix is reused inside its time-to-live window — a five-minute or one-hour cache stays warm at 600 questions a day and is nearly useless at 30. Second, these are API list prices and they move: check the current vendor pricing pages before you build a business case on them, and remember token cost is rarely the biggest line in an SME AI project anyway. What a project actually costs is a different question, and we answer it separately.
When should you actually fine-tune?
Fine-tuning is the right tool in three situations, none of which is ‘the model does not know our products’.
The first is output form: you need every response to land in the same structure, tone, or schema, every time, and prompting keeps drifting. The second is distillation for cost: you have a narrow, high-volume task where a frontier model performs well, and you want a small tuned model to match it at a fraction of the inference cost. The third is a genuinely idiosyncratic domain vocabulary that base models keep getting wrong.
That narrowing is not just our opinion. In May 2026 OpenAI began winding down its self-serve fine-tuning platform: new organisations could no longer create training jobs, existing customers lose new job creation on 6 January 2027, and already-tuned models run only until their base model is deprecated. The stated reason was that current base models follow instructions and formats well enough that prompt-based approaches are cheaper and faster. The largest closed-model vendor walked away from hosted fine-tuning because most of what people were using it for had stopped being necessary — which is exactly the argument above, made by the party with the least incentive to make it.
In practice most SME fine-tuning that is worth doing is a thin adapter (LoRA or QLoRA) on top of a strong base model, sitting alongside retrieval rather than replacing it. The behaviour comes from the adapter, the facts come from retrieval. If someone proposes fine-tuning as the way to make the model ‘know your business’, that is the moment to ask which of the three situations above applies.
How do you decide, in one pass?
Take the ten questions you most want answered and sort them:
- The answer is a fact recorded in a system → tool call. Give the model three or four named, validated operations against the ERP or bookkeeping package, not a database dump. The 2026-07-28 MCP specification made this plumbing stateless and ordinary, which is the point at which it stops being a specialist project and becomes a normal AI agent build.
- The answer is written down in prose → retrieval. Manuals, contracts, policies, past proposals.
- There is no answer, only a required shape → fine-tune, thinly, or improve the prompt first.
- You need reasoning across one whole document → long context, no retrieval. Reviewing a single 200-page tender is exactly what the big window is for.
- Fewer than ten questions, asked occasionally → paste the context. Not every problem deserves an architecture.
Most 20-to-50-person firms find that six or seven of their ten questions land in the first bucket, one or two in the second, and none in the third. That result is not a disappointment. It means the work is an integration project with a language model attached, which is far more predictable to scope, cheaper to run, and much easier to keep correct than a model that has been taught to remember things.
What has to be true before any of this works
Three preconditions, in order. Someone owns the data source and can say what is authoritative when two systems disagree — and no, you do not need a data warehouse first. Access rules are written down before anything is indexed, because retroactively adding permissions to a live assistant is painful and, under the AVG, uncomfortable; a short internal AI policy is the cheapest way to settle that. And you have that evaluation set: the questions, the agreed answers, and a number you can watch move.
One reassurance that removes the fear behind the original question: on the business and API tiers of the major providers, your prompts and files are not used to train their models by default. The risk you are managing is not the model quietly absorbing your data. It is far more ordinary — who inside your own company can now ask questions they previously could not, and whether the answers they get are right.
Last updated 19 August 2026.
