RAG vs fine-tuning vs prompting: how to give AI your company's knowledge

Almost every company that wants AI to "know our business" ends up in the same conversation. Someone has heard that you have to train the model on your documents, someone else says you only need search, and a vendor proposes a project that costs more than either. The choice matters because the three common approaches fail in different ways, and the wrong one is expensive to unwind.

RAG vs fine-tuning vs prompting: how do they compare?

Use prompting when the context is small and stable, RAG when the AI must answer from large or changing documents and show its sources, and fine-tuning when the model must behave the same way every time.

• Use prompting when everything the AI needs fits in the request and rarely changes.
• Use RAG when answers must come from documents that change, differ by user permission or need a citation.
• Use fine-tuning when the answers are right but the format, tone or category is inconsistent.

What are prompting, RAG and fine-tuning?

Each approach answers a different question about where the AI gets its information and its habits.

• Prompting means writing instructions and supplying context in the request itself. You tell the model its role, the format you want, and paste in the material it needs for this task.
• Retrieval-augmented generation (RAG) means the system first searches your documents for the passages relevant to the question, then gives those passages to the model along with the question. The model answers from what was retrieved.
• Fine-tuning means training an existing model further on examples you supply, so that it behaves in a particular way without being told each time.

A useful way to hold the difference: prompting and RAG change what the model sees. Fine-tuning changes how the model behaves. OpenAI's guidance frames it the same way, describing RAG as giving the model access to domain-specific context, and fine-tuning as making sure the model can learn a task so it performs it consistently and reliably.

When is prompting enough?

More often than people expect. OpenAI's accuracy guide says prompt engineering is typically the best place to start, and notes that tasks such as summarization and translation can reach production accuracy with prompting alone.

Prompting fits when the context is small, stable and the same for everyone. A twelve-page policy, a pricing sheet that changes twice a year, a house style guide, a fixed set of product descriptions: all of these can sit directly in the request. Modern models accept long inputs, so "small" is generous. You get the simplest possible design, nothing to index, and nothing to keep in sync.

It stops being enough in three situations: the material is too large or varies too much to include every time, different users are allowed to see different documents, or the answers need to cite a specific source.

What are the advantages of RAG over fine-tuning?

RAG is the right default when the AI has to answer from facts that change, from more material than fits in one request, or with a source the user can check.

The practical advantages over fine-tuning are the ones a business reader cares about. AWS's guidance for choosing between the approaches notes that RAG can incorporate the latest documents in a few minutes, that its responses can point to the information source, and that grounding the answer in retrieved text reduces the risk of hallucination, where a model produces a plausible but wrong answer. A fine-tuned model, by contrast, does not provide a reference to the source in its responses.

That makes retrieval the natural fit for questions like "what is our current returns policy for wholesale customers" or "which clause in this contract covers early termination." The answer lives in a document, the document gets updated, and the user wants to see where it came from. Our guide to building an AI assistant on your company knowledge covers this pattern end to end.

RAG has limits worth knowing. AWS points out that it does not work well for summarizing information from entire documents, because retrieval returns passages rather than the whole file. And irrelevant retrieved content can make answers worse, which is why the quality of the search step, not the model, usually decides whether a RAG system is any good.

What are the advantages of fine-tuning over RAG?

Fine-tuning beats RAG when the problem is consistency of behavior rather than missing information. AWS lists two advantages: a model trained without labels on your own content can create text that more closely matches your organization's style, and one trained on proprietary or regulatory data can help follow in-house or industry-specific standards. OpenAI ties it to cases where you need consistent format, tone and reasoning, and it notes that the quality of training data matters more than the quantity.

Good candidates:

• A fixed output format. Every extraction must return the same fields in the same structure, and prompting still produces occasional drift.
• A narrow classification. Routing support requests into twenty internal categories, or tagging documents with your own taxonomy, where you have hundreds of labeled examples.
• A house voice or style. Drafts that must match how your company writes, at volume, without a long prompt on every call.
• Shorter prompts at scale. When a very long instruction set is repeated on millions of calls, training the behavior in can reduce what you send each time.

The mistake to avoid is fine-tuning to add facts. Training bakes in a snapshot of whatever the data said on the day you trained. It cannot cite a source, it does not update when a policy changes, and it can raise the risk of confident wrong answers. AWS lists the same drawbacks: fine-tuning can take hours to days, so it is a poor solution when your documents change frequently, and it can increase the risk of hallucination when used to answer questions.

Illustrative example: Consider a distributor whose customer service team answers questions from a large pricing and returns policy that is revised every month. An early proposal suggests fine-tuning a model on the policy so it "knows" it. The first version works on the day it is trained. Three weeks later a returns window changes, the model keeps quoting the old one, and nobody can tell which answers are out of date because none of them cites a source. Rebuilding the same assistant with retrieval fixes both problems: the policy document is re-indexed when it changes, and every answer links to the paragraph it came from. Fine-tuning is only worth revisiting later, if the replies are accurate but inconsistent in tone.

Can you combine RAG and fine-tuning?

Yes, RAG and fine-tuning can be combined, and the approaches are additive rather than exclusive. OpenAI's guide describes them that way, and the AWS guidance notes that you can combine fine-tuning and RAG in a single system, with the model both trained for the task and given retrieved context. In practice, start with prompting, add retrieval when the context outgrows the prompt, and fine-tune only when you have measured a behavior problem that the other two did not solve.

What drives the cost of RAG, fine-tuning and prompting?

Each approach spends money in a different place, and the cheap one to build is not always the cheap one to run.

• Prompting costs are mostly per-call. Every token you paste in is billed on every request, so a large static prompt on a high-volume workflow adds up. Some providers discount repeated prompt prefixes, which is worth checking.
• RAG has setup costs (connecting sources, cleaning documents, building the index) and running costs (storage and search for the index, plus the longer model input that retrieved passages create). The largest ongoing cost is human: someone has to own document quality, because stale or contradictory source files produce stale or contradictory answers.
• Fine-tuning has up-front costs for preparing and labeling training data and running the training, then a recurring cost every time the base model is retired or your requirements change, because the tuned model has to be rebuilt. Fine-tuned models also tie you more tightly to one provider, which adds to vendor lock-in.

No honest price exists for any of the three without knowing your volume and data, so treat anyone who gives you a number before seeing your documents with caution. A discovery call is the right place to size it against your own workload.

How do you keep retrieval inside document permissions?

Document permissions are the control most RAG pilots skip, and skipping them causes the worst incidents. If a RAG assistant searches across every document in the index, it can quote an HR file, a board paper or a salary table to anyone who asks the right question. The model is not the control. Access has to be enforced in the retrieval step, before the passages ever reach the model.

Microsoft's documentation for Azure AI Search describes the pattern: permission metadata is stored alongside each indexed document, and at query time the search service compares the caller's identity to that metadata and returns only the documents the caller is authorized to read. Several mechanisms exist, including simple security filters, access control lists from file storage, Microsoft Purview sensitivity labels and SharePoint permissions. The same documentation is explicit about a trade-off that applies to any RAG design: permission changes in the source system only show up in search results after that metadata is synchronized into the index.

That gives you a short checklist for any retrieval project:

1. Decide which sources are in scope, and leave out anything you cannot permission correctly.
2. Carry each document's access rules into the index, not only its text.
3. Filter by the signed-in user's identity on every query, never by a shared service account.
4. Set how quickly permission changes in SharePoint, a file share or a CRM must reach the index, and test that a revoked user loses access in that time.
5. Log what each user asked and which documents were returned, so an access question can be answered later.

Fine-tuning has the opposite problem. Anything in the training data can surface in any answer to any user, and you cannot remove one document after the fact. Do not train on content that only some people are allowed to see. For the wider controls, see data security and governance for AI.

How do you decide which approach to use?

Work through these questions in order and stop at the first one that applies.

1. Does everything the AI needs fit in one request and rarely change? Use prompting.
2. Does the AI need to answer from documents that are large, change often, differ by user, or must be cited? Use RAG, with permissions designed in from the start.
3. Are answers correct but inconsistent in format, tone or category? Add fine-tuning, with a test set that proves it helped.
4. Is the goal to make the model "know" a body of facts? Go back to question 2. That is a retrieval problem.

Whichever you choose, measure it. Build a set of real questions with agreed answers before you build, and rerun it after every change to the prompt, the index or the model. Our guide to testing AI before it touches your operations explains how. Without that, the choice between these approaches becomes an argument about opinions instead of results.

Common mistakes

• Fine-tuning to add facts. It produces answers that sound authoritative, cannot cite a source and go stale silently.
• Skipping the prompting baseline. Teams jump to retrieval or training without finding out what a good prompt already does.
• Indexing everything. More documents do not improve answers. Duplicates, drafts and outdated versions make retrieval worse.
• Ignoring permissions until after the pilot. Retrofitting access control onto an index built without it usually means rebuilding the index.
• No owner for the content. Retrieval is only as current as the documents behind it, and someone named has to keep them accurate.
• Treating the choice as permanent. Models and tooling change quickly, so keep the approach behind a clean interface and keep your test set portable.

How Kastling approaches this

Choosing between these approaches is part of the design work in AI & Cloud Infrastructure: cloud architecture and deployment on AWS or Azure, AI application deployment, and the data connections and pipelines behind them.

In practice that means starting from the business task and the documents involved, rather than from a preferred technique. Kastling connects the systems a client already runs, agrees data access and ownership up front, designs permissions into any retrieval layer, and tests the result against real business scenarios before it goes live. Where a workload needs it, the team brings in AWS and Azure specialists. The team is founder-led, and the scope, responsibilities and operating costs are set out in the proposal rather than assumed.