Building an AI assistant on your company knowledge

Ask a new starter how they find the current expenses policy and you will hear the real answer: they ask the colleague who sits nearest. The document exists, probably in three versions across two systems, and nobody is confident which one is live. That is the problem a knowledge assistant is meant to solve, and it is a content problem as much as a technology one.

What is retrieval-augmented generation, in plain English?

Retrieval-augmented generation, usually shortened to RAG, is the pattern behind almost every company knowledge assistant. AWS describes it as "the process of optimizing the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response" (AWS).

In practice it works in four steps.

1. Index. Your approved documents are split into passages and stored in a way that supports search by meaning as well as by keyword.
2. Retrieve. When someone asks a question, the system finds the passages most likely to contain the answer, filtered by what that person is allowed to see.
3. Ground. Those passages are supplied to the model alongside the question, with instructions to answer from them.
4. Cite. The answer comes back with links to the source documents, which AWS notes lets users verify information themselves and increases trust in the result.

Two consequences follow, and both matter more than the choice of model. Accuracy is a function of the sources, so a well written policy indexed correctly beats a larger model reading a stale one. And nothing is being trained, so when a document is corrected the answer changes as soon as it is reindexed.

Which sources should the assistant be allowed to read?

The instinct is to connect everything. Resist it. Indexing every folder in the company is how an assistant begins quoting a 2021 policy that was replaced twice.

Work through three tiers.

• Authoritative. Current policies, approved procedures, the live price list, signed contracts, the product catalog. These are the sources the assistant should answer from, and each one needs a named owner and a review date.
• Useful with context. Past project documentation, meeting notes, resolved support tickets. Valuable for background, but they should not be quoted as policy.
• Excluded. Drafts, personal drives, anything superseded, and anything whose sensitivity means it should never appear in an answer.

Then do the unglamorous part. Retire old versions rather than leaving them beside the new one. Give each source a date. Fix the document titled "Final v3 (updated)" that three teams reference differently. Most disappointing assistant pilots trace back to this step being skipped, not to the retrieval.

How do permissions work?

The rule is simple to state: a person must never see through the assistant anything they could not open directly. Enforcing it means filtering at retrieval, using the asking user's own identity, rather than filtering the answer after the fact.

Microsoft describes exactly this model for Microsoft 365 Copilot. It uses Microsoft Graph to reach data "in the user's unique context", and states that "Copilot only accesses data that an individual user is authorized to access" and "can't access data that the user doesn't have permission to access", while honoring Conditional Access policies and multifactor authentication (Microsoft Learn). A custom assistant has to reproduce that behavior across whatever systems it connects to, which is usually the hardest engineering problem in the project.

Three checks belong in the plan.

1. Test with real roles. Sign in as accounts across different departments and seniority levels and confirm that results differ correctly.
2. Audit for existing oversharing. Search makes findable what was previously merely present. A confidential file in an open folder was always a problem, but nobody could find it before.
3. Log every query and source returned. So a question about what the assistant disclosed has an answer.

What do citations prove, and what do they not?

Citations are the single most useful feature of a knowledge assistant, and the most misread. A citation shows which passage the answer was drawn from, so a reader can check it. It does not prove the answer is a correct reading of that passage, and it does not prove the passage is current.

Design for checking rather than trust. Link to the specific section or page rather than the document. Show the retrieved text alongside the answer, so the reader can compare in one glance. Display the document's date and owner, because an answer sourced from a superseded policy looks identical to a correct one. Where an answer draws on several documents, attribute each claim separately.

What should happen when information is missing or sources conflict?

Most assistants are designed for the case where the answer exists in one place. The interesting cases are the other two.

Missing. When retrieval returns nothing relevant, the assistant should say so plainly and point to who owns that subject, rather than filling the gap from general knowledge. Test this deliberately by asking questions your documents do not cover. An assistant that answers everything confidently is worse than one that declines.

Conflicting. When two documents disagree, the assistant should show both and say which is more recent or more authoritative rather than silently choosing. In practice a conflict is a content defect, so route it to the source owner as a task. Over a few months this turns the assistant into a way of finding stale documentation, which is worth as much as the answers.

Out of scope. Decide where the boundary sits. Legal interpretation, HR casework and anything requiring professional judgment should return the relevant material and the right contact, not an opinion.

The OWASP Top 10 for LLM Applications 2025 names misinformation, sensitive information disclosure and prompt injection among the leading risks for this kind of system (OWASP). A knowledge assistant touches all three, which is why source control, permissions and refusal behavior are the design, not the polish.

Does a customer-facing assistant change the requirements?

Yes, considerably. The mechanism is the same and the tolerance for error is not.

A common and sensible sequence is to launch internally first, use several months of real questions to find the gaps in your content, then expose a narrower, tested subset to customers.

How do you evaluate an assistant before and after launch?

Impressions from a demo tell you almost nothing. Build a test set instead.

1. Collect real questions. Fifty to two hundred, gathered from the service desk, the shared inbox and the people who get interrupted most.
2. Write the correct answer and the source for each. This is the work. It is also a content audit you would benefit from anyway.
3. Include the awkward ones. Questions your documents do not answer, questions where two documents disagree, and questions that should be refused.
4. Score each answer on four things. Is it correct, is the cited source right, is it complete, and did it decline when it should have?
5. Rerun the set on every change. New sources, a new model version, a changed prompt. Quality moves in both directions.

After launch, track usage rather than assuming it. Measure how many people ask a question in a given week, the share of answers marked useful, how often people click through to the source, and which questions return nothing. That last measure is the roadmap for your content.

Illustrative example: Consider a professional services firm whose consultants repeatedly ask the same questions about expense limits, engagement letter templates and which insurer covers which client type. The firm indexes three sources only: the current staff handbook, the approved template library and the client onboarding procedure, each with a named owner and a review date. Permissions are inherited from SharePoint, so a consultant cannot retrieve a document they could not open. A test set of 120 real questions is built from the operations inbox, including fifteen the documents do not cover. The assistant is launched to one practice group, and the questions that return nothing are reviewed monthly and turned into documentation tasks.

For testing method in more depth, see how to test AI before it touches your operations, and for the access and retention decisions, data security and governance for AI in the enterprise.

Should you buy Microsoft 365 Copilot, Gemini for Workspace or Glean instead?

Often, yes. Buying is the right answer more often than suppliers admit.

• Microsoft 365 Copilot suits organizations whose knowledge already sits in Microsoft 365, because it works within your tenant and inherits Microsoft Graph permissions. Microsoft lists Microsoft 365 Copilot Business as an add-on at $18.00 per user per month paid yearly, noting that "a separate license for a qualifying Microsoft 365 plan is required", alongside bundles such as Microsoft 365 Business Standard with Copilot at $23.50 per user per month (Microsoft).
• Gemini for Workspace is the equivalent for Google Workspace customers, now included in the plans rather than sold separately. Google lists Business Starter at $7.00 per user per month with the Gemini assistant in Gmail, and Business Standard at $14.00 with Gemini across Gmail, Docs, Meet and more (Google Workspace).
• Glean and similar enterprise search platforms connect many systems at once. Glean describes itself as "an AI platform for work that unifies enterprise search, assistants, and agents on your company's context" (Glean).

The mixed answer is common and perfectly reasonable: licensed assistants for general documents, and a custom assistant for the one domain where the answers live somewhere the suite cannot reach. For the broader picture of where retrieval fits alongside other automation, read the practical guide to AI workflow automation.

How Kastling approaches company knowledge assistants

Company knowledge and search is part of our AI Integration & Automation service: helping employees find and synthesize information from approved internal sources, with source references and suitable access rules. A discovery call is free and covers which questions people actually ask, where the answers live today and who owns them.

Before anything is built we agree which sources are authoritative, how permissions will be enforced and what should happen when information is missing, conflicting or uncertain. We test against real questions with known answers, involve the people who will use it, and agree data access, retention and ownership up front. Source references support review; they do not guarantee that every generated answer is correct, and we would rather say that plainly than promise otherwise.