Staff Keep Asking the Same Questions? An AI Knowledge Assistant in Slack

Every growing team pays the same quiet tax. Someone asks in a channel where the refund policy lives. Someone else asks how to request a laptop, or which onboarding checklist is current. A senior person stops what they are doing to answer, often from memory and sometimes wrongly.

The documents exist. They are spread across Notion, Confluence, Google Drive and SharePoint, with different owners and naming habits. Search inside each tool is weak, so people ask a colleague instead. That drains focus from experienced staff, slows down new hires and lets outdated answers spread. An internal knowledge assistant fixes this by answering where people already ask: Slack or Microsoft Teams.

What this assistant does

The assistant uses retrieval-augmented generation (RAG): it finds the relevant passages in your documents, then has a language model answer using only those passages. In practice, it does five things:

• Answers questions in Slack or Teams. Staff mention the bot in a channel or message it directly, in plain language.
• Shows its sources. Every answer links to the documents it used, so people can check the original and its date.
• Respects document permissions. A person only gets answers from documents they can already open in the source system.
• Keeps itself up to date. When a document is edited, moved or deleted, the index follows within an agreed time.
• Says when it does not know. If nothing relevant is found, it says so and suggests who to ask.

It does not replace your wiki. It makes the wiki usable by putting good search and a short summary one message away.

How it works, step by step

1. Ingest documents. Connectors read pages and files from Notion, Confluence, Google Drive and SharePoint through their official APIs. For each document we store its ID, owner, link, last-modified date and permissions.
2. Split and index. Long documents are split into sections that keep their headings. Each section becomes an embedding, a numeric fingerprint of its meaning, stored in a vector database. A keyword index sits alongside it for exact terms like policy codes.
3. Retrieve for each question. When someone asks, the system identifies them, filters out anything they cannot access, then finds the best-matching sections.
4. Answer with citations. The model writes a short answer from those sections only and links each point to its source. If sources disagree or are missing, it says so.
5. Log and improve. Questions, retrieved sources and thumbs-up or thumbs-down feedback are logged, keeping personal data to a minimum. We review weak answers and fix the cause, which is often a missing or outdated document.

Keeping the index fresh is a job of its own. We use change notifications (webhooks) where the source offers them, and a scheduled sync where it does not. Deleted documents are removed from the index, not just hidden.

Tools we use

• Backend: Python with FastAPI for the ingestion jobs and the question-answering service.
• Vector database: pgvector inside Postgres for most teams, or Qdrant when the index is large or needs heavy filtering.
• Language model: OpenAI or Anthropic Claude through their business APIs, or a private open-source model when data cannot leave your cloud.
• Chat front end: a Slack app or a Microsoft Teams bot, plus an optional web page for people outside those tools.
• Connectors: the official APIs of Notion, Confluence and Google Drive, and Microsoft Graph for SharePoint.

Check each vendor's current pricing, data-retention terms and API limits before you commit. They change, and they affect both cost and privacy.

What you need to get started

• A list of the sources that matter most, and an owner for each one.
• Admin access, or an admin willing to approve app installs, in Slack or Teams and in each document system.
• Forty to a hundred real questions staff have asked recently, with the correct answer or the document that holds it.
• A decision on which model provider is acceptable for your data, made with whoever owns security and privacy.
• A named person who will fix or retire documents when the assistant exposes gaps.

Typical scope and timeline

A first version is typically 2 to 4 weeks of work, depending on how many sources you connect and how complex their permissions are. Treat that as an estimate, not a promise. One Confluence space feeding a Slack bot sits at the short end. Four sources with mixed sharing rules and a Teams rollout sits at the long end.

A common sequence looks like this:

• Week 1: connect one or two sources, build the evaluation set and agree the permission model.
• Week 2: retrieval, answers with citations, and the Slack or Teams app for a pilot group.
• Weeks 3 to 4: remaining sources, sync for edits and deletions, monitoring and a wider rollout.

How we keep answers accurate

• An evaluation set of real questions. We test every change against the questions you supplied. We track how many get a correct answer with the right source.
• Citations on every answer. People can verify in one click, and wrong sources are easy to spot and report.
• Refusal when unsure. If the retrieved passages do not support an answer, the assistant says it could not find one.
• Freshness signals. Answers show each source's last-updated date. Stale documents can be ranked lower or flagged for review.
• Monitoring. A weekly report lists unanswered questions, negative feedback and the most-cited documents. Unanswered questions are your documentation backlog.

Risks and how we handle them

Privacy

Not every document belongs in the index. We agree what is in scope, exclude areas such as HR case files, and keep logs lean. Our guide to RAG privacy by design covers deletion, regional hosting and logging in detail.

Permissions

The biggest risk is a correct answer from a document the asker should not see. We filter by the asker's identity at retrieval time, before anything reaches the model. A prompt instruction is never the only control.

Wrong answers

Language models can sound confident when they are wrong. Citations, refusal rules and the evaluation set reduce this. Staff are also told to check the source for anything with real consequences.

Outdated documents

The assistant is only as current as your documents. Sync keeps the index in step with the sources, but two conflicting policies will still confuse it. The weekly report makes those conflicts visible so owners can resolve them.

When not to build this

• Your documentation is thin or badly out of date. Fix the twenty most-used documents first; the assistant cannot invent good content.
• Your team is small enough that a tidy wiki and a pinned channel already work.
• The AI search built into your existing tool covers your sources and permissions well enough. Try it first.
• Nobody will own the content. Without an owner, answers drift as documents age.

How UnlockLive can help

We build internal assistants end to end: connectors, permission-aware retrieval, the Slack or Teams app, and the evaluation and monitoring that keep it honest. See our RAG development service. If you also want the assistant to trigger tasks, such as opening a ticket or starting a request, see AI Workflow Automation.

Related reads in this series: automating employee onboarding, the HR policy and contract Q&A assistant, and private RAG for regulated data. If you want a second opinion on your sources and scope, book a free 30-minute call.