How to build a RAG chatbot over your own docs

By the end of this guide you will have a chatbot that answers questions from your own documents, shows which file each answer came from, and has been tested against a set of real questions. The pattern is called retrieval-augmented generation, or RAG: your documents are split and stored in a searchable index, and the chatbot looks up the relevant passages before the language model writes its reply.

If you want to build a RAG chatbot, the real decision is how much of the pipeline you write yourself. You can code it with a framework, use a visual builder, or hand the plumbing to a managed service. This guide covers all three and compares seven tools from our Developer platforms category.

Key takeaways

• A RAG chatbot has four moving parts: loading and splitting documents, an embedding and vector store, a retrieval step, and a language model that answers from the retrieved text.
• The main decision is build style: code with a framework (LangChain, LlamaIndex), drag and drop in a visual builder (Dify, Flowise), or use a managed RAG service (Agentset) with a vector database such as Pinecone or LanceDB.
• Free tiers exist but are capped, for example 1,000 pages and 10,000 retrievals on Agentset, or 10,000 credits a month on LlamaIndex's hosted platform.
• Test with real questions before launch. A chatbot that cites the wrong passage is worse than one that says it does not know.

What you need before you start

• A set of documents. PDFs, Word files, web pages or exported wiki pages that you have the right to use. Remove anything confidential that the chatbot's users should not see.
• Twenty or so real questions. Questions your users would ask, with the document that holds the answer. You will use these to test.
• A language model account. An API key from the model provider you plan to use, or a platform that supplies model credits.
• Some development comfort. The framework route needs Python or JavaScript. The visual builders need much less code.
• Time. A working prototype takes under a day. A tested version you trust takes longer.
• Difficulty. Moderate for the visual builders, harder for a code-first build.

Which kind of AI tool fits this job

Match the tool to how much control you need.

A code framework for full control. LangChain is an open-source framework for building applications and agents on language models, with LangSmith as a hosted add-on for tracing and evals. LlamaIndex is an open-source framework for ingesting, indexing and querying your own documents, plus the hosted LlamaParse platform for parsing and extraction, with parsing that handles 130+ file formats.

A visual builder when you want a working chatbot fast. Dify has a knowledge base for RAG, chatbot and workflow app types, and publishes apps as a web app or an API. It runs on Dify Cloud or as the open-source Community edition. Flowise is an open-source visual builder with Assistant, Chatflow and Agentflow modes, an embeddable chat widget and an API. You can self-host it or use Flowise Cloud.

A managed RAG service when you do not want to run the pipeline. Agentset handles file parsing, chunking, embeddings, vector storage, hybrid search and reranking, and returns citations with every answer. You use JavaScript or Python SDKs.

A vector store for the index itself. Pinecone is a fully managed vector database with serverless indexes, namespaces and inference for embedding and reranking. LanceDB is an open-source vector database that runs embedded in your own application, with Python, TypeScript and Rust SDKs. You pick one of these when you build with a framework, or when your builder lets you choose the store.

Step 1: Collect and clean your documents

Retrieval is only as good as the text you give it, so start with the sources.

1. List every source the chatbot should know: help pages, policies, manuals, specifications.
2. Remove duplicates and out-of-date versions, so the bot does not cite old rules.
3. Convert scans and images to text where you can. LlamaIndex's LlamaParse lists 130+ file formats, including images and spreadsheets, and Agentset lists support for images and tables.
4. Add metadata where you can, such as document type, product or date, because filters help later.

Check it worked by opening a few cleaned files and confirming the text is readable and complete.

Step 2: Choose how you will build it

You end this step with a build route and an account.

1. Pick the route: code with a framework, drag and drop in a builder, or use a managed service.
2. Check the free tier against your document volume. Agentset's Free plan caps at 1,000 pages and 10,000 retrievals a month, and Dify's Sandbox allows 200 message credits, 5 apps and 1 team member.
3. Decide where it will run. Dify, Flowise and LanceDB can be self-hosted. Pinecone is managed.
4. Use the comparison table below to line up pricing and stated limits.

Check it worked by confirming your document count and expected monthly questions fit inside the plan you chose.

Step 3: Ingest, split and embed the documents

The goal is an index you can search by meaning.

1. Upload the files to the knowledge base or load them in code. Dify has a knowledge base with batch document upload, and Flowise lets an Assistant retrieve answers from uploaded files.
2. Let the tool split the text into passages. Keep passages small enough to be specific and large enough to make sense on their own.
3. Choose an embedding model. Pinecone offers embedding and reranking models through Pinecone Inference, and Agentset lets you pick your own embedding model.
4. Store the vectors. Use namespaces in Pinecone to keep different document sets apart, or a LanceDB table if you want an embedded store.

Check it worked by searching the index for a phrase from one document and seeing the right passage come back.

Step 4: Set up retrieval

Retrieval decides which passages the model sees, so it deserves the most tuning.

1. Start with plain semantic search, then add hybrid search if exact terms such as product codes matter. Pinecone, LanceDB and Agentset list hybrid search.
2. Add reranking to reorder the top passages. Agentset and Pinecone both list it.
3. Use metadata filters to limit results by product, version or audience. Agentset lists metadata filtering.
4. Keep the number of passages passed to the model modest, so the answer is not diluted.

Check it worked by running five of your test questions and reading the retrieved passages before the model answers.

Step 5: Connect the language model and write the prompt

Now the model turns retrieved text into an answer.

1. Select a model through your builder or framework. Dify lets you try models from several providers using message credits before you switch to your own API key.
2. Write a short system prompt: answer only from the supplied passages, say you do not know when the passages do not cover the question, and cite the source.
3. Show citations in the interface. Agentset returns citations with every answer, and Dify lets you annotate chat app responses.
4. Keep the prompt in version control or in a prompt tool, so you can change it safely.

Check it worked by asking a question your documents cannot answer and confirming the bot says it does not know.

Step 6: Test with your real questions

This is the step that separates a demo from a tool you can ship.

1. Run your full question set and record the answer, the cited passage and whether it is correct.
2. Mark failures as retrieval problems (the right passage was not found) or generation problems (the passage was found but misused).
3. Fix retrieval first: cleaner documents, different passage sizes, filters or reranking.
4. Use tracing and evaluation features if you have them. LangSmith lists tracing, monitoring and evals, and Flowise lists evaluations with datasets on its Cloud and Enterprise plans.

Check it worked by re-running the set after each change and seeing the number of correct answers go up.

Step 7: Deploy, secure and monitor

Put the chatbot where people will use it, with sensible limits.

1. Publish it. Dify publishes as a web app or an API, Flowise has an embeddable chat widget, and Agentset offers JavaScript and Python SDKs and an MCP server.
2. Restrict access to documents by audience. Do not index material that some users should not see.
3. Tell users that answers come from an AI and may be wrong, and give them a way to reach a person.
4. Review logs of real questions each week and add missing documents.

Check it worked by having a colleague outside the project use it for a day and noting any bad answers.

Tools for this job compared

Tool	Good fit for	Pricing	Stated limits
LangChain	Code-first agents and RAG, with LangSmith tracing (self-hosted options on Enterprise)	Freemium, paid from $39/mo (checked 24 Oct 2026)	Developer plan: 1 seat and 5k base traces a month; usage is metered, so cost varies
LlamaIndex	Parsing and indexing your documents with an open-source framework	Freemium, paid from $50/mo (checked 24 Oct 2026)	Free plan stops at 10,000 credits; files cached 48 hours unless you opt out; framework needs programming
Pinecone	A managed vector store for the index	Freemium, paid from $20/mo (checked 24 Oct 2026)	Starter: AWS us-east-1, up to 5 indexes, 2 GB storage; Standard and Enterprise have monthly minimums
Dify	A visual RAG chatbot builder, cloud or self-hosted	Freemium, paid from $59/mo (checked 24 Oct 2026)	Sandbox: 200 message credits, 5 apps, 1 team member; Community edition is a single workspace
Flowise	Visual chatbots with an embeddable widget	Open source	Cloud Free: 2 flows and 2 assistants; evaluations are Cloud and Enterprise only; paid prices not published
Agentset	Managed ingestion, search and cited answers through SDKs	Freemium, paid from $49/mo (checked 24 Oct 2026)	Free: 1,000 pages and 10,000 retrievals; connectors not on Free and cost $100 each on Pro
LanceDB	An embedded, open-source vector store	Open source	OSS runs in a single process sharing CPU with your app; Enterprise pricing not published

Prices and plan limits change, so confirm the current terms on each vendor's site before you commit.

Common mistakes to avoid

• Indexing everything you have. Old and duplicate files lead to contradictory answers. Fix it by curating sources and removing outdated versions first.
• Skipping the test set. Without real questions you only see the demo cases that work. Fix it by writing questions and expected sources before you build, then re-running them after every change.
• Tuning the prompt when retrieval is the problem. If the right passage never reaches the model, no prompt will help. Fix it by reading retrieved passages before the answer and fixing documents, filters or reranking first.
• Exposing private documents. A shared index can leak material to the wrong audience. Fix it by separating data with namespaces or filters and restricting who can query each set.
• Letting the bot guess. A confident wrong answer damages trust. Fix it by instructing the model to say when the sources do not cover a question, and by showing citations.

How we put this together

This is an evidence-based guide built from each tool's listing in our directory, which is compiled from the vendor's own site and pricing page. It is not a review based on using the tools. We don't rank tools, and nobody paid to be included. Some outbound links may be affiliate links; see our disclosure.

Frequently asked questions

Can I build a RAG chatbot for free?

Yes, at small scale. LangChain and LlamaIndex have free open-source libraries, Flowise and Dify can be self-hosted, and LanceDB's open-source library is free. Hosted tiers are capped: Agentset's Free plan allows 1,000 pages and 10,000 retrievals, and you will still pay for the language model you call.

Do I need to write code?

Not necessarily. Dify and Flowise are visual builders, though you will still configure models and knowledge sources. A framework such as LangChain or LlamaIndex requires programming, as LlamaIndex's listing states.

Is RAG better than fine-tuning for my own documents?

For question answering over documents that change, RAG is usually the simpler starting point because you update the index rather than retrain a model. It also lets the bot cite its sources. Fine-tuning is better suited to changing a model's style or behaviour; see our guide on fine-tuning an LLM on your own data.

Can I keep my documents private?

It depends on where you run the pieces. Dify, Flowise, LanceDB and LlamaIndex's framework can be self-hosted. Pinecone is a managed service, and Agentset is managed with self-hosting on its Enterprise plan. LlamaIndex's hosted platform caches files for 48 hours and says they are not used for model training. Read each vendor's data terms before uploading confidential files.

How long does it take to build?

None of these vendors publish a build time, and it depends on your document set and route. Expect testing, tuning retrieval and securing access to take longer than the first prototype, since that work is what makes the chatbot trustworthy.

Compare more options in our Developer platforms category. For the storage layer, see how to store and search embeddings in a vector database.