Open source RAG knowledge base local deployment nanny-level tutorial: Ollama + AnythingLLM privatization Q&A

🛒 For operation and maintenance and business personnel without in-depth development background, a complete tutorial on building a private knowledge base with zero code.

Tutorial Objectives

This tutorial uses two open source components to build a local RAG knowledge base that can be used for Q&A by uploading documents, answers with sources, and can be run offline:

  • Ollama: Inference service for running large models locally.
  • AnythingLLM: One-stop knowledge base application with interface (including vector library and Q&A).

There is no need to write code in the whole process, and it is suitable for operation and maintenance, operations and business personnel.

Preparation Checklist

  • [ ] A computer or server: 16G memory or more is recommended; macOS/Linux/Windows are acceptable.
  • [ ] Prepare several test documents (PDF/Word/TXT/Markdown are acceptable).
  • [ ] The network is available (the model and software need to be downloaded for the first time, and can be used offline after that).
  • [ ] Reserve 8-20G disk space (depending on model size).
  • [ ] (Optional) NVIDIA graphics card significantly improves answering speed.

Version Tip: The following version number, model size and download address are subject to the official real-time page; the model pull command is subject to the Ollama official model library.

Step 1: Install and configure Ollama

Visit the Ollama official website to download the corresponding system installation package (drag directly into Applications for macOS; use the installation script for Linux; use the installation package for Windows). After installation, open the terminal to verify:

ollama --version

Normally the version number will be output, such as ollama version 0.x.x.

Step 2: Pull the local model

The knowledge base requires two models: one for generating answers and one for vectorizing the document (embedding). First pull the dialogue model:

ollama pull qwen2.5:7b

Then pull the vectorization model:

ollama pull nomic-embed-text

Verify that the model is ready:

ollama list

The model is large in size, and it may take some time to pull it for the first time depending on the network speed; the model naming and available list are subject to the official Ollama real-time page.

Step 3: Install AnythingLLM

Download the desktop version installation package from the AnythingLLM official website and install it (supports macOS/Windows/Linux). When starting for the first time, you will enter the initialization wizard:

  1. Select the "Local Vector Library" type (the built-in LanceDB is used by default, no additional installation is required).
  2. Select LLM provider as Ollama.
  3. Fill in the Ollama address http://localhost:11434, and select qwen2.5:7b pulled in the previous step.
  4. Select the Embedding provider as Ollama and the model as nomic-embed-text.

After saving the configuration, the connection is completed.

Step 4: Create a knowledge base and upload documents

  1. Click "Workspace" on the left side of AnythingLLM and create a new workspace, such as Company System.
  2. Enter the workspace → "Upload Document" and drag in the prepared divided document.
  3. Click "Process" and wait for vectorization to complete.

After vectorization is successful, each document will display a "Processed" status, indicating that it has entered the searchable vector library.

Step 5: Start Q&A and verify traceability

Enter a question into the workspace dialog box, such as "According to the documentation, how many steps does the reimbursement process take?"

Expected performance:

  • The content of the answer comes from the document you uploaded, rather than speaking in general terms.
  • The quoted source fragment is displayed below the answer. You can click to jump to the original text location.

Open AnythingLLM's "Chat" settings and turn on "Show citations" to show the source of every answer.

Step 6: Effect tuning

If the answer is not satisfactory, try in order:

  1. Adjust chunking: Chunk size and overlap can be configured in the AnythingLLM settings (chunk size is recommended to be 500-1000, overlap 50-100).
  2. Change to a stronger model: ollama pull qwen2.5:14b or larger model.
  3. Change to a better vector model: use a higher-dimensional embedding model and switch in the configuration.
  4. Supplementary documentation: Add missing knowledge to the knowledge base and process it again.

Step 7: Customize prompt words and answer style

AnythingLLM allows you to customize system prompts (System Prompt) to make answers more suitable for the team's tone. For example, fill in the following in "Chat Settings":

You are the company’s knowledge assistant. Only answer based on uploaded documents, do not make up;
Answer in concise Chinese, giving conclusions first and then the basis;
When no evidence is found, it is clearly stated that "there is no relevant content in the database."

Customizing prompt words can make answers more "like your own" and is also an effective means to reduce hallucinations.

Step 8: Data backup and upgrade

  • Backup: The data directory of AnythingLLM (including vector library and configuration) is copied regularly. It is recommended to include it in the daily backup task.
  • Upgrade: Back up before upgrading, then update the software version and re-verify the Q&A.
  • Document update: After the document content changes, delete the old version and re-upload it to avoid conflicts between the old and new versions.

Verification method

  • Verification 1: ollama list can see the dialogue model and vector model.
  • Verification 2: After uploading the document, the status is "Processed".
  • Verification 3: Ask a factual question in the document, and the answer can cite the corresponding source.
  • Verification 4: Disconnect from the external network, Q&A is still available (confirm that it is local inference).
  • Verification 5: After customizing the prompt words, unfounded questions can be correctly rejected.

Common mistakes and precautions

  1. The document format is confusing: first unify it into selectable text (OCR the scanned document first), and then import it in batches.
  2. The block is too large: Retrieval distortion occurs when a single block exceeds the model context. Adjust according to the recommended value.
  3. Multi-language mixing: Try to unify the languages ​​in a knowledge base, or divide work areas according to languages.
  4. Forgot to turn on references: Be sure to turn on "Show reference sources" in the configuration. This is a hard item for acceptance.

Frequently Asked Questions and Troubleshooting (FAQ)

  1. What should I do if AnythingLLM cannot connect to Ollama?

First confirm that ollama serve is running (ollama serve listens to 11434 by default), and then use the browser to access http://localhost:11434 to verify; do not include extra slashes in the address in the configuration.

  1. Model download is slow or fails?

    When the network is limited, use the mirror address or try again at a different time; you can also download the GGUF model and import it manually.

  2. The answer does not quote the document at all, like chatting nonsense?

    Check whether Embedding is configured as a local Ollama model; if not configured, the document cannot be retrieved correctly.

  3. Is the uploaded PDF garbled or incompletely recognized?

    AnythingLLM relies on the built-in parser; the scanned PDF needs to be OCR-converted into selectable text first.

  4. Insufficient memory and laggy operation?

    Change to a smaller quantization model (such as qwen2.5:3b), and close other large memory programs; 7B model recommends more than 16G memory.

  5. How can multiple computers share a knowledge base?

    Deploy AnythingLLM as a Docker server and configure a shared data volume for clients to access through the LAN.

Summary and next steps

At this point, you already have a set of RAG applications of "local model + local knowledge base + traceable question and answer". Suggested next step: first use real business documents for a small-scale trial for 1-2 weeks to collect high-frequency problems and retrieval blind spots, and then decide whether to upgrade the model, switch vector libraries, or deploy to multiple users.

Advancement and Expansion

  • Change to FastGPT / Flowise: migrate when visual workflow and complex branches are required.
  • Vector library upgrade: After the amount of documents becomes large, replace LanceDB with a professional vector library such as Milvus.
  • Hybrid search: Enable keyword and vector search at the same time to improve long-tail question hits.
  • Multi-user permissions: Deploy the server and access the account system to achieve department-level knowledge isolation.
  • Access Agent: Make the knowledge base a search tool for agents, linking work orders, customer service and other business processes.

User Reviews

  • Loading reviews...