Why I Built This

I wanted a chatbot that answers questions from my own documents — without sending data to a cloud API. This RAG setup runs fully offline with Ollama and ChromaDB on my machine.

  • Privacy — your data stays on your machine
  • No API costs — no usage fees or rate limits
  • Speed — no network delay for each query
  • Control — you choose what goes into the knowledge base
  • Offline use — works without internet after setup

Architecture Overview

My local RAG system consists of four main components:

ComponentPurposeTechnology
Document ProcessingClean and chunk text documentsPython + Regex
Vector StoreStore and search embeddingsChromaDB
EmbeddingsConvert text to vectorssentence-transformers
LLM GenerationGenerate responsesOllama + Llama3.1

How It Works

Documents → Preprocessing → Embeddings → ChromaDB Storage
                                             ↓
User Query → Query Embedding → Vector Search → Context Retrieval
                                             ↓
Context + Query → Ollama LLM → Generated Response

Technical Implementation

Prerequisites

Before starting, ensure you have:

  • Python 3.8+
  • Ollama installed with the llama3.1 model
  • A virtual environment set up

Install Ollama:

Arch Linux: sudo pacman -S ollama then sudo systemctl enable --now ollama

Ubuntu: curl -fsSL https://ollama.com/install.sh | sh

Pull the model: ollama pull llama3.1

Dependencies

pip install chromadb>=0.4.0
pip install sentence-transformers>=2.0.0
pip install numpy>=1.21.0
pip install torch>=1.9.0

Project Structure

rag/
├── docs/
│   ├── about_me.txt      # Personal information
│   ├── education.txt     # AI in education knowledge
│   ├── finance.txt       # AI in finance knowledge
│   └── healthcare.txt    # AI in healthcare knowledge
├── preprocess.py         # Document preprocessing
├── rag_local_ollama.py   # Main RAG system
└── requirements.txt      # Dependencies

Document Preprocessing

The preprocessing pipeline handles document cleaning and chunking:

import os
import re

def clean_text(text):
    # Remove HTML tags and normalize whitespace
    text = re.sub(r'<[^>]+>', '', text)
    text = re.sub(r'\s+', ' ', text).strip()
    return text

def chunk_text(text, max_words=100):
    # Split into manageable chunks for embeddings
    words = text.split()
    return [' '.join(words[i:i+max_words]) 
            for i in range(0, len(words), max_words)]

def preprocess_documents(doc_folder):
    all_chunks = []
    for filename in os.listdir(doc_folder):
        if filename.endswith(".txt"):
            # Process each document
            path = os.path.join(doc_folder, filename)
            with open(path, 'r') as f:
                raw = f.read()
                cleaned = clean_text(raw)
                chunks = chunk_text(cleaned)
                all_chunks.extend(chunks)
    return all_chunks

Vector Database Setup

Using ChromaDB for efficient similarity search:

import chromadb
from chromadb.config import Settings
from sentence_transformers import SentenceTransformer

# Initialize ChromaDB (disable telemetry for privacy)
client = chromadb.Client(Settings(anonymized_telemetry=False))
collection = client.get_or_create_collection(name="my_knowledge")

# Initialize embedding model
model = SentenceTransformer("all-MiniLM-L6-v2")

# Store documents with embeddings
for i, doc in enumerate(documents):
    embedding = model.encode(doc).tolist()
    collection.add(
        documents=[doc], 
        ids=[str(i)], 
        embeddings=[embedding]
    )

Query Processing & LLM Integration

The main RAG pipeline handles user queries:

import subprocess

# Process user query
query = input("Ask your question: ")
query_embed = model.encode(query).tolist()

# Retrieve relevant context
results = collection.query(
    query_embeddings=[query_embed], 
    n_results=2
)
contexts = results['documents'][0]
context = "\n".join(contexts)

# Build prompt for Ollama
prompt = f"""Answer the following question using only the information in the context. Be concise and factual.

Context:
{context}

Question:
{query}

Answer:"""

# Generate response using Ollama
result = subprocess.run(
    ["ollama", "run", "llama3.1"],
    input=prompt,
    capture_output=True,
    text=True
)

print("\nLLM Response:")
print(result.stdout.strip())

Knowledge Base Content

My system includes curated information about:

Personal Background

  • Name: Saad Ahmad
  • Location: Khyber Pakhtunkhwa, Pakistan
  • Focus: AI agents and intelligent systems
  • Skills: Python, ChromaDB, sentence-transformers, Ollama

AI in Education

  • Personalized learning platforms
  • Adaptive learning systems
  • AI-powered tutoring
  • Administrative automation
  • Predictive analytics

AI in Finance

  • Algorithmic trading
  • Fraud detection
  • Credit scoring and risk assessment
  • Robo-advisors
  • RegTech & compliance

AI in Healthcare

  • Medical imaging and diagnostics
  • Predictive analytics
  • Drug discovery
  • Virtual health assistants
  • Personalized treatment

Testing the System

Example Queries

Personal Information:

Query: "What is Saad Ahmad's location and background?"
Response: "Saad Ahmad's location is DIKhan, Khyber Pakhtunkhwa, Pakistan. 
His background includes being a passionate learner focused on building 
AI agents and intelligent systems..."

Domain Knowledge:

Query: "How is AI being used in healthcare?"
Response: "AI is revolutionizing healthcare through medical imaging 
for diagnostics, predictive analytics for disease onset, drug discovery 
acceleration, and personalized treatment plans..."

Performance Insights

What Works Well

  • Fast Response Times: Local processing eliminates network latency
  • Accurate Retrieval: Semantic search finds relevant context effectively
  • Privacy Maintained: No data ever leaves the local system
  • Cost Effective: Zero ongoing operational costs

Areas for Improvement

  • Chunk Size Optimization: Experiment with different chunk sizes
  • Advanced Embeddings: Try domain-specific embedding models
  • Query Expansion: Implement query reformulation techniques
  • Caching: Add response caching for repeated queries

Future Enhancements

Planned Upgrades

  1. LangChain Integration: More sophisticated prompt management
  2. Multi-Modal Support: Add image and document processing
  3. Real-time Updates: Dynamic knowledge base updates
  4. Advanced Chunking: Semantic chunking strategies
  5. Evaluation Metrics: Implement RAG evaluation framework

Integration Possibilities

  • Slack Bot: Deploy as a company knowledge assistant
  • Web Interface: Build a user-friendly web dashboard
  • API Service: Create REST API for external applications
  • Voice Interface: Add speech-to-text capabilities

Technical Specifications

System Requirements

  • RAM: 8GB+ (for model loading)
  • Storage: 5GB+ (for models and embeddings)
  • CPU: Multi-core recommended
  • GPU: Optional (speeds up embedding generation)

Model Details

  • LLM: Llama3.1 (4.9GB)
  • Embeddings: all-MiniLM-L6-v2 (80MB)
  • Vector DB: ChromaDB (lightweight, embedded)

Conclusion

This local RAG system demonstrates that powerful AI applications don’t require cloud dependencies or expensive APIs. With open-source tools like Ollama, ChromaDB, and sentence-transformers, you can build sophisticated knowledge systems that respect privacy while delivering excellent performance.

The combination of retrieval-augmented generation with local processing opens up new possibilities for personalized AI assistants, company knowledge bases, and privacy-focused applications.

Source Code

The complete implementation is available on GitHub, including all preprocessing scripts, the main RAG pipeline, and sample documents for testing.

Happy building!