MCP-Native RAG Wiki Generator with Claude Agent Integration

Document Ingestion Pipeline

The system employs a modular ingestion layer supporting PDF, Markdown, and plaintext parsing through unified ecosystem processors. Documents undergo semantic chunking with configurable overlap strategies before embedding via OpenAI or local text-embedding-3-large models.

MCP Orchestration Layer

Core innovation lies in the bidirectional MCP (Model Context Protocol) implementation. The server exposes three primary tool schemas to Claude Desktop:

• search_documents(query: string, top_k: int): Vector similarity search via Supabase RPC
• create_wiki_entry(title: string, content: string, parent_id?: uuid): Atomic wiki node creation
• update_knowledge_graph(source_id: uuid, target_id: uuid, relation: string): Maintains graph coherence

Storage Backend

Core Abstractions

The architecture distinguishes between Ephemeral Context (MCP tool call results) and Persistent Knowledge (committed wiki nodes). This separation allows Claude to perform multi-hop reasoning across document chunks while maintaining atomic, version-controlled wiki entries. Tradeoff: MCP round-trip latency (400-800ms) vs. direct API integration, accepted for ecosystem compatibility.

The system treats wiki generation as an agentic, iterative refinement process rather than single-shot summarization, utilizing MCP's bidirectional context flow to enable Claude to 'ask questions' of the document corpus during synthesis.

Key Technical Innovations

1. Stateful Knowledge Synthesis Loop: Unlike standard RAG Q&A systems, this implements a feedback loop where Claude can perform multiple retrieval passes during wiki generation, referencing newly created pages to ensure terminological consistency across the knowledge base.
2. Semantic Link Prediction: Uses vector similarity between wiki node embeddings to auto-suggest [[WikiLinks]], creating a browsable graph structure that mimics Roam Research/Obsidian bi-directional linking without manual curation.
3. MCP Server as Persistence Layer: Inverts typical architecture by making the vector database accessible via MCP tools rather than direct API calls, enabling non-technical users to leverage the system through Claude Desktop without code deployment.
4. Incremental Compilation Strategy: Implements Karpathy's "living document" philosophy through delta-updates; when source documents change, only affected wiki nodes are regenerated using dependency tracking via the source_document_chunks join table.
5. Multi-Modal Context Windows: Extends beyond text to support image extraction via vision tool descriptions, allowing diagrams within uploaded PDFs to be described and integrated into wiki entries.

Implementation Detail

// MCP Tool Schema for Wiki Generation
{
  name: "synthesize_wiki_node",
  description: "Creates atomic knowledge unit from retrieved context",
  inputSchema: {
    type: "object",
    properties: {
      topic: { type: "string" },
      context_chunks: { 
        type: "array", 
        items: { type: "string", description: "uuid of embedding" }
      },
      link_to_existing: { 
        type: "array", 
        items: { type: "string", description: "wiki page uuid" }
      }
    }
  }
}

Latency Characteristics

The MCP architecture introduces inherent latency overhead due to JSON-RPC serialization and Claude Desktop process communication. Benchmarks indicate 400-1200ms per tool call, making this unsuitable for real-time streaming but acceptable for asynchronous wiki compilation.

Scalability Constraints

Current implementation couples vector search directly to Supabase Row Level Security (RLS) policies, enabling multi-tenant isolation but capping throughput at ~100 QPS before connection pool exhaustion. Memory footprint scales with O(n×m) where n = document chunks and m = wiki nodes, though HNSW indexing maintains sub-linear search complexity.

Bottlenecks

• MCP Serialization: Large context chunks (>8k tokens) transferred via JSON-RPC incur 20-30% token overhead compared to native API streaming.
• Graph Consistency: Maintaining bi-directional wiki links requires atomic transactions across multiple tables, creating lock contention during bulk imports.

Competitive Landscape

Production Usage Patterns

While early-stage (104 stars), the repository demonstrates traction among:

1. Research Labs: Automating literature review synthesis from arXiv PDF corpora
2. Technical Writers: Generating living documentation from scattered markdown repositories
3. Consulting Firms: Creating client-specific knowledge bases from uploaded SOWs and whitepapers
4. Open Source Maintainers: Auto-generating wiki architectures from chaotic GitHub discussions and READMEs

Integration Points

• Claude Desktop: Primary interface via claude_desktop_config.json MCP server registration
• Supabase Auth: Row-level security policies enable multi-user deployments with document isolation
• Git Sync: Optional export to Git repositories via git-wiki exporter module for static site generation
• Obsidian Bridge: Community-contributed script for exporting generated wiki to Obsidian vault format

Migration Path

Organizations currently using static wiki generators (MkDocs, Docusaurus) can adopt incrementally by ingesting existing markdown into Supabase and using llmwiki's --suggest-structure flag to identify content gaps and recommend new inter-page links without disrupting existing URLs.