RAG pipeline with hybrid chunking & multi-query retrieval

An embeddings pipeline that turns each client's documentation into a grounded knowledge base for an LLM chatbot.

Role: Software Engineer — pipeline design & build
Period: 2024 — Present
Company: Digital marketing agency (confidential)

Hybrid

chunking: semantic segmentation + overlap

N×

query variants per question for better recall

Per-client

isolated knowledge bases in Qdrant

#Context

The agency offers each client a chatbot that can answer questions about that client's brand, products and technical documentation. Off-the-shelf chatbot widgets hallucinated or gave generic answers — the value was in grounding responses in each client's own material, and each client's knowledge had to stay isolated from the others.

#Problem & constraints

Client documents are heterogeneous: PDFs, brand books, technical manuals, web pages — naive fixed-size chunking destroys semantic structure.
Users phrase questions differently than documents phrase answers — single-query retrieval misses relevant chunks (low recall).
Multi-tenant by design: client A's embeddings must never surface in client B's chatbot.

#Architecture

Offline ingestion (top) and query-time retrieval (bottom) per client collection.

Documents flow through an ingestion pipeline: Unstructured parses each file into semantically coherent segments (titles, paragraphs, tables), which are re-chunked with overlap so no idea is split across a boundary. Each chunk is embedded with OpenAI embeddings and upserted into a per-client Qdrant collection. At query time, the user's question is first rewritten by an LLM into several semantic variants (multi-query), each variant retrieves its own top-k chunks, and the union — deduplicated and ranked — hydrates the agent's context window before it composes an answer grounded in the client's documentation.

chat.client-brand.com

What's the return window for outlet items?

Outlet items can be returned within 15 days with the original receipt, per the returns policy.

sources:brand-guide.pdf · p.12returns-policy.pdf · p.3product-manual.pdf · p.41

* Interface recreated with synthetic data — the production system holds client-confidential data.

#Key decisions & tradeoffs

01Hybrid chunking over fixed-size splitting

Decision: Segment documents semantically with Unstructured first, then apply overlap re-chunking on top.
Alternatives: Fixed-size character/token windows (the default in most RAG tutorials).
Why: Semantic boundaries keep ideas whole — a paragraph about return policy stays together — while overlap guards against context loss at segment edges. Retrieval quality improved visibly on real client docs.
Cost of being wrong: Ingestion is slower and costs more (Unstructured parsing) than dumb splitting — acceptable because ingestion is offline and infrequent.

02Multi-query retrieval for recall

Decision: Rewrite each user question into several semantic variants with an LLM, retrieve for each, and merge results before hydrating context.
Alternatives: Single-query vector search, or hybrid BM25 + vector search.
Why: The vocabulary gap between 'how a user asks' and 'how the document says it' was the main source of bad answers. Query variants attack exactly that failure mode with one extra LLM call — cheap compared to a wrong answer.
Cost of being wrong: Latency per question increases (one rewrite call + N retrieval calls) and more chunks in context means more prompt tokens.

03Per-client Qdrant collections

Decision: Isolate each client's knowledge base in its own vector collection rather than one shared collection with metadata filters.
Alternatives: Single collection with a client_id payload filter on every query.
Why: Physical isolation makes cross-client leakage structurally impossible, not just filter-dependent — a bug in filter logic can never leak one client's docs into another's chatbot.
Cost of being wrong: More collections to manage and migrate; per-collection operations (re-indexing, backups) multiply as clients grow.

#Results

Grounded

answers sourced from each client's own documentation

Higher recall

from multi-query retrieval vs single-query baseline

Zero

cross-client leakage by construction

#What I'd do differently

I'd build the evaluation harness first — we tuned chunking and retrieval by eye before we had a golden Q&A set per client, and a small eval suite would have turned 'seems better' into measured recall from day one. I'd also version the embedding pipeline config per client; re-ingesting with a new chunking strategy currently means a full collection rebuild.

#Stack

OpenAIQdrantUnstructuredNode.jsTypeScriptLangChainPostgreSQLRAG