RAG for customer support: how to make AI replies actually accurate

Without retrieval, an LLM answering "how do I reset my password?" will give a plausible answer that's probably wrong — different button, different flow, different URL. RAG (retrieval-augmented generation) fixes this by injecting the right chunk of your docs into the prompt before the model speaks.

The pipeline at a glance

Three stages: ingest, retrieve, generate.

1. Ingest

• Upload PDFs, Markdown, Office files via the Knowledge UI
• Worker extracts text, chunks at ~800 tokens with 100 token overlap
• Each chunk is embedded with text-embedding-3-small (1536d)
• Stored in Postgres with the pgvector extension

2. Retrieve

When a visitor sends a message, we embed the query and run a cosine-distance search against the tenant's chunks. Top-K (default 5) gets pulled in, with a similarity threshold to drop irrelevant matches.

SELECT id, content, 1 - (embedding <=> $1) AS similarity
FROM knowledge_chunks
WHERE tenant_id = $2
ORDER BY embedding <=> $1
LIMIT 5;

3. Generate

Retrieved chunks are prepended to the system prompt with explicit instructions: cite sources, refuse if context doesn't cover the question, hand off to a human if confidence is low.

Cutting the token bill

Three tricks save real money on large knowledge bases:

• Embedding cache — same query text → same embedding. SHA1 hash key, 24h TTL in Redis. Cuts re-embedding cost to zero on repeat questions.
• Prompt caching — Anthropic's cache_controland OpenAI's automatic caching mean the static system prompt only counts once per 5-minute window.
• Threshold-based skipping— if no chunk passes the similarity threshold, don't inject anything. Let the model say "I don't know."

When RAG isn't enough

RAG handles "how do I X?" questions well. It fails on personalized data ("where's my order?") — that needs function calling, which we cover in a separate post. The combination is what gives you Intercom-level AI for a fraction of the cost.