RAG Architecture Diagram: How a Retrieval-Augmented Generation App Fits Together

Most RAG architecture diagrams have three boxes: a user, a vector database and an LLM. That picture explains the idea in a meeting. It does not help anyone run the system, because the decisions that decide answer quality and safety are not on it: how documents are chunked, which embedding model is used, who may see which chunk, how results are ranked, and how anyone knows the answers are good.

This guide draws the full picture of a retrieval-augmented generation app as two paths around shared models and stores, then covers what to label and the mistakes that show up most often.

Two paths, not one

The term comes from Lewis et al. (2020), who combined a language model with a dense vector index of Wikipedia that a retriever searched at generation time. Most production RAG apps keep the idea without training the parts together: retrieve relevant passages, put them in the prompt, and ask the model to answer from them.

That gives every RAG app two paths:

• Ingestion runs continuously in the background. It turns documents into chunks, vectors and metadata.
• Query runs once per question. It turns a question into an answer with sources.

They meet at the shared parts: the embedding model, which must be the same version on both paths, the vector index and the metadata store. A diagram that shows only the query path hides half the system and most of the bugs.

The ingestion path

1. Sources and change detection. Connectors, webhooks or a crawler notice new, changed and deleted documents and put jobs on a queue. Deletions and permission changes are jobs too.
2. Parsing. Extract text while keeping structure: headings, lists, tables, and metadata such as title, URL, owner, last update and access control list.
3. Chunking. Split along the document's structure first, then by size. Chunk size and overlap are parameters you tune against your own questions, not universal numbers. Keep the heading path in each chunk so a paragraph still makes sense on its own.
4. Embedding. Call one embedding model and pin its version. Store the model name and version with every vector. Vectors from different models are not comparable, so changing the model means re-embedding the whole corpus.
5. Indexing. Upsert each chunk with its vector, document ID, chunk number, version and access tags. Keep a keyword index next to the vector index if you plan hybrid search.
6. Refresh and deletion. Make upserts idempotent, delete the chunks of removed documents, and track freshness: how long a change in the source takes to reach the index.

The query path

1. Identify the user. Resolve the user's groups before retrieval starts.
2. Rewrite the question (optional). A follow-up such as "and for contractors?" needs the conversation to become a standalone query.
3. Retrieve. Vector search finds passages with similar meaning; keyword search finds exact terms such as error codes and product names. Hybrid search runs both and merges the lists, often with reciprocal rank fusion, which scores each document by 1/(k + rank) summed over the lists. The original RRF paper fixed k at 60.
4. Filter by permission inside the query. Pass the user's groups as a filter on the access tags, so chunks the user may not see never reach the prompt. Check how your store applies filters. pgvector's documentation notes that with approximate indexes the filter runs after the index scan: with HNSW's default ef_search of 40, a filter that matches 10% of rows returns about 4 rows on average. Iterative scans, partial indexes or partitioning fix that.
5. Rerank. Retrieve generously, for example 50 candidates, and let a reranking model score them against the question. Keep the best few, for example 8.
6. Assemble the prompt. Instructions, the selected chunks with IDs and titles, and the question, inside a token budget. Ask for citations by chunk ID.
7. Generate and cite. Call the LLM, stream the answer and turn chunk IDs into links the user can check.

Around both paths

• Treat retrieved text as untrusted input. A document can contain instructions aimed at the model. OWASP lists this as indirect prompt injection. Keep tools and data the model can reach as narrow as the user's own access.
• Trace every request. Log the rewritten query, the retrieved and reranked chunk IDs with scores, the prompt version, the model versions and the latency of each step.
• Evaluate on a fixed question set. Run it on every change to chunking, models, retrieval or the prompt. The RAGAS paper names three useful dimensions: faithfulness to the retrieved context, answer relevance and context relevance. Also check whether the expected source appears in the top results.
• Cache with care. Caching query embeddings is usually safe. Caching answers is only safe per permission set.

The diagram

How to read it:

• Left, ingestion. Source changes become jobs on a queue. The ingestion worker parses and chunks each document, calls the embedding model, upserts chunks and vectors with access tags into the vector index, and writes document metadata to Postgres.
• Center, shared models and stores. The same pinned embedding model serves both paths. The LLM sits here too, called only by the query path.
• Right, query. The RAG API looks up the user's groups, embeds the question, runs hybrid search with the permission filter, sends 50 candidates to the reranker, and calls the LLM with the best 8 chunks.
• Bottom left. An example latency budget per step, so a slow answer can be traced to the step that broke its budget.

What to label

• Model versions on every model node, including the embedding model.
• Chunking parameters on the ingestion worker or in a note.
• Where permissions are enforced, on the retrieval edge, not in a footnote.
• How many results flow between steps: 50 retrieved, 8 in the prompt.
• Freshness and latency targets: how fast a source change is searchable, and the budget per query step.

Common mistakes

• One box called "AI". Retrieval, reranking and generation fail in different ways and need separate boxes.
• No ingestion path. Stale or missing answers often start there.
• No permissions. If the index holds documents with different audiences, a diagram without access tags describes a data leak.
• No evaluation. Without a question set, every change to the prompt or chunk size is a guess.
• Mixed embedding versions. Re-embedding half the corpus with a new model quietly breaks search.

When RAG is not the answer

• The answer lives in structured data. Query the database or an API instead of embedding table rows.
• The task needs a whole document. Summarizing one contract works better with the full contract in the prompt, if it fits.
• The corpus is tiny. A few pages can go straight into the prompt.
• You need different behavior, not more knowledge. That is a prompt or fine-tuning problem.

For event-driven ingestion with queues, see the event-driven architecture diagram guide.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add two Core Frame nodes, "Ingestion path" on the left and "Query path" on the right.
3. In the ingestion frame, add a Data pack Object Store for source documents, a Data pack Queue for ingestion jobs and a C4 Model Container for the ingestion worker.
4. In the center, add an ML pack Model for the embedding model, an ML pack Vector index, a Data pack Postgres for metadata and permissions, and a second Model for the LLM.
5. In the query frame, add a C4 Model Person for the user, a Container for the RAG API and an ML pack Model for the reranker.
6. Label each edge with what flows, including "top 50" and "top 8", and put the latency budget in a Core Markdown block.

The example uses 13 diagram nodes, within the Free plan. If you work with an AI agent, it can draw a first version for you over MCP from a description of both paths; see how to create architecture diagrams with AI agents. For the general method, see how to draw a software architecture diagram.

FAQ

Do I need a dedicated vector database for RAG? No. A vector extension in a database you already run, such as pgvector in PostgreSQL, or a search engine with vector support works for many apps. Choose by corpus size, filtering needs and what your team operates well.

What chunk size should I use? There is no universal number. Split along headings and paragraphs, then compare a few sizes and overlaps on your own question set.

Where should permission checks happen? At ingestion, by copying each document's access list into its chunks, and at retrieval, by filtering on it. Never rely on the prompt to hide content.

How do I evaluate a RAG app? Keep a fixed set of real questions with expected sources. Measure whether retrieval finds those sources, and whether answers are faithful to the retrieved context and relevant to the question.

Should the LLM call tools instead of using RAG? For live or structured data, yes: a tool that queries the system of record beats an index of copies. Many apps use both.

Start free — one seat, no credit card.