RAG (Retrieval-Augmented Generation)

Glossary

• Embedding(s):

  Turning words into vectors

RAG Architecture: Overview

• Retrieval: Fetch relevant info from an external knowledge base (using semantic search in a vector database).
• Generation: Use an LLM to generate a natural language answer using both the user query and the retrieved info.

RAG Workflow: Step-by-Step

Simple

1. User Query: User sends a question or prompt.
2. Query Embedding: An embedding model converts the query into a vector.
3. Vector Search: The vector DB compares the query vector to stored document vectors, retrieving the most semantically similar documents.
4. Context Assembly: Retrieved documents are compiled into a prompt along with the original question.
5. LLM Inference: The inference model (LLM) takes the prompt (query + context) and generates a coherent, context-aware response.
6. Return Response: The answer is returned to the user.

Optional

• KV Caching: Frequently used prompts and their generated responses are cached for speed, reducing repetitive computation.
• Agentic Workflow: If multi-step actions are needed (planning, API calls), an agentic layer may manage additional steps, each repeating some or all of the above process.

Agent/Orchestrator

1. User/API sends a query.
2. The API Gateway / Ingress receives the query and routes it to the Agent/Orchestrator.
3. The Agent/Orchestrator:
• Embeds the query using an embedding model service (can be a pod or external service).
• Searches the Vector DB with the query embedding to retrieve relevant documents.
• Assembles the prompt: combines user query + retrieved docs.
• Optionally checks the KV Cache (e.g., Redis) for repeated queries/answers.
4. The LLM Serving Engine (e.g., vLLM, TGI, ONNX, TensorRT-LLM) receives the prompt and generates the answer.
5. The response is returned to the user via the API gateway.
6. If caching is enabled, frequent queries/responses are stored in the KV Cache for speed.

Where Each Technology Sits:

• Embedding Model: Pod/service (HuggingFace, OpenAI endpoint, or similar)
• Vector DB: Pod/service (Weaviate, Qdrant, Milvus, Pinecone, FAISS, etc.)
• LLM Serving Engine: Pod/service (vLLM, TGI, etc.) or cloud LLM endpoint
• KV Cache: Pod/service (Redis, Memcached)
• Agent/Orchestrator: Pod/service (your business logic, maybe a FastAPI app, LangChain agent, etc.)
• API Gateway / Ingress: K8s ingress controller, NGINX, Istio, etc.

Embedding Model vs. Inference Model

• They are separate models but work together: embedding model finds what to show, LLM decides how to use it to answer your question.

Popular Vector Databases

• Cloud-managed options (Pinecone, Weaviate, Qdrant Cloud) are easiest to get started with for production, but self-hosting (Weaviate, Qdrant, Milvus) is common for privacy/compliance.
• FAISS is often used for prototyping or when you want to tightly control everything in-memory.

Common Use Cases That Connect All the Above

• AI Chatbots with Company Knowledge: Users ask questions; RAG retrieves and summarizes internal docs, policies, FAQs.
• Semantic Search Applications: Employees or users search a knowledge base using meaning, not just keywords.
• Document Q&A: RAG finds and reads long legal, scientific, or technical documents to answer questions.
• Code/Support Copilots: RAG retrieves relevant support tickets, code snippets, or documentation, and LLM explains, summarizes, or troubleshoots.
• Agentic Workflow Example: “Book a meeting with Alice next week and summarize her last three emails”—an agent retrieves data, generates summaries, and takes action step by step.

How This Is Served on Kubernetes (Managed or Self-Hosted)

Typical Architecture:

• Pods/Deployments:
• Embedding Model Service: Converts text to vectors (can be REST API, open-source model, or cloud embedding endpoint).
• Vector DB Service: (Pinecone/Weaviate/Qdrant/Milvus pod, or external managed service).
• LLM Inference Service: (TGI, vLLM, ONNX Runtime, or cloud LLM endpoint).
• Agent/Orchestrator: (If using agentic workflows, runs logic, manages prompts).
• API Gateway/Ingress: Receives user queries and routes requests.
• Managed (e.g., GKE, Vertex AI):
• Kubernetes provides scaling, monitoring, and deployment.
• Vertex AI can host managed LLMs and vector search endpoints as a service, which can be integrated with other K8s services.
• Secrets, autoscaling, logging, and monitoring handled by the cloud provider.
• Self-Hosted:
• All components (vector DB, embedding service, LLM server, orchestrator) run as containers/pods.
• Use K8s tools for config, scaling, rolling updates, and networking.
• KV Caching:
• Add a cache layer (e.g., Redis, Memcached) as a Kubernetes service.
• Used by agent/orchestrator and LLM inference pods to store and retrieve frequent prompt/response pairs or token sequences.

TL;DR Summary Table