Back to Basics: LLM, RAG, and GraphRAG

Every few months the AI tooling conversation resets. New wrappers, new orchestration layers, new acronyms. But the underlying concepts that matter most — LLM, RAG, GraphRAG — have been stable for a while now, and it's worth stepping back to see them clearly before layering anything on top.

LLM: The Foundation

A large language model is a neural network trained to predict the next token in a sequence, at massive scale. That single objective, applied to enough text, produces a model that can reason, summarize, translate, and generate — not because those capabilities were explicitly programmed, but because they emerge as useful strategies for prediction.

The practical implication is that an LLM is a general-purpose text transformer. It holds a compressed representation of its training data in its weights. The limit is that those weights are frozen after training: the model knows nothing about events after its cutoff, has no access to private data, and cannot update its beliefs from conversation to conversation (beyond the current context window).

That boundary is important. Many production failures come from treating the model as a database — expecting it to recall a specific figure, document, or fact precisely. The model doesn't store facts; it stores patterns. When you need facts, you need retrieval.

RAG: Retrieval-Augmented Generation

Retrieval-augmented generation addresses the knowledge boundary directly. Instead of asking the model to recall something from its weights, you retrieve the relevant content from an external store and inject it into the prompt as context. The model then reads and reasons over what you've given it, rather than guessing from memory.

The standard pipeline: encode your documents as vector embeddings, store them in a vector database, embed the user's query the same way, find the nearest neighbors, and prepend those chunks to the prompt. The model sees both the question and the relevant excerpts and produces a grounded answer.

RAG works well when the knowledge base is a flat collection of documents — support articles, product manuals, research papers — and when queries are primarily semantic lookups. It fails when the relationships between documents matter more than the content of any single document. You can retrieve the right chunks and still miss the right answer if the answer lives in the connection between them.

GraphRAG: Retrieval Over a Knowledge Graph

GraphRAG, popularized by Microsoft Research in 2024, extends RAG by building a knowledge graph over the document corpus before retrieval. Entities and relationships are extracted from the source text and stored as nodes and edges. At query time, retrieval traverses the graph rather than just ranking vectors by cosine similarity.

The difference becomes meaningful for queries like "what projects did this person influence and who collaborated with them on each?" A vector search will find documents mentioning the person; it won't trace the influence chain. A graph traversal can.

The tradeoff is cost and complexity. Building and maintaining a knowledge graph is expensive — entity extraction requires its own LLM calls, the graph schema must be designed carefully, and updates are harder than appending new vectors. For most question-answering use cases over a static document set, standard RAG with good chunking and reranking is sufficient. GraphRAG earns its overhead in domains with dense, structured relationships: biomedical research, legal case networks, enterprise knowledge bases where org charts and project histories matter.

Choosing the Right Layer

The decision tree is simpler than the ecosystem makes it look. Start with the raw LLM for tasks that are generative — drafting, summarizing, classifying content you provide inline. Add RAG when the model needs to read documents it wasn't trained on, or when you need grounded citations. Add GraphRAG only when the meaningful structure lives in relationships between entities across many documents, and when the retrieval quality of flat RAG is provably insufficient for your use case.

Most products need the first two layers, maybe with a reranker in the RAG pipeline. Very few need GraphRAG. The temptation to reach for the most sophisticated tool is understandable — the papers are compelling — but the right architecture is the simplest one that produces correct answers. Work up from the baseline rather than down from the frontier.

The fundamentals stay stable even as the surface layer churns. Understanding what each piece actually does — and what problem it was designed to solve — is the only reliable way to make sound architectural decisions as the tooling continues to evolve.