If you are an architect who is new to AI, the vocabulary can feel like a wall. Every vendor deck, design review and job description now assumes you know what a context window is, why RAG matters, and what people mean when they say “agent”. This guide is a map. It explains the twelve concepts you will meet first, in plain English, and for each one it tells you why it matters to an architect. That is the part most explanations skip.
You do not need a data science background to follow it. You need the same instincts you already use for any system: where does data come from, what can fail, what does it cost, and who is allowed to do what.
1. Large language model (LLM)
What it is: A model trained on very large amounts of text to predict the next piece of text. That simple objective, at enormous scale, produces a system that can summarize, translate, write code, answer questions and follow instructions.
Why architects care: Treat the LLM as a powerful but probabilistic component, not a database and not a rules engine. It is usually consumed as an API (hosted by a provider) or run on your own infrastructure. That choice drives your cost model, latency, data residency and vendor risk.
2. Tokens
What it is: Models do not read words; they read tokens, which are chunks of text (often part of a word). Both your input and the model’s output are measured in tokens.
Why architects care: Tokens are the unit of cost and of capacity. Pricing, rate limits and response time all scale with tokens. A design that sends a 50-page document with every request costs very differently from one that sends a two-paragraph summary.
3. Context window
What it is: The maximum number of tokens a model can consider in one request: instructions, conversation history, retrieved documents and the answer it is writing.
Why architects care: The context window is the model’s working memory, and it is not persistent. Anything the model should “know” for this request must be inside it. Much of AI architecture is deciding what goes into the context window, in what order, and what gets left out.
4. Prompts and system prompts
What it is: The prompt is the input you send. The system prompt is the standing instruction that sets the model’s role, rules and output format, usually written by the application, not the user.
Why architects care: Prompts are configuration, and they deserve the same discipline as code: version control, review, testing and rollback. Never put secrets in a system prompt; assume it can be extracted.
5. Non-determinism
What it is: The same input can produce different outputs. Settings such as temperature make responses more or less varied, but you should not design as if output is ever guaranteed to be identical.
Why architects care: This breaks assumptions from traditional integration, where the same request gives the same response. Plan for validation of outputs, structured output formats (for example, JSON that you check against a schema), retries, and tests that measure quality rather than exact matches.
6. Hallucination
What it is: A confident, fluent answer that is wrong or invented. It happens because the model generates plausible text; it does not look facts up unless you give it a way to.
Why architects care: You reduce hallucination mainly through design, not by asking the model nicely. Ground answers in your own data (see RAG), ask for citations, constrain the task, and add human review where mistakes are expensive.
7. Embeddings and vector search
What it is: An embedding turns a piece of text into a list of numbers that captures its meaning. Texts with similar meaning end up close together. A vector database stores these embeddings so you can search by meaning (“find policies about travel refunds”) instead of by exact keywords.
Why architects care: Vector search is a new kind of index in your data landscape. It needs a pipeline to keep it fresh, it must respect the same access rules as the source data, and it often works best combined with classic keyword search.
8. Retrieval-augmented generation (RAG)
What it is: Before the model answers, the application retrieves relevant documents (often with vector search) and places them in the context window. The model then answers using that material.
Why architects care: RAG is the most common enterprise pattern because it keeps knowledge in systems you control, stays current without retraining, and can cite sources. The hard parts are classic architecture problems: data quality, chunking, freshness, and making sure people only retrieve what they are allowed to see. We compare RAG with the alternatives in RAG, fine-tuning or long context?
9. Fine-tuning
What it is: Further training a model on your own examples so it adopts a style, format or specialized behavior.
Why architects care: Fine-tuning changes how a model behaves; it is a poor way to teach it facts that change. It adds a model lifecycle to manage (training data, versions, evaluation, retraining), so reach for it only when prompting and RAG are not enough.
10. Tool calling
What it is: The model is given descriptions of functions (tools), such as “look up order status” or “create a ticket”. Instead of only writing text, it can ask your application to call one of them with specific arguments. Your code runs the tool and returns the result.
Why architects care: This is where AI meets your APIs. Every tool is an integration point with real side effects, so it needs authentication, authorization, input validation, rate limits and logging, exactly like any other API consumer. Except this consumer can be manipulated by the text it reads.
11. Agents
What it is: An agent is a loop: the model plans a step, calls a tool, looks at the result, and decides what to do next, until the goal is met. Multi-agent systems split work across several specialized agents.
Why architects care: Agents move AI from answering to acting. That raises the stakes on permissions, approvals, cost limits and audit trails. Start with narrow agents that have few tools and clear stopping conditions. Our guide Guardrails for AI agents covers the patterns.
12. MCP, evals and guardrails
Model Context Protocol (MCP) is an open standard for connecting AI applications to tools and data sources through a common interface, so you build an integration once and many AI clients can use it. It is quickly becoming the “API layer” for agents; see what the latest MCP specification changed.
Evals are automated tests for AI quality: a set of representative inputs with criteria for a good answer, run every time you change a prompt, model or retrieval setting.
Guardrails are the controls around the model: input and output checks, permission limits, approval steps and budgets.
Why architects care: Together these are what make AI production-grade. MCP standardizes integration, evals tell you whether a change made things better or worse, and guardrails keep a probabilistic component inside safe boundaries.
How the pieces fit together
A typical enterprise AI request looks like this:
- A user asks a question in an application.
- The application checks who the user is and what they may access.
- It retrieves relevant, permitted documents (RAG) using vector and keyword search.
- It builds the prompt: system instructions, retrieved context and the question, within the context window.
- The LLM answers, or asks to call a tool (possibly through MCP) to fetch live data or take an action.
- Guardrails validate the output and any action; risky actions wait for human approval.
- Everything is logged, and evals track quality over time.
Notice how much of that list is not “AI” at all. Identity, data pipelines, integration, policy and observability are architecture. That is why architects are so well placed to lead this shift.