AI Application Architecture, Layer by Layer: What Each Box Is For

A diagram titled "AI Application Architecture", shared by the account HackProduct, has been making the rounds. It fits a whole production AI app onto one screen. A user request at the top fans out into a frontend and an agent orchestrator. Those feed three columns, Knowledge (RAG), Models (LLM) and Tools (MCP), which all flow down into a data layer, then observability and evaluations, then deployment. Each box carries a few familiar logos.

Diagrams like this are useful because they name the parts. They can also mislead, because a picture with eight layers and thirty logos looks like a shopping list. This post walks the stack one layer at a time: what each layer does, what the tools in the diagram are, how a single request moves through it, and what a small team can leave out on day one. The tools named below are the diagram's examples, not endorsements, and each layer has good alternatives.

The stack at a glance

1. Frontend: where the request starts

The frontend is more than a text box. AI responses are slow by web standards, so a good interface streams tokens as they arrive, shows progress while tools run, lets the user stop a run, and shows where an answer came from. React and Next.js are the usual choice for a product. Streamlit, a Python framework for data apps, is the quick option for internal tools and prototypes, because the person who wrote the model code can put an interface on it without learning a web stack.

2. Agent orchestration: the control loop

The orchestrator is the part that decides what happens next. For a simple chatbot that decision is trivial: send the message to a model and return the answer. For an agent it is a loop: look at the goal, choose a step, run it, look at the result, choose again.

Anthropic's guide to building agents draws a line that helps here. Workflows follow predefined code paths, while agents let the model dynamically direct its own process and tool use. Its advice is to find the simplest solution possible and only increase complexity when needed (Anthropic, 2024a). Many products that call themselves agents are, and should be, workflows.

The three frameworks in the diagram take different angles:

• LangGraph describes itself as a low-level orchestration framework for long-running, stateful agents. Its selling points are durable execution that resumes after a failure, human-in-the-loop pauses for review, and short- and long-term memory (LangChain, n.d.).

• CrewAI is a Python framework built around role-based agents working together as a "crew", plus event-driven "flows" for tighter control (CrewAI, n.d.).

• Mastra is a TypeScript framework for AI applications and agents, for teams whose stack is already JavaScript (Mastra, n.d.).

3. Knowledge (RAG): giving the model the right facts

A model knows what was in its training data and nothing about your company's documents. Retrieval-augmented generation, or RAG, fixes that by looking up relevant text first and putting it into the prompt. The idea was formalised in a 2020 paper that paired a generator with a retriever over Wikipedia (Lewis et al., 2020).

The diagram's Knowledge column has four parts:

• Documents are the raw source: wikis, PDFs, tickets, code.

• A vector database stores embeddings, numeric fingerprints of text chunks, so that passages with similar meaning can be found even when they share no keywords.

• Memory keeps what the system has learned about this user or this conversation.

• Retrieval runs the search, often mixing vector similarity with keyword search and a reranking step.

Underneath sits the context builder, the most underrated box in the picture. A model's context window is finite and every token costs money and time. The context builder decides what goes in: the system instructions, the most relevant retrieved passages, a summary of the conversation so far, and the tool definitions. Bad retrieval or a cluttered context produces confident wrong answers, which is why so much of the evaluation layer exists to check this step.

4. Models (LLM): the swappable engine

The Models column lists Claude, GPT, Gemini and Llama. The first three are hosted models reached through an API. Llama is Meta's family of open-weight models, which a team can download and run on its own hardware under Meta's licence.

The architectural lesson is to treat the model as a component behind an interface, not the centre of the design. Models change every few months, prices fall, and different tasks suit different models. Teams that route requests through one internal function can try a new model, send easy requests to a cheaper one, or fall back when a provider has an outage, without rewriting the rest of the app.

5. Tools (MCP): letting the model act

Tools are how a model goes from talking to doing: reading a GitHub issue, posting in Slack, browsing a page, calling an internal API. The diagram labels the column MCP, the Model Context Protocol. Anthropic introduced MCP in November 2024 as an open standard for connecting AI assistants to the systems where data lives (Anthropic, 2024b), and in December 2025 donated it to the Agentic AI Foundation, a fund under the Linux Foundation co-founded with Block and OpenAI (Anthropic, 2025).

The appeal is reuse. An MCP server for GitHub is written once and can be used by any MCP-capable app, instead of every app writing its own GitHub integration. The specification describes three things a server can offer: prompts, which the user chooses; resources, which the application attaches as context; and tools, which the model can call to take actions (Model Context Protocol, 2025).

Tools are also where the risk is. A model that can post messages or merge code can be talked into doing the wrong thing by text hidden in a web page or a document. Sensible defaults are the least access that does the job, read-only tools where possible, and a human confirmation step before anything that sends, deletes or spends.

6. Data layer: where state lives

In the diagram the Knowledge, Models and Tools columns all point down into one data layer. In practice the data layer is several stores, each good at one thing:

• PostgreSQL holds the application's records: users, conversations, documents and permissions. With the pgvector extension it can also store embeddings and run similarity search, which is often enough for a first version.

• Redis is the fast in-memory store for caches, sessions, rate limits and short-lived job state.

• S3-style object storage keeps the files: uploads, source documents, generated outputs.

• Qdrant is a dedicated vector database for when embeddings outgrow what the main database handles comfortably.

7. Observability and evals: knowing whether it works

Traditional software either works or throws an error. AI software can return a fluent, well-formatted answer that is wrong. That makes this layer more important than it looks in the diagram, where it sits near the bottom.

Observability means recording every step of every request as a trace: the prompt that was sent, the passages that were retrieved, each tool call and its result, token counts, latency and cost. Langfuse, an open-source platform that can be self-hosted, combines tracing with prompt management and evaluations (Langfuse, n.d.). Arize's Phoenix offers tracing and evaluation built on OpenTelemetry-based instrumentation (Arize AI, n.d.).

Evals measure quality. RAGAS proposed metrics that score a RAG pipeline without hand-written reference answers, such as whether an answer is faithful to the retrieved context (Es et al., 2024). TruLens frames RAG quality as a triad: context relevance, groundedness and answer relevance (TruLens, n.d.). A practical setup runs a fixed test set before every change and samples real traffic afterwards, so a new prompt or model is judged on evidence, not on a few hand-picked examples.

8. Deployment: running it

The bottom row is Docker, Kubernetes, Vercel and Cloudflare. Containers package the API and background workers; Kubernetes runs many containers at scale; Vercel and Cloudflare host frontends and serverless functions close to users.

One AI-specific wrinkle: agent runs can take minutes, which is longer than many serverless functions are allowed to run. Long jobs usually go to a queue and a background worker, with the frontend polling or subscribing for progress.

One request, end to end

Take a request such as "Summarise this week's open bug reports and post the summary in the engineering channel."

1. The frontend sends the message and opens a stream for progress.

2. The orchestrator plans the steps: fetch the issues, summarise them, post the result.

3. It calls a GitHub tool through an MCP server, which returns the open issues labelled as bugs.

4. Retrieval pulls the team's triage guidelines from the knowledge base, and the context builder packs the issues, the guidelines and the instructions into one prompt.

5. A model writes the summary.

6. Before posting, the orchestrator pauses for the user to approve, then calls the Slack tool.

7. The conversation, the summary and the run's state are saved in the data layer.

8. Every step, with its tokens, cost and latency, is recorded as a trace for observability, and the run can be added to the eval set if the user rates it.

What to leave out at the start

The diagram shows a mature system. A first version can be much smaller:

• One model API, called through one internal function so it can be swapped later.

• PostgreSQL with pgvector for both records and embeddings, until search volume or latency says otherwise.

• Plain code for the control flow. A framework earns its place once there is branching, retries, long runs or human approval steps.

• Direct API calls for the first one or two tools, moving to MCP when the same tools are needed by more than one app.

• Tracing and a small eval set from day one. Twenty or thirty real questions with known good answers will catch more regressions than any amount of architecture.

The layers that are cheapest to add early and most painful to add late are observability and evals. The layers that are easiest to over-build are orchestration and the data stores.

References

Anthropic. (2024a, December 19). Building effective agents. https://www.anthropic.com/engineering/building-effective-agents

Anthropic. (2024b, November 25). Introducing the Model Context Protocol. https://www.anthropic.com/news/model-context-protocol

Anthropic. (2025, December 9). Donating the Model Context Protocol and establishing the Agentic AI Foundation. https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation

Arize AI. (n.d.). Phoenix: AI observability and evaluation [Computer software]. GitHub. Retrieved October 11, 2026, from https://github.com/Arize-ai/phoenix

CrewAI. (n.d.). crewAI [Computer software]. GitHub. Retrieved October 11, 2026, from https://github.com/crewAIInc/crewAI

Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (pp. 150–158). Association for Computational Linguistics. https://aclanthology.org/2024.eacl-demo.16/

LangChain. (n.d.). LangGraph [Computer software]. GitHub. Retrieved October 11, 2026, from https://github.com/langchain-ai/langgraph

Langfuse. (n.d.). Langfuse [Computer software]. GitHub. Retrieved October 11, 2026, from https://github.com/langfuse/langfuse

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33 (pp. 9459–9474). https://arxiv.org/abs/2005.11401

Mastra. (n.d.). Mastra [Computer software]. GitHub. Retrieved October 11, 2026, from https://github.com/mastra-ai/mastra

Model Context Protocol. (2025, November 25). Specification: Server features. https://modelcontextprotocol.io/specification/2025-11-25/server

TruLens. (n.d.). The RAG triad. Retrieved October 11, 2026, from https://www.trulens.org/getting_started/core_concepts/rag_triad/