Agentic AI System Design
A complete reference to the architectural decisions behind robust agentic AI systems—from model routing and tool contracts through memory, orchestration, evaluation, approvals, reliability, cost, RAG, observability, security, and privacy.
Single-Agent vs Multi-Agent
Single-Agent
One primary agent owns the entire workflow.
It may still call multiple tools, retrieve documents, update memory, and run many reasoning steps—but the control loop stays centralized.
Example: a customer support agent classifies a request, retrieves account context, calls a billing API, asks for confirmation, then generates the final response.
Multi-Agent
Workflow is split across specialized agents: planning, knowledge retrieval, code writing, review, and execution.
The key idea is separation of responsibilities: defined roles, input contracts, output contracts, and routing logic between agents.
Use multi-agent when…
- Task has clear specialization
- Parallel work helps
- Review loops are valuable
- Workflows are long-running
But multi-agent adds…
- More coordination overhead
- More failure modes and states to track
- More logs to inspect
- More places where cost and latency grow
Model Routing, Not One Model
| Step type | Model tier |
|---|---|
| Intent classification, routing, extraction, schema filling, simple summarization | Small / cheap / fast model |
| Book vs reschedule vs cancel vs question detection | Small model is sufficient |
| Date, time, doctor name, appointment type → JSON schema | May not need an LLM at all |
| Ambiguous constraints, deeper reasoning, synthesis—such as “I’m traveling next week, avoid mornings” or “after my lab results come in” | Stronger reasoning model |
Structured outputs
Any step feeding another system returns predictable structure—never free-form prose. Use JSON Schema, Pydantic models, or function/tool calling.
Three-question contract per step
1. Which model runs this step?
2. What output contract does it return?
3. What happens if it fails or returns invalid output?
Tools Are APIs—Give Them Contracts
Tools are interfaces between the model and the external world: database lookups, CRM/calendar/payment APIs, code interpreters, search, ticketing, Slack, and Google actions.
Anti-pattern
update_user accepts one vague string: “update this user based on the request.” That is too open-ended.
Safer contract
user_id · field_to_update · new_value · reason · source_request_id · confirmation_required
Tool outputs must be machine-readable: structured error on failure, structured result on success. The agent should never have to parse messy prose from your backend.
Risk tiers
Fetching available appointment slots is very different from canceling an appointment. Understand the risk factor per tier.
MCP (Model Context Protocol)
One emerging standardized pattern for exposing tools, resources, and context to agents. Optional. Even without MCP, the principle is identical: tools need contracts, permissions, boundaries, and logs.
Memory Is Not One Thing
State
Current execution context of the workflow.
- Which step are we on?
- What has been collected so far?
- Which tools were called, and what did they return?
- Has the user confirmed?
- Did the workflow pass or fail?
Memory
Broader, cross-run information.
- Conversation history
- User preferences
- Past actions
- Retrieved knowledge and document context
- Summaries and long-form information useful later
Storage by access pattern
| Data | Store |
|---|---|
| Conversation & workflow state | Redis, DynamoDB, Postgres, MongoDB, or another low-latency store |
| Application state | Your application database |
| Knowledge retrieval (RAG) | Pinecone, PG Vector, Weaviate, Elasticsearch, OpenSearch, or a managed knowledge base |
| Long-term archive | Cheaper object storage such as an S3 bucket |
Short-term vs long-term
Short-term: what you pass into the prompt for the current turn.
Long-term: retrieved selectively.
Never stuff everything into the context window. Retrieve the smallest useful context for the current step.
Memory design = data architecture
- What to store
- Where to store it
- How long to keep it
- How to retrieve it
- What is safe to send to the model
Explicit Control Flow
Orchestration is the control layer: user request → intermediate steps → tool calls → final output. It can be implemented as plain code, LangGraph, Temporal, LlamaIndex Workflows, LangChain, custom state machines/queues, or a combination.
Simple flows → deterministic pipeline
Not every workflow needs planning and reflection. If the sequence is mostly known, design it as a pipeline or state machine. Use agentic reasoning only when the workflow genuinely needs dynamic decisions.
Complex flows → graph orchestration
Graphs represent branching, retries, loops, approval gates, and fallbacks.
- Extraction fails → retry with a different prompt or model
- No available appointment → branch to suggest alternate
- Cancellation request → branch to confirmation path
Agent-to-agent routing
For every handoff define which agent owns the next step, what information gets passed, the expected output format, and how conflicts are resolved. Without this, multi-agent systems become extremely difficult to debug.
Trace-Level Evals, Not Just Final Answers
In agentic systems, “no exception thrown” no longer means “it works.” Evaluate every important step of the trajectory: intent classification, retrieval quality, tool selection, tool arguments, policy compliance, confirmation behavior, final answer quality, and task success.
Test sets
This set becomes your regression suite whenever you change the model, prompt, retrieval logic, or tool schema.
Production evals
- Sampled async evals—LLM-as-a-judge scores a percentage of conversations offline
- High-signal user feedback → eval queue
- LLM-as-judge is useful but not perfect: combine model grading + deterministic checks + human review
Produce metrics, not just examples
Human in the Loop for High Impact
These actions should never happen just because the model inferred the intent:
Deterministic validation
- Check ownership
- Check permissions
- Check action is allowed
- Check required fields are present
- Check user confirmed the exact action
Source of truth
Your application code—not the agent—is the source of truth for business rules. An execution agent must not blindly trust a planning agent.
System Reliable Even When the Model Isn’t
Decompose prompts
A single joint prompt that classifies, retrieves, decides policy, calls tools, and writes the response is very hard to test. Smaller steps are easier to evaluate and debug.
Validate → retry → fallback
If the next step depends on the model’s output, never rely on free-form text. Validate the schema, retry on invalid, and fall back if it still fails.
Model call = unreliable dependency
Timeouts, malformed outputs, rate limits, provider errors, and degraded quality all need explicit handling paths.
Deterministic validation ≠ model reasoning
Model extracts a date → code validates the date. Model picks a user ID → code verifies permissions. Model proposes a tool call → code validates arguments.
Every Lever, Applied Together
| Lever | Rule |
|---|---|
| Model routing | Small models for simple classification/extraction; larger models only when ambiguity, reasoning, or synthesis is required |
| Token limits | Limit output tokens aggressively. Two-sentence answer → no essay; downstream needs JSON → no prose |
| Caching | Cache retrieval results, repeated policy lookups, tool metadata, and stable context |
| Batching / async | For non-blocking work such as evaluation and summaries |
| Streaming | Stream user-facing responses that may take time |
| Progress states | Long-running tool calls → show progress, never a blank screen |
| Early scope gates | Cheap filters, rules, and intent gates before expensive reasoning; out-of-scope requests burn tokens and tool calls |
Track: tokens in · tokens out · cost per step · cost per conversation · cost per successful task.
Pass the Right Context
Each source has different freshness, trust, privacy, and latency characteristics. Choose per step; do not dump everything.
Retrieval pipeline
- Document chunking
- Metadata filters
- Hybrid search when useful
- Reranking
- Freshness controls
- Source attribution when trust matters
Trusted vs untrusted
- Retrieved docs must not override system instructions
- Tool outputs are data, not instructions
- User content is isolated from developer/system instructions
- Long conversations → summarization + checkpoints, stored separately from raw logs
Log the Anatomy of Every Run
| Category | Fields |
|---|---|
| Identity | Model name · model version · prompt version · step name · workflow ID · conversation ID |
| Tool use | Tool name · tool arguments, with sensitive values masked |
| Performance | Latency · time to first token · tokens in · tokens out · cost |
| Resilience | Retry count · fallbacks triggered · errors |
| Feedback & evals | Whether feedback was used · eval scores |
Everything Touching the Model Is Attacker-Controlled
User messages
Direct prompt injection
Retrieved documents
Indirect prompt injection
Tool outputs
Poisoned data
Model outputs
Unsafe commands—SQL, HTML, or other code
Defense rules
Minimum Data, Defined Boundary
Send the minimum
Model needs appointment ID + availability → do not send the full patient report. It only summarizes a ticket → do not send the full customer history.
PII & retention
Mask PII at the right time. Set retention policies for logs, traces, and conversation archives. Keep sensitive fields out of the prompt unless required for the task.
Data boundary
Your model provider, vector database, observability platform, and logging systems are all part of your data boundary.