What is an internal agent SDK? One shared way for many teams to build AI agents

In one picture

An AI agent is a program where a language model decides, step by step, what to do next: answer, look something up, or use a tool (a function the agent is allowed to call, such as "reset a password"). An SDK (software development kit) is a ready-made toolkit that programmers build on. An internal agent SDK is that toolkit for agents, owned by one platform team and used by every other team in the company. Each team describes its agent. The SDK handles everything every agent needs.

Why it matters

Without a shared SDK, every team builds its own agent plumbing. Ten teams make ten different retry loops, keep API keys in ten places, and each has its own idea of when a person must approve an action. Nobody can say what all the agents together cost, or which of them can take actions in real systems. When a model provider has an outage, ten agents break in ten different ways.

A shared SDK makes three questions easy to answer:

• What is this agent allowed to do, and who signs off on the risky parts? It's written in the agent's spec and the org's policy, not buried in code.
• What happens when something fails halfway through? The run was saved after every step, so it picks up where it stopped.
• What does each team spend, and on which models? Every model call goes through one gateway that counts it.

The building blocks

What each team writes:

Part	Holds / does	Everyday example
Agent spec	The agent's name, instructions (its prompt), the tools it may use, and the shape its answer must take	A job description: duties, permitted tools, what the deliverable looks like
Tools	Small functions the agent can call, each with typed inputs and a risk level	The equipment a new hire is issued, with the dangerous kit locked
Org policy	Which models a team may use, its spending budget, and which tools need a person's approval	A department's expense policy

What the SDK provides to everyone:

Part	Holds / does	Everyday example
Typed contracts	Exact descriptions of every input and output (a request, a tool's arguments, the final answer), checked automatically. Data that doesn't fit is rejected with a clear error. Example library: Pydantic.	A form with required fields that won't submit until it's filled in right
Orchestrator	Runs the agent as a series of steps (think, use a tool, think again, answer), with a step limit and automatic retries when a step fails. Example: LangGraph.	A project manager walking a checklist, and redoing a step that went wrong
Checkpoint store	Saves the agent's full state after every step, under a run id. A run can stop and resume later, even on another machine. Example: LangGraph checkpointers.	Saving your game before the boss fight
Approval gate	Pauses the run before a risky tool, asks a person, and resumes with their yes or no. Example: LangGraph interrupts.	A manager's signature on a purchase order
Model gateway	One door to every model provider. Teams ask for a tier ("smart", "fast"), not a vendor. The gateway picks the model, switches to a backup if the main one fails, and holds the keys. Examples: LiteLLM proxy; LangChain as the model interface in code.	A company travel desk: you say where you need to go; it books the airline, rebooks you if a flight is cancelled, and applies the travel policy
Cost meter	Counts tokens (the units models charge by) and money per run and per team, and stops a team at its budget. Example: LiteLLM virtual keys with budgets.	A prepaid company card with a monthly limit

Two of these are really memory. In the vocabulary of the CoALA framework (Sumers et al., 2023), the state the orchestrator passes between steps is the agent's working memory, and the checkpoint store plus an audit log of past runs is its episodic memory: a record of what happened.

How it works

Follow one request: an employee asks the IT helpdesk agent, "My VPN is broken."

1. Check. The SDK checks the request against the agent's contract and the org policy. A malformed request stops here, before any model is paid for.
2. Think. The orchestrator sends the conversation, the prompt and the list of allowed tools to the model through the gateway. The model replies with either a final answer or a request to use a tool, such as reset_vpn(user="ana").
3. Gate. If the tool is marked risky, the approval gate pauses the run. The checkpoint store saves the exact state, and a person sees what the agent wants to do and why. The run can wait for minutes or days; nothing is held in memory.
4. Act. On a yes, the run resumes from the checkpoint and the tool runs. If the tool fails for a temporary reason (say, a timeout), the orchestrator retries it a set number of times. On a no, the refusal goes back to the model as the tool's result.
5. Repeat. The tool's result goes back to the model, and steps 2–4 repeat until the model has an answer or the run hits its step limit or budget.
6. Answer. The final answer is checked against the agent's output contract (for example "a ticket with a status and a summary"). If it doesn't fit, the model is asked to fix it, a limited number of times. Then the typed result goes back to the caller, and the cost meter records what the run spent.

The team that built the helpdesk agent wrote none of steps 1–6. It wrote a prompt, two tools and one line saying reset_vpn needs approval.

Patterns & variations

Pattern	What changes	When to use it
Single tool-calling loop	The shape in Fig. 2: one model, a set of tools, a loop until done. Anthropic calls the basic unit the "augmented LLM" (Anthropic, 2024)	The default. Most team agents start and stay here
Fixed workflows: prompt chaining, routing (Anthropic, 2024)	The orchestrator follows steps the team defined in advance instead of letting the model choose each step	Predictable tasks, like "extract invoice fields, then validate them"
Orchestrator–workers (Anthropic, 2024)	One agent splits the job and hands parts to worker agents, each with its own spec and tools	Big jobs whose parts can't be known in advance
Shared tool servers (MCP)	Tools live in separate servers that speak the Model Context Protocol, so many agents and teams reuse them	When the same tools (search, ticketing) are needed by many agents
Gateway as a service vs. in the SDK	The gateway runs as its own service with central keys and budgets, or as code inside the SDK	A service once you have more than a few teams; in-code for a prototype
Approval by risk tier	Low-risk tools run freely, medium ones are logged, high-risk ones always need a person, following least-privilege advice (OWASP, 2025)	Whenever agents touch real systems: money, accounts, customer data

Pitfalls

Pitfall	What you'll notice	Remedy
The SDK tries to do everything	Teams wait on the platform team for every change, or fork the SDK	Keep the SDK to the shared parts in Fig. 1; let teams add their own tools and prompts freely
Vendor model names in team code	Switching providers, or surviving an outage, means editing every agent	Teams ask for tiers ("smart", "fast"); only the gateway knows vendors
Approvals without saved state	A pending approval is lost when a server restarts, and the person's "yes" goes nowhere	Save a checkpoint before pausing; resume by run id
Trusting raw model output	Downstream systems crash on a missing field or a date in the wrong format	Check every answer against its contract; re-ask on failure, then fail loudly
No step or spending limit	A confused agent loops, calling tools and the model over and over, and the bill spikes	A step limit in the orchestrator and a hard budget per team at the gateway
Approval fatigue	People click "yes" without reading, because everything asks	Gate only high-risk tools, and show the exact action and reason on the approval screen

Go further

• See it built: None yet. A good demo would be a small open-source SDK with two example "orgs" (an IT helpdesk agent with an approval step, an invoice reader with a typed output), a model gateway with a backup model and per-team budgets, and a web page for approving pending actions.
• Read: Anthropic's Building effective agents (2024) for the patterns, then the LangGraph pages on persistence, interrupts and fault tolerance for how saving, pausing and retrying work in practice.
• Next: CoALA: how language agents remember, decide and act gives the vocabulary for what an agent remembers and does. Explainers on agent harnesses and AI Ops are planned.