AI
PydanticAI Source Audit: Replay Telemetry, Deferred Tools, and Cache Collapse
Source audit of pydantic/pydantic-ai (commit a2f30fab), examining _agent_graph state nodes, _deferred approvals, _durable_replay telemetry, and _cache_health.


Why Do Durable Agent Loops Double-Count Telemetry and Lose Track of Pending Approvals?
Most Python agent SDKs treat a run as a single in-memory while loop: call a model, execute tools, append messages, and return text. That model breaks down the moment an agent runs inside a durable workflow engine (such as Temporal or DBOS), pauses for human approval mid-turn, or sustains long multi-turn conversations against provider prompt caches.
When a durable engine resumes a crashed or suspended workflow, it replays the agent run from the beginning while returning stored results for completed units. Without explicit replay accounting, OpenTelemetry instrumentation re-emits spans and double-bills token and cost metrics for work that never re-executed. Meanwhile, pausing for human approval leaves dangling tool calls in message history unless the framework separates pending external requests from resolved tool results, and silent edits to early messages invalidate provider prompt caches without alerting operators.
Inspecting pydantic/pydantic-ai at pinned commit a2f30fab734755e41c9f6b72e09f7d93340e6c6b reveals four concrete internal subsystems that address these failure modes directly: _agent_graph.py, _deferred.py, _durable_replay.py, and _cache_health.py.
%%{init: {"flowchart": {"useMaxWidth": false}}}%%
flowchart TD
subgraph GraphLoop ["1. Typed Agent State Machine"]
UserNode["UserPromptNode"]
ModelNode["ModelRequestNode"]
ToolsNode["CallToolsNode"]
FinalNode["SetFinalResult"]
end
subgraph DeferredBoundary ["2. Deferred Execution Boundary"]
Requests["DeferredToolRequests"]
Approvals["approvals: approval IDs"]
Externals["calls: question or external IDs"]
Results["DeferredToolResults"]
end
subgraph TelemetryGuard ["3. Durable Replay and Cache Guard"]
ReplayScope["replay_scope and ContextVar _Tally"]
CacheStore["ConversationCacheMarkStore"]
end
UserNode --> ModelNode
ModelNode --> ToolsNode
ToolsNode -->|Normal tool return| ModelNode
ToolsNode -->|Deferred approval or external call| Requests
ToolsNode -->|Structured output ready| FinalNode
Requests --> Approvals
Requests --> Externals
Approvals -.->|Human decision| Results
Externals -.->|External caller output| Results
Results -.->|Next agent run| UserNode
ModelNode --> ReplayScope
ModelNode --> CacheStore
style GraphLoop fill:#f8fafc,stroke:#38bdf8,stroke-width:2px,color:#111827
style DeferredBoundary fill:#fff7ed,stroke:#f97316,stroke-width:2px,color:#111827
style TelemetryGuard fill:#f0fdf4,stroke:#22c55e,stroke-width:2px,color:#111827
Modeling Agent Turns as Explicit pydantic_graph Nodes
In pydantic_ai_slim/pydantic_ai/_agent_graph.py, the agent execution loop is built on top of pydantic_graph (BaseNode, End, Graph, GraphBuilder, GraphRunContext). Rather than hiding turn transitions inside recursive callbacks, the runtime decomposes execution into explicit AgentNode subclasses carrying GraphAgentState and GraphAgentDeps:
UserPromptNode: Anchors initial user input, repairs or prepares history (_clean_message_history,_repair_dangling_tool_calls,_merge_consecutive_messages), and resolves deferred tool inputs from a prior pause.ModelRequestNode: Applies request deadlines (start_request_deadline,enforce_request_deadline), handles continuation segments (_ContinuationStreamedResponse,MAX_GENERATION_CONTINUATIONS), and manages fallback model attempts (AttemptLog,carried_attempts).CallToolsNode: Separates normal local tool execution (process_tool_calls) from tools that raise approval or external-execution boundaries.SetFinalResult: Finalizes validated output once structured or plain output criteria pass.
This explicit node decomposition makes step-by-step iteration (agent.iter(...)) and persistence across process boundaries straightforward, because every boundary between a model request and a tool invocation is a typed state transition. In that respect, it shares the strict phase separation we saw in LlamaIndex Workflows State Machines, while keeping state transitions strongly typed with Pydantic models rather than loosely subscribed event queues.
Separating Approvals, Questions, and External Calls in _deferred.py
A common mistake in human-in-the-loop agent harnesses is treating every paused tool call as a generic “approval” prompt. In pydantic_ai_slim/pydantic_ai/_deferred.py, PydanticAI narrows and structures this boundary with DeferredToolKind = Literal['approval', 'question', 'external'] and DeferredToolRequests.
- Three Distinct Pause Semantics:
requests.approvalsholds calls waiting onToolApproved(override_args=...)orToolDenied(message=...).requests.callsholds both'external'calls (whose execution happens outside the agent runtime) and'question'calls (created viaCallDeferred(kind='question'), such asAskUser), distinguished by an internal_kindsmapping serialized only when questions exist. - Fail-Closed ID Matching:
DeferredToolRequests.build_results(...)validates that every key inapprovalsexists inself.approvalsand every key incallsexists inself.calls, raising aUserErrororValueErroron mismatched IDs instead of silently dropping responses. - Partial Resolution Tracking: Calling
requests.remaining(results)filters resolved IDs viafilter_deferred_results(requests, results)and returns either a smallerDeferredToolRequestsobject with remaining metadata and_kinds, orNoneonce every pending call and approval is settled. - Caller-Owned Authorization: Notably,
DeferredToolResults.answered_byrecords who answered each tool call (tool_call_idto user, bot, or policy name), and the source docstring explicitly warns that PydanticAI does not verifyanswered_byitself—hosts must authorize respondents before populating it.
Separating unresolved requests from the message history ensures that developers do not confuse historical checkpoints with active pending approvals, avoiding the checkpoint-inspection pitfalls analyzed in LangGraph History Is Not Pending Approval and aligning with host-boundary controls examined in MCP Skills Integrity Is Not Approval.
Suppressing Duplicate Telemetry During Durable Replay (_durable_replay.py)
Durable execution engines such as Temporal replay deterministic workflow code from the beginning after a pause or worker restart. If an agent’s OpenTelemetry instrumentation runs normally during replay, every already-completed model call and tool invocation emits a duplicate span and double-counts token usage.
pydantic_ai_slim/pydantic_ai/_durable_replay.py solves this cleanly using contextvars.ContextVar without coupling core instrumentation to a specific workflow vendor:
1
2
3
4
5
6
from pydantic_ai._durable_replay import record_durable_unit, replay_scope, start_as_current_span
with replay_scope():
with start_as_current_span(tracer, "agent_turn") as span:
# Engine returns cached result instead of executing body
record_durable_unit(executed=False)
- Scoped
ContextVarIsolation:replay_scope(is_replaying=...)sets_scopeand a root_Tally(parent=None)inContextVars so concurrent async tasks inside the same process do not contaminate each other’s counts. - Hierarchical Credit Propagation: When
start_as_current_spanopens a span, it creates a child_Tally(parent=parent)and captures aReplaySnapshot(_executed,_replayed). Whenever a durable unit completes,record_durable_unit(executed=...)walks up thetally.parentchain, incrementingexecutedorreplayedon every enclosing span. - Selective Span Suppression: In the
finallyblock ofstart_as_current_span,state.replayed()returnsTrueifis_replaying()is true, or if zero units executed (self._tally.executed == self._executed) while at least one unit replayed (self._tally.replayed > self._replayed). Whenstate.replayed()isTrue,span.end()is never called—so the OpenTelemetry SDK never exports the duplicate span—andparent.credit(executed=True)is skipped.
This design handles mixed runs naturally: if a workflow replays three earlier steps and then executes one new step inside the same parent turn span, executed > 0, so the parent span closes and exports normally while purely replayed child spans vanish.
Detecting Silent Prompt-Cache Collapse Across Runs (_cache_health.py)
Long-running agents depend heavily on provider prompt caching to control latency and cost. Yet a single subtle mutation—re-ordering system instructions, editing an early message, or letting a run sit idle past a provider’s retention TTL—can cause a cache miss on thousands of prefix tokens while the model call still returns HTTP 200.
pydantic_ai_slim/pydantic_ai/_cache_health.py introduces a shared diagnostic layer consumed by both OpenTelemetry instrumentation and Python warnings (WarnOnCacheBusts):
| Threshold / Type | Value in Source | Engineering Purpose |
|---|---|---|
MIN_CACHEABLE_TOKENS | 4096 | Prevents false alarms below largest model minimum cacheable prefix (4,096 tokens) |
MIN_MISSED_RATIO | 0.05 | Triggers collapse check only when shortfall exceeds 5% of established cached prefix |
MIN_MISSED_TOKENS | 2000 | Filters out small provider rounding variations below 2,000 missed tokens |
CacheKey | (provider_name, provider_url, model_name) | Scopes retention clock and established prefix per provider cache endpoint |
CollapseReason | 'unexpected' \| 'ttl_expired' \| 'compacted' \| 'unknown' \| 'unreported' | Distinguishes real prefix drift ('unexpected') from normal TTL expiry or CompactionPart |
Because ConversationCacheMarkStore keys CacheMarks by RunContext.conversation_id and keeps an LRU-bounded OrderedDict across runs, the first request of a resumed conversation is checked against the prefix established by the previous run. By comparing elapsed time against _expected_cache_retention and tracking CompactionPart digests (compactions count and latest_compaction hash), the store avoids raising false alerts when a drop is simply caused by intentional history compaction or TTL expiration.
Key Takeaways
- Explicit Graph Nodes Simplify Persistence: Decomposing the turn loop into
UserPromptNode,ModelRequestNode,CallToolsNode, andSetFinalResultgives durable runners clean serialization boundaries and history-repair hooks. - Typed Deferral Prevents Ambiguous HITL State:
DeferredToolRequestsseparates'approval','question', and'external'pauses by ID and enforces strict matching inbuild_resultsandremaining. - Parent-Chained
ContextVarTallies Eliminate Replay Double-Counting:_durable_replay.pysuppresses OTel span export whenever all durable units inside a span were replayed, without leaking state across concurrent async tasks. - Cross-Run Cache Health Separates Drift from Expiry:
ConversationCacheMarkStoreflags unexpected prefix invalidations (>5% and >=2,000 tokens lost) across conversation continuations while filtering out expected TTL expirations and compactions.
Open Questions
- Authorization Coupling on
answered_by: Because PydanticAI intentionally leaves authorization checks onDeferredToolResults.answered_byto the host application, how should multi-tenant hosts enforce cryptographic or session-bound signer verification before callingbuild_results? - Distributed Cache-Mark Sharing:
ConversationCacheMarkStoreoperates as an in-process LRUOrderedDict; in horizontally scaled worker pools where consecutive turns of oneconversation_idhit different pods, will teams need an external store adapter to preserve'unexpected'vs'ttl_expired'attribution across workers? - Custom Span Processors vs Un-ended Spans: Leaving replayed OpenTelemetry spans un-ended prevents export in standard SDKs, yet long-lived processes using custom
on_startspan processors should verify those processors do not retain un-ended span references in memory.
References
- PydanticAI Repository — Canonical upstream repository, pinned commit
a2f30fab734755e41c9f6b72e09f7d93340e6c6b. pydantic_ai/_agent_graph.py—GraphAgentState,UserPromptNode,ModelRequestNode,CallToolsNode, and history repair helpers.pydantic_ai/_deferred.py—DeferredToolRequests,DeferredToolResults,DeferredToolKind, andfilter_deferred_results.pydantic_ai/_durable_replay.py—replay_scope,record_durable_unit,ReplaySnapshot, andstart_as_current_span.pydantic_ai/_cache_health.py—ConversationCacheMarkStore,CollapseReason, and prompt-cache collapse thresholds.
Editorial method: claims in this article point to primary documentation, pinned source code, runnable checks, or dated measurements, with AI-assisted research under the site's evidence-gated policy. How this site verifies claims and discloses AI assistance · Report a correction.
Working on something like this?
I take a small number of paid, scoped reviews: AI agent/RAG architecture diagnosis, Unity CI & build-automation audits, and multimodal QA design review. Each one ends in a written findings document.
Work with me