Pydantic AI Harness makes self-written tools wait for the next run
An agent that can write a new tool for itself sounds efficient. The harder question is when that tool becomes part of the agent's authority.
Pydantic AI Harness draws a useful line. Its Capability Creation feature lets an agent write, validate, and persist a capability during a run. The capability is not loaded until the next run. Pydantic's explanation says the complete capability set is resolved when a run starts and remains fixed.
That is more important than the loose promise of a “self-improving agent.” Producing Python source is not the risky step. Letting code proposed inside an active session expand what that same session can do is.
A capability can change the agent's contract
Pydantic AI v2 defines capabilities as self-contained units of agent behavior. They can bundle instructions, tools, hooks, model settings, and related logic. Pydantic AI Harness is the companion open-source library, with reusable capabilities for work such as filesystem access, planning, context management, guardrails, and tool approval.
Capability Creation allows an agent to author another unit when the current toolkit does not fit the job. Pydantic says the harness imports generated code and runs static checks before treating it as a real capability. The result is then persisted for a later run.
If the capability is created on turn nine, it does not appear on turn ten. A new run has to begin.
Compare that with a live tool registry that can be mutated mid-session. After a mutation, early and late tool calls no longer share the same contract. A trace may show what the agent called, yet leave reviewers reconstructing which instructions, hooks, and permissions existed at that moment. A fixed manifest makes the run easier to replay and explain.
The CSV tool that should not go live immediately
Suppose a repository agent encounters a proprietary CSV export it cannot normalize. It writes a parser capability. If it can also install a dependency, register a shell tool, and immediately run the parser against production data, generation, testing, approval, and deployment have become one model decision.
The agent may have copied unsafe assumptions from an untrusted repository file. Its parser may request broad filesystem access when it needs one input directory. The tests it generated may cover only the same happy path encoded in the implementation. A dependency may be unpinned. None of those failures requires malicious intent.
Waiting for a new run creates room to inspect the actual artifact. The generated source can be diffed. Imports and dependencies can be checked against policy. Tests can run in isolation with hostile inputs. Reviewers can see every tool, hook, and permission before deciding whether the next run should load it.
Pydantic also notes that resolving capabilities once gives providers a stable tool and instruction prefix, which can help prompt caching. The more consequential result is operational: the run starts and ends under one capability contract.
Put a deployment gate between the two runs
The right mental model is not that the agent learned a skill. It proposed a software change.
Start by storing the generated capability in a staging area with the task, source context, model configuration, and base capability version that produced it. Import it in an isolated environment. Apply formatting, static analysis, dependency rules, unit tests, and adversarial inputs. Network access and broad shell commands should be rejected unless the capability has a specific reason to use them.
Then review the diff together with its declared tools, hooks, permissions, and dependencies. Privileged capabilities need explicit approval. Once accepted, store the artifact under an immutable version and place that version in the next run's manifest. Avoid a floating “latest” reference that can change between an incident and its replay.
This is the same principle behind least-privilege agent tools. A generated capability should expose the narrowest operation the task needs, not wrap an unrestricted shell. Its tool interface should make accepted inputs and failure modes visible before execution.
A new run is not automatic approval
The boundary becomes ceremonial if a system immediately starts another run, loads every persisted artifact, and supplies the same secrets and network access. Time passed, but no trust decision occurred.
Pydantic AI Harness supplies a place to enforce that decision. Teams still need locked dependencies, an allowlisted import surface, isolated tests, explicit tool permissions, and approval rules for credentials, user data, deployment systems, or network access.
Versioning also matters on both sides of the boundary. Pydantic AI Harness currently uses 0.x versioning. Its documentation says features are tested end to end and intended for production, but APIs may move between minor releases, with deprecation warnings where practical. Reproducing a run therefore requires both the capability version and the Harness version.
Step persistence makes that record even more important. Pydantic's design supports checkpointing, continuing, and forking runs. A continued run should retain the capability manifest it started with. If a fork intentionally adopts a new capability, that change belongs in the fork's history rather than being written back into the parent.
Useful agent traces should record the manifest, artifact hashes, validation results, approval identity, and activation time. Without those details, “the agent wrote a tool” is an explanation nobody can audit.
Keep proposal separate from authority
Capability Creation makes agent self-extension concrete: proposed behavior and active authority are separate states. The same lifecycle applies whether model calls go directly to a provider or through an API gateway such as ISH API.
An agent may write the missing CSV tool. It should not approve and activate that tool in the same run. Persist the proposal, test and review the artifact, then make a fresh run the first place an approved version can be used.
