Throughout this series, I've argued that reliable AI agents require explicit structure - defined entities, valid states, constrained operations, enforced invariants. The structure is what makes agent behavior predictable and errors correctable.
But there's an obvious objection: if agents are capable enough to do complex work, why can't they create the structure themselves? Why do humans need to define schemas, design state machines, specify validation rules? Can't we just delegate that too?
This question gets at something deeper than engineering trade-offs. It's asking where human judgment is actually irreducible - and whether that boundary is fixed or moving.
Can Agents Create Their Own Structure?
The appeal is clear. Agents can already:
- Write database schemas based on natural language descriptions
- Generate type definitions and validation logic
- Propose categories after analyzing examples
- Design workflows based on observing how work gets done
If an agent can write code, and structure is just a specialized kind of code, then agents should be able to create structure. The whole framework I've been describing would become a bootstrapping problem rather than an ongoing human responsibility.
Maybe you describe what you want in plain language. The agent proposes structure. You review it, tweak it, approve it. The agent then operates within the structure it created. Humans stay in the loop but at a higher level - steering rather than building.
This isn't fantasy. It's already happening in constrained domains. Coding agents generate schemas. Analysis agents propose categorization systems. The line between "agent uses structure" and "agent creates structure" is blurring.
So where's the limit?
The Irreducibility of Specification
Here's the core problem: by the time you've provided enough detail for an agent to create the right structure, you've essentially written the specification yourself.
Consider a progression:
You write the code. All decisions are explicit in the implementation. Maximum effort, maximum control.
You write detailed requirements. The agent implements them. You've moved up a level of abstraction, but you still made all the decisions. You just expressed them differently.
You describe what you want with examples. The agent infers the requirements and implements them. Less upfront work. But now: how do you know the agent inferred correctly? You have to check the output. Checking requires knowing what you wanted. Which means the specification existed in your head - you just didn't write it down.
You give a loose directive. "Handle customer support." The agent figures out everything - what categories exist, what states are valid, what operations are allowed. But inevitably it does something you didn't want. You correct it. That correction is specification, delivered incrementally. The information still flows from you to the system; it just flows through feedback rather than upfront design.
The work doesn't disappear. It transforms.
You can specify upfront (front-loaded effort, predictable behavior) or specify through correction (distributed effort, unpredictable intermediate states). But the information about what you want has to get from your head into the system somehow. That transfer is the irreducible part.
This suggests a limit: agents can do the work, but someone has to define what the work is. The wanting is yours. The specification of success is yours. You can delegate implementation of your intent, but not the intent itself.
But Verification Is Easier Than Generation
There's a real asymmetry we haven't fully exploited: it's easier to recognize a correct solution than to generate one.
You might struggle to specify exactly what you want upfront. But when you see the agent's proposal, you can often tell whether it matches your intent. "Yes, that's what I meant." "No, that's not quite right." This is genuine signal that doesn't require you to have a complete specification in advance.
This suggests a different workflow. Be loose in your description. Let the agent propose structure. Verify the output. Iterate.
For many domains, this works. The agent generates candidate schemas, state machines, validation rules. You review them against your intuition about the domain. You test with examples. You refine through feedback. Eventually you converge on something that captures your intent - without ever having to specify it precisely upfront.
The agent did the generation work. You did the verification work. Comparative advantage, properly allocated.
So maybe the limit isn't "humans must specify" but "humans must verify." And verification is easier. The framework holds, but the human role is smaller than it first appeared.
The Trust Problem
Here's where it gets complicated: verification requires knowing what you're looking for.
If you understand the domain deeply, you can review a proposed schema and see whether it captures the important distinctions. You can look at a state machine and recognize missing transitions. You can spot invariants that should hold but aren't enforced.
But what if you're not technical? What if you understand your business but can't read a schema? How do you verify that the structure captures your intent?
Options:
Behavioral verification. Ignore the structure itself. Test with examples. "When this situation occurs, what does the system do?" If the behavior matches your expectations across enough cases, trust the structure. You're treating it as a black box, verified by its outputs.
Delegated verification. Have someone you trust review the structure. A technical advisor. A consultant. An auditor. You don't verify directly; you verify that a competent, trustworthy person verified.
Earned trust. Let the system run. Monitor for problems. If it works in production over time, trust it. The structure is verified by survival.
Each approach has failure modes.
Behavioral verification misses edge cases. Your test examples don't cover everything. The structure might be subtly wrong in ways that don't manifest until an unusual situation arises.
Delegated verification depends on the delegate. Do they understand your intent well enough? Are their incentives aligned? Are they actually competent? And here's the deeper problem: if they know both the domain and the engineering well enough to verify the solution, why not have them create it in the first place? In practice, no single person has both - domain experts and engineers need to collaborate. That collaboration is tricky, and it's why many projects fail even without AI agents involved. The gap between what gets ordered and what gets built has always been the hard part.
Earned trust is backward-looking. Past performance doesn't guarantee future behavior. The structure might be fine until the business changes, or volume scales, or an adversary probes for weaknesses.
None of these are fully satisfying. But collectively, they're often good enough. Most systems in the world are running on some combination of spot-checking, delegated expertise, and "it hasn't broken yet."
What Can a Human Do That an Agent Can't?
If an agent can propose structure and a non-technical human can verify behaviorally, where do technical humans fit? What does a human engineer do that an agent can't?
Push back meaningfully. A human who's learning your domain will say "you told me X, but that contradicts Y - which do you actually want?" They surface inconsistencies in your own understanding. They're modeling your intent, not just your words.
Express genuine confusion. An agent will produce something. It might be confidently wrong. A human who doesn't understand will ask questions. The confusion is informative - it reveals where the specification is ambiguous.
Have stakes. If a human engineer builds the wrong thing, there are consequences. Reputation. Employment. Professional accountability. This creates alignment pressure that "the agent made a mistake" doesn't.
Maintain context over time. A human who's worked with you for years understands not just the current request but the history, the politics, the unspoken constraints. They know why certain decisions were made and which ones are load-bearing.
These advantages are eroding.
Agents are getting better at surfacing inconsistencies. They're learning to express uncertainty rather than confident hallucination. Context windows are growing; memory systems are emerging. The gaps are real but shrinking.
Maybe in five years, none of these will be differentiators. Or maybe they'll prove more durable than they appear - something about embodied stakes, long-term relationships, genuine skin in the game that resists automation.
I don't know. I don't think anyone knows.
The Interpretability Requirement
Here's a requirement that might prove more durable: when things go wrong, someone needs to understand why.
Behavioral verification works when the system is working. You check outputs, they match expectations, everything's fine. But when something fails - an edge case produces wrong behavior, an invariant is violated, a customer is harmed - you need to debug.
Debugging requires understanding the structure. Not just "what did the system do" but "why did it do that" and "what needs to change." This is structural inspection, not behavioral verification.
If the structure is opaque - generated by an agent in a form no one understands - debugging becomes archaeology. You're reverse-engineering intent from behavior, guessing at what the structure was supposed to do.
Maybe an agent can do this archaeology. "Explain why this structure produces this behavior. Propose a fix." If the explaining agent is good enough, you've closed the loop without human structural understanding.
But notice what you're trusting: the agent's explanation of a structure the agent created. You're back to the grading-your-own-homework problem. The explanation might be wrong in the same way the structure is wrong.
This suggests a requirement: structure must be interpretable by someone - human or agent - who can be held accountable for the interpretation. If no one can explain why the system behaves as it does, you've lost the ability to govern it.
Today, that usually means human interpretability. The structure should be something a competent engineer can read, understand, and modify. The same principle applies in other domains - a lawyer might use templates and automated data collection, but a human attorney with real stakes on the line still needs to sign off on work that goes to clients or gets filed in court. This is a real constraint on agent-generated structure: not just "does it work" but "can we understand it."
Tomorrow? Maybe interpretability can be agent-mediated. But the requirement for some interpretable account doesn't go away. It just might be satisfied differently.
The Values Beneath the Structure
There's another irreducibility that's easy to miss: structure encodes values.
When you define categories, you're deciding what distinctions matter. When you specify invariants, you're declaring what must never happen. When you design states and transitions, you're encoding a theory of how the domain works and should work.
These aren't neutral technical decisions. They embed priorities, assumptions, judgments about what's important.
Who gets to decide that "billing" and "technical" are the categories that matter? The agent can classify individual complaints, but someone chose that this distinction - rather than "urgent versus routine" or "new customer versus existing" - is how the business should see its problems. That choice affects routing, response time, metrics.
Who gets to decide that refunds over $1,000 require human approval? That threshold embeds a judgment about risk tolerance, customer trust, operational efficiency.
An agent can propose these decisions. But the legitimacy comes from human acceptance. "Yes, that's how we want to treat this." "No, that threshold is too aggressive." The agent surfaces options; humans choose values.
This might be the deepest irreducibility. Not "humans must do the technical work" but "humans must own the value judgments that the technical work encodes." You can delegate the implementation of your values. You can't delegate the values themselves - not without losing something important about accountability and self-determination.
What's Worth Building?
Given all this uncertainty, where should effort go?
Structure that enables verification. If behavioral verification is how non-technical humans will check agent-generated structure, make it easy. Observable outputs. Test harnesses. Clear examples of expected behavior. Audit logs that show what happened. The structure should be verifiable even if it's not directly interpretable.
Agents that propose and explain, not just generate. The agent should surface its reasoning. "I created this category because these examples seemed to cluster together." "This invariant prevents the scenario you described in example 4." Explanations help humans verify and build trust. They also create artifacts for debugging when things go wrong.
Feedback loops that surface failures fast. If earned trust is part of the verification story, failures need to be visible quickly. Monitoring. Alerting. Exception tracking. The faster you learn that the structure isn't working, the less damage from running on broken assumptions.
Human oversight at points where values matter most. Not everywhere - that defeats the purpose of automation. But at the junctures where value judgments are encoded: what categories exist, what thresholds apply, what invariants are enforced. These are the decisions that should be human-approved even if they're agent-proposed.
Interpretable structure as a default. Until we have robust agent-mediated interpretation, structure should be human-readable. Schemas that make sense. State machines that can be diagrammed. Validation rules that can be explained. This is a constraint that costs something - agent-generated structure might be more powerful if it didn't need to be human-interpretable. But the cost is worth paying for governability.
Humility about where the limits are. The boundary between what agents can and can't do is moving. What seems irreducibly human today might be automatable tomorrow. What seems automatable might have hidden prerequisites we haven't recognized. Build systems that can evolve as our understanding evolves.
The Open Questions
I've argued throughout this series that structure is the bottleneck for reliable agent systems. That remains true. But I want to end with honesty about what I don't know.
Will behavioral verification prove sufficient? Maybe testing against examples, plus production monitoring, plus fast feedback loops is enough. Maybe structural interpretability is a nice-to-have, not a requirement. We'll find out as agent-generated systems scale.
Will the human advantages persist? Stakes, relationships, genuine confusion as signal - these feel important. But I can't prove they won't be replicated or routed around. The history of automation is full of "essentially human" capabilities that turned out to be automatable. Bank tellers seemed irreplaceable until ATMs. Switchboard operators were essential until automatic switching. Chess required human intuition until it didn't.
Where exactly is the intent boundary? I've argued that humans must specify intent, but specification can be implicit in feedback. How implicit can it get before you've lost meaningful human control? I don't have a crisp answer.
What happens when structure becomes opaque? If agent-generated structure outperforms human-interpretable structure, there will be pressure to use it despite the governance costs. How do we navigate that trade-off?
These questions don't have settled answers. The framework I've offered is a starting point for thinking about them, not a final resolution.
What I'm confident about: the structure problem is real. Agents operating without explicit structure will continue to fail - not in predictable instances, but in predictable patterns. The specific failures surprise you; the fact of failure shouldn't. The solution involves making structure explicit - whether created by humans, agents, or some collaboration.
What I'm less confident about: exactly where the human role is irreducible versus merely current. The frontier is moving. For now, humans remain essential for navigating the mess of business logic and organizational complexity. But I don't know if that changes suddenly - AGI arrives and makes all of this obsolete - or gradually, through a series of ever-improving tools until one day we realize there's nothing left for us to do. Five years from now, this essay might read as either prescient or quaint.
Build for the world we're in now. Stay alert to how it's changing. Hold your assumptions lightly.
The structure matters. Who creates it, and how, is still being written.
This essay is part of a series on building reliable AI agent systems.
Overview: The Structure Problem
Previous: What Software Engineers Actually Do
Next: Why Coding Agents Work — Real-world evidence for the framework