Agents Don't Need More Autonomy. They Need a Gateway and a Memory.

The first agent demo always works. The tenth agent in production is what breaks you.

Give a capable model a database connection and a goal, and it will do something impressive. Give it to a real accounting system with real approvals, and you get a very fast way to create a very hard-to-audit mess.

I run several businesses on one platform, AyoWork, and I’ve spent the last year learning where agents belong and where they don’t. The short version: the model is the least interesting part of the system.

Table of Contents

• Why Free-Form Agents Fail in a Ledger
• The Architecture: Seven Boundaries, One Rule
• Memory: The Part Nobody Demos
• The Loop: Agents Reviewing Agents
• Where Physics Fits
• Summary
• Technical Q&A

Why Free-Form Agents Fail in a Ledger

A double-entry ledger is a deterministic machine. Debits equal credits, or the transaction is rejected. A language model is a probabilistic machine. It is usually right, which is exactly the problem.

“Usually right” is fine for drafting an email. It is not fine for posting a journal entry, changing a bill of materials, or releasing a build to a CNC cell. In my world (carbon composites, accounting, e-commerce) a wrong action costs money or scrap, and you often find out days later.

So the design question changes. It isn’t “how smart can the agent get?” It’s “what is the agent allowed to touch, and what proves what it did?”

KEY INSIGHT: Reliability in agent systems comes from the boundaries around the model, not from the model. Treat the model like an unreliable but talented junior engineer: great at proposing, never trusted to write to production directly.

The Architecture: Seven Boundaries, One Rule

The one rule: agents never mutate business state directly. Every read and write goes through a gateway that maps to deterministic services.

Here is how the responsibilities split in AyoWork’s agent platform:

Layer	Owns	Explicitly does not own
Orchestrator agent	Routing, multi-step planning, progress summaries	Direct ERP writes
Domain agents	Accounting, catalog, workflow ops, integrations	Cross-domain permission escalation
Tool + workflow gateway	The only approved path to read/write state	Free-form SQL, raw model mutation
RBAC + approval layer	Identity, scopes, action classes, audit events	Reasoning
Model router	Picks the model, retrieval corpus, and output schema per task	Business authorization
Safety layer	Prompt-injection screening, sensitive-data controls	Final approval of material actions
Observability	Traces, latency, cost, evals, failure clusters	Workflow ownership

Notice what each row refuses to do. That column on the right is the design. Most agent failures I’ve seen trace back to a layer quietly absorbing a job that belonged to another one.

Why the gateway matters most

The gateway turns “agent intent” into a typed, permissioned, logged action. An agent asks for create_draft_journal_entry, not for a SQL connection. The service behind it validates, the approval layer decides whether a human signs off, and the audit trail records who or what asked.

That is the same pattern I used on high-speed sortation controls: don’t let upstream chaos reach the diverter. Put deterministic logic at the last metre.

The model router is boring on purpose

AyoWork standardizes on a small set of models: a fast reasoning model for agent and workflow work, and a multimodal embedding model for knowledge retrieval. Embeddings are 3072-dimension vectors stored in PostgreSQL with an HNSW index, so retrieval lives next to the data it describes.

Boring is the goal. I want to swap a model by changing a routing policy, not by rewriting an agent.

Memory: The Part Nobody Demos

An agent with no memory re-learns your business every morning. An agent with unstructured memory confidently repeats stale facts. Neither scales.

I use a shared knowledge layer (GBrain) across every agent I work with: IDE agents, terminal agents, and a server-side daemon. It follows a simple operating loop:

1. Retrieve relevant knowledge before planning.
2. Verify any state-sensitive claim against its source: the repo, the deploy record, the runtime.
3. Do the scoped work.
4. Record the outcome, decision, or failure with provenance.
5. Link it to the project and system it affects.

Step 2 is the one people skip. Memory is a map. The repository, the deployment, and the ledger are the territory. When they disagree, the territory wins and the map gets corrected.

Every durable entry carries a claim type (fact, decision, commitment, event, tested result), a date and scope, and a status: current, historical, superseded, or needs revalidation. That last field is what stops yesterday’s green test from being mistaken for today’s acceptance.

Failures are first-class data

The most valuable records aren’t successes. They’re “we tried X, it failed because Y, here’s the retest rule.” A future agent that can read that doesn’t repeat the experiment. In composites we call that a lessons-learned file on the shop wall. Same idea, indexed and searchable.

The Loop: Agents Reviewing Agents

Orchestration isn’t only about runtime agents. It also governs how the platform itself gets built.

Our change pipeline looks like this:

1. Jules, Google’s autonomous coding agent, picks scoped tasks: a security fix for raw exception disclosure in HTTP 500 responses, an N+1 query in shipment accounting, an accessibility attribute on a BOM accordion.
2. Each change opens a PR against a self-hosted Forgejo forge and runs on 16-core CI runners.
3. A server-side reviewer agent audits the diff, checks CI, and merges only when checks pass.
4. The outcome is written back to the shared memory as a dated event.

The point isn’t that robots merge code. It’s that each stage has a different failure mode, so each stage is a different agent with a different job. The author doesn’t grade its own homework.

KEY INSIGHT: Separate the proposer from the verifier. A single agent that writes, tests and approves its own change is a single point of failure with good grammar.

Where Physics Fits

Back to atoms. The same pattern governs the manufacturing side of AyoWork.

Manufacturing capacity and tolerances live in the PDM/PLM schema as data: machine envelopes, minimum wall thicknesses, mold draft rules, thermal expansion coefficients. A design agent working through CAD tooling can then check a part against what the shop can actually build before anything is cut.

Take silicone-intensified compression molding. Silicone expands roughly twelve times more than aluminum per degree, so a core sized at room temperature is undersized at cure. That is physics, not preference. Encode the coefficient once, as a constraint the agent can’t ignore, and you stop relying on anyone remembering it.

That is the general principle: put the physics in the constraints, let the model work inside them. The model proposes geometry. The solver and the gateway decide whether it’s real.

Summary

• Agents propose; deterministic services execute. A gateway is the only path to business state.
• Boundaries define reliability. Each layer should visibly refuse the jobs it doesn’t own.
• Memory needs provenance and status. Retrieve, verify against source, record, link.
• Separate proposer and verifier. Different agents, different failure modes.
• Encode physics as constraints. Don’t depend on a model remembering a coefficient.

Technical Q&A

Should agents have direct database access?

No. Give them typed tools that wrap deterministic services. You get validation, permissions, and an audit trail for free, and you can change the database without retraining anything.

How do you keep agent memory from going stale?

Store provenance and a status on every entry, and verify state-sensitive claims against the source before acting. A fact that was true last month is a hypothesis today.

Is a multi-agent setup worth the complexity for a small team?

For anything that touches money or production, yes, mostly for the separation of duties rather than the parallelism. For a single-purpose internal tool, one agent behind a good gateway is plenty.

Next Step

I’m building this in the open and writing up each layer as I go. If you’re wiring agents into an ERP, PLM, or controls stack, I’d like to compare notes. Find me on LinkedIn, or browse the AyoWork project.