Why I Didn’t Build a Multi-Agent AI Role-Based Development Architecture

Human-in-loop AI-assisted development provides a critical cost-control and business value creation capability that can reduce AI token costs by over 50% and surface significant opportunities to increase the quality and value of AI-assisted business activities.

Feature image for "Why I didn't Build a Multi-Agentic AI Role-Based Architecture" blog post.

In this series · Part 1 of 2

On this page

Free Revenue Lifecycle Assessment

Connect with Marissa Wright to receive a free Revenue Lifecycle Assessment Report on your own business.

Book a consult →

My AI architecture is deliberately simple. Four decades of technology commercialization domain experience is the reason it works.

There is a dubious pattern making the rounds among AI “vibe coders”: stand up a team of autonomous agents, assign each one a role – a product manager, a developer, a DevSecOps engineer, a QA lead – and let them hand work back and forth like a software organization in miniature. It is an impressive thing to watch. It also risks scope drift, higher AI token consumption and costs, removes a creative human touch-point, and an opportunity for non-developers to learn more about development frameworks and CI/CD processes.

For my consulting business, I took a different approach. My architecture runs on a single capable agent, with me accountable at every gate, implementing a durable trail of artifacts:

  • An append-only change log
  • A curated backlog
  • Explicit permission tiers
  • Reviewed drafts that reach production only through a controlled pipeline

No simulated committee. No agents role-playing. No compounded scope drift. Here I'll explain why that choice is deliberate, what it costs me, and why domain expertise is the reason it holds.

Multi-Agent Scope Drift

Large Language Models (LLMs) are not deterministic: they are probabilistic. This means that they have a probability of being correct, but not a certainty of being correct.

In this way, a multi-agent framework is fragile as each small error in judgement that is not corrected by a human can compound over time, and increase exponentially with the size of the project.

Every handoff between agents carries a small loss: a requirement slightly misread, an assumption quietly added, a corner interpreted rather than confirmed. Because no human sits at the seam to catch it, those small losses compound. The dev agent optimizes against the product agent's not-quite-right output; the QA agent signs off against the dev agent's; and with each pass the work drifts a little further from the brief that started it.

There is no single moment where it visibly breaks. It simply accumulates errors the way a game of telephone does, until what ships is a confident, internally consistent answer to a question no one actually asked. In a system with one accountable operator, that drift gets caught at the gate. In an org chart of agents, the drift becomes the product.

When each agent treats the previous agent's output as ground truth (no re-grounding to the brief, no human gate), errors stop adding and start multiplying.

The cleanest way to see it: if each hop stays on-brief with probability p, the whole chain is correct with probability pⁿ. That's geometric decay. At p = 0.95 per hop, ten hops is 0.95¹⁰ ≈ 60%; twenty hops ≈ 36%. Confidence effectively collapses.

Human Domain Expertise and Creative Input

I've discovered along the way that during the design and development processes, my original vision changes. I'm a visual and experiential learner, and I'm making many changes during the AI-enabled design and implementation process. Anyone who has managed a project knows that scope always creeps as we move from hypothetical to reality. Things I didn't anticipate (and the AI design phase didn't anticipate) come up and have to be dealt with. Or the envisioned workflow can be improved and optimized through changes in the design scope.

One example: I used LucidChart to map an enterprise cross-functional sales process and customer lifecycle from MQL>SQL>Opportunity>Close>Delivery>Success. Laid out in a flow chart, there are 100 steps in the sales process. When i went to implement that in my CRM, I found that the process could be clustered and reduced by 20% around key customer interactions, with 47 automated Tasks split between Sales and Partner/Internal Professional Services, leaving 26 customer-driven decision points in the process - the sales-visible process is now 74% smaller.

During the design phase, I rationalized every step in the process interactively with Claude Code so that it could query my CRM directly to check the ability to implement each decision. If left to an automated agent-run process without me as a domain expert validating every decision, the final product would certainly have drifted from reality, and the optimization may never have happened.

Learning

One of my objectives when using Claude Code and CoWork for development was to learn and understand more about modern software development processes at the developer level. That will happen over time - I've set aside one hour a week to learn about the methods, processes, technologies, and architectural decisions that Claude Code has implemented for my environment.

I had enough experience over 44 years of work in the tech sector to know that I wanted

  • Robust cybersecurity with a continuous loop of assessment during development and post-deployment
  • An audit trail and rollback mechanism for changes that the AI makes to any environment it touches
  • Git repositories for CI/CD, version control, code storage, documentation, changelogs, specifications, and an additional code review process outside of Claude Code
  • Containerization to reduce the risks associated with running a local dev environment on my Mac laptop
  • Secure access control (Apple Keychain, Microsoft Entra)
  • Integration and orchestration between all of my business productivity tools, data analytics platforms, CRM, and my local Mac filesystem

The knowledge I've gained just by making decisions during the integration and development processes and staying involved in each step has been enormous. I feel that it is critical that I know what the AI is doing - and why.

Runaway Costs

An analysis by Claude of the last 30 days of my work showed that 12.6% of my prompts are "brakes" - corrective or scope-limit language. The key question: "Do my low-supervision sessions (the closest thing I have to autonomous agent-managed activities) actually cost more per unit of delivered work?"

The autonomous stretch preceding a brake was 811K tokens: statistically indistinguishable from the 911K preceding an ordinary prompt. Drift doesn't announce itself. You don't catch it because it looks bad; you catch it because you look regularly. That suggests that that supervision frequency, not supervision quality, is what bounds scope.

The upside of agentic AI autonomous development

  1. I'm paying each prompt for the AI to re-read caches, not for thinking. 97% of my 4.57B tokens are cache reads. Think of it as re-reading the entire case binder before answering each new question: the answer is cheap, the re-reads are costly.
  2. Human gates have a measurable floor cost. My prompts and the turns answering them accounted for about 12% of the month, at a median of 324K tokens of context re-read per answer.
  3. Agent work runs in far cheaper contexts. My subagent turns carry a median 54K context vs 280K on the main thread: 5.2× cheaper per prompt, because a subagent starts fresh instead of inheriting the whole session.

A turn late in a long session costs roughly 8× a turn early on, because it re-reads everything before it. That makes the total cost of a session grow with the square of its length, so anything that shortens the main thread pays off disproportionately – and the same holds for late-stage rework caused by agent drift. I measured that effect in detail in Part 2 of this series.

The upside of human-in-loop + AI assisted development

The human-in-loop and autonomous agent effects pull in opposite directions:

  • Autonomy saves: Moving a 20-turn exploration block out of a mature main thread (~400K/turn ≈ 8M) into a subagent (~60K/turn ≈ 1.2M) is roughly 6× cheaper and it spares every later turn from carrying those 20 turns forever.
  • Human-in-loop also saves: My manual QA gate costs ~324K tokens. A three-reviewer agent panel would cost ~1.5–2M (modelled, not observed – I have not run one). That autonomous agent review panel costs ~5× more expensive per gate, because I would then be paying tokens for judgment I currently supply for free.

But that doesn't factor in the cost of scope drift.

The Value of Continuous Creative Input

In the previous section, I described a 74% reduction in manually managed processes. These opportunities would not have surfaced without a domain expert in the loop. The deliverable on this project provided for a fully-integrated and automated sales process in Microsoft Dynamics 365 CRM that guides a company's revenue generating activities. The dollar value of the optimization is realized immediately and compounds over time.

The Argument for Human-in-Loop Arbitration

I'm not arguing to "keep a human in the loop for everything." The two things separate cleanly and the numbers are clear:

  • Intent arbitration is cheap and only humans can do it. A brake costs ~324K tokens and one moment of my attention. It's the highest-leverage token I spend.
  • Execution arbitration is expensive and agents do it fine. Delegating a work block to a subagent runs ~5.2× cheaper because it doesn't inherit your 280K-token main thread. But a human has to make the decision when this happens.
  • The value of human creativity and domain expertise in the design and development cycle has a significant impact on the implementation of the AI-assisted work. A 74% reduction in manual touches in a sales process has far more value than any value that might be gained through autonomous agent-governed development.

The human-in-loop is a critical cost-control and business value mechanism in agentic AI development.

What I Run

The architectural diagram is somewhat complex. I provide a simpler explanation below in a prettified interactive diagram below. But the diagram I maintain in my Git repo is this:

A set of scheduled automations does the recurring work: a federal-grant monitor tracking nine programs for a Smart City client, a weekly SEO report, daily analytics, a monthly site-health audit, a sales-intelligence pipeline that maps enrichment data into my CRM. A handful of live-data connectors let me query my own analytics and search performance in plain language. My documents live cloud-native, secured within my Microsoft Entra environment, controlled by governance documents in GitHub, mirrored to Microsoft OneDrive, and not trapped on my local machine.

What matters is not the parts. It is the disciplines wrapped around them:

Every consequential action passes a permission gate: allow, ask, or deny. Every change lands in a log I can audit and roll back. Nothing publishes to my website because an agent decided it was ready; it publishes because I reviewed the draft and released it. The architecture externalizes artifacts and controls – not roles.

Here is my environment on one page, and beside it, the same environment translated into the multi-agent org popular with vibe coders. I had it diagrammed both ways so the comparison is easy to understand.

Authgnosis · architecture vs. multi-agent translation

Two governance models, one environment

The canonical Claude Code environment, governed two ways. The connectors, automations, state, and outputs are identical in both – the only thing that changes is the top and bottom: who directs the work, and who is allowed to say no.

Human gate / control Substitute agent Structural strength Failure point
As it runs
Claude Code, inside a human control domain
Claude CodeAI AGENT · directed by Marissa
The gate is a human, external to the system – one genuinely independent reviewer who can't be overruled.
Translation
An orchestrator and a team of role-agents
Orchestrator / PM Agentplans · assigns · releases
ProductDevDevSecOpsQA
One base model wearing five hats – no independent will, correlated blind spots.
Identical in both
Connectors · automations · state · outputs · apps – the work doesn't change
MCP · Local
ga4 server
gsc server
authgnosis-mcp
Docker · ga4+gsc
authgnosis-batch
Docker · 5 batch jobs
egress-gateway (3proxy)
fail-closed · 16-host allowlist
Volumes
tokens · mcp-state · spherelink
MCP · Remote
authgnosis
Google BigQuery
QuickBooks
Microsoft 365
Dataverse MCP
Adobe · Canva · Lucid · SketchUp
External Platforms
authgnosis.com
Google Cloud
Intuit
Microsoft 365
Dynamics 365 CRM
DataForSEO API
Google Maps API
transit times · Job Digest
Microsoft Graph API
digest delivery
Infatica.io proxy
rotating IPs · Job Digest + RFP
Schedulers
launchd
Automation
ga4 daily email
health check
federal grant monitor
watchdog
grant subscriptions
site publisher
anthropic API node
seo monitor
GSC + DataForSEO
job digest monitor
Infatica → Claude → Graph
config snapshot
seo briefing sync
on scan update
rfp monitor
Infatica · upcoming
State & Repos
Config · Credentials
OneDrive sync
Time Machine
local automation data
Audit loop → claude-environment
authgnosis-crm
claude-changelog
Outputs
Website optimization
GA4 daily email report
Monthly health report
Federal grant monitoring
Watchdog failure alerts
Grant status emails
Activity feed
Job digest email
Config audit trail
Dev IDE
Xcode
Terminal
Remote Access
Custom Apps
Documentor
Streetlighting ROI Model
Outlook Connector for MS CRM
Release path
Curated draft → your review → publish
Human-gated release
Backed by the append-only change log – auditable, roll-back-able.
One real gate, and it's enforceable.
Release path
Agent-to-agent approval chain
Orchestrator releases
Each agent signs off the previous one's work.
Five sign-offs, zero independent gates – no actor that can't be overruled.
Where the translation actually lands
DimensionHuman-gated (as it runs)Multi-agent translation
The gateHuman, external, accountableAn agent inside the system, unaccountable
Independent reviewOne genuinely independent reviewerAgents reviewing agents – same blind spots
Controlallow / ask / deny, enforceableNo actor that cannot be overruled
ThroughputBounded by your attentionParallel fan-out – the one real gain
Failure modeCaught at your gateCorrelated, confident, and silent
The middle band is the same in both worlds – the multi-agent translation doesn't do different work. It rebuilds, in software, the cross-functional judgment already supplied by one experienced operator, and trades a single accountable gate for a team that shares one mind. You gain parallelism. You lose the only independent reviewer in the room.

Merged from the canonical internal architecture.

Look at the band in the middle.

The connectors, the scheduled automations, the state repositories, the outputs – identical in both worlds. The work does not change. What changes is only the top and the bottom: who directs the work, and who is allowed to say no and when. That single difference is my whole argument.

Why the simple design works – for me

A multi-agent “org chart” is an attempt to simulate the cross-functional judgment of a seasoned operating team: the product instinct, the security reflex, the QA skepticism, the deployment caution. I spent a quarter-century being that judgment.

For 25 years as a VP of Sales and CRO, I have often been a human check-point across product management, development, DevSecOps, and professional services: the person accountable for turning what those teams built into revenue and keeping it defensible. In one case, I was one of five senior managers on the cybersecurity response team that handled repeated public disclosures against our platform: mitigation, customer messaging, DMCA enforcement, all at once, all under fire. I have sat at that table, with revenue and reputation genuinely on the line. I don't need an agent to play the "skeptic in the room" because I have been in the room helping solve these problems and representing customers' interests.

That is why one disciplined agent plus my domain expertise and oversight outperforms four agents pretending to be a team. The scarce ingredient in this work was never more agents. It was judgment about which output to trust, challenge, or kill. This architecture is built to give my judgment clean leverage, not to replace it.

The honest trade-offs

Where the multi-agent framework genuinely wins: It buys parallel throughput. You can fan work across many agents at once while controlling each process. It forces separation-of-concerns artifacts to exist. And an agent reviewing work it did not write catches more than an author grading its own paper. Those are real benefits, and I borrow the last one deliberately: I run an independent review pass on everything that matters.

Where it quietly fails: Four agents built on the same underlying model are not a cross-functional team; they are one mind wearing four hats.

When that mind is confidently wrong, all four hats are wrong together, and the pipeline hands you five approvals that cleared, in truth, zero independent gates.

That is not rigor. It is the theater of rigor, and it is most dangerous precisely for a builder who lacks the domain experience to notice when the “team” has talked itself into a mistake. It also provides no real control: a single agent orchestrator drives every agent, so there is no independent actor who cannot be overruled until significant drift has already occurred.

Separation of duties that no one can enforce is not a control. It is set dressing. The shared band in the diagram is the tell: the work is identical in both worlds, so the only thing the org chart can add is governance: and its governance cannot enforce a "No".

Where my design pays for its simplicity. My model has costs too, and I will name them.

Throughput scales with my attention: I am the bottleneck by design. Independent design and review is a habit I have to run, not a structure that runs itself. And my compute still leans on controlled hosts rather than a fully redundant always-on fabric. Those are real limitations. They are also the right limitations for a practice where being auditable, accountable, and correct matters more than being fast.

The part the tooling can't supply

I am an enthusiastic adopter of agentic AI. It is woven through how I and my clients work now, and it makes me faster than teams several times my size. But I have watched enough technology cycles to know the difference between a tool that amplifies judgment and a tool that masks its absence.

The org-chart-of-agents is often the second thing: a way to look like a disciplined operating team without having led one.

My architecture makes the opposite bet: that the durable advantage is the judgment you bring to the system, and the best design is the one that puts domain expertise and human judgment where it can drive value, keeps a clean audit trail, and never mistakes activity for assurance.

Post FAQ

Is a multi-agent AI framework better than a single agent?

Not for most real work. A multi-agent “org chart” rebuilds, in software, the cross-functional judgment one experienced operator already supplies, and swaps a single accountable gate for agents that share one underlying model. You gain parallel throughput on broad, scalable tasks; you lose independent review and enforceable control on tasks where correctness matters.

What is the main risk of a multi-agent AI system?

Compounding error and scope drift. When each agent treats the previous agent’s output as ground truth, small errors multiply across handoffs instead of cancelling, and the odds the whole chain stayed on-brief decay geometrically – producing a confident, internally consistent answer to a question no one actually asked.

What is a human-in-the-loop AI architecture?

One capable AI agent executing under a human operator who is accountable at every gate: allow / ask / deny permissions, an append-only audit trail, and human review before anything ships. It trades some throughput for genuine control and one independent reviewer the system cannot overrule.

What repos do you use?

Private GitHub repos for my local system, website, each application being developed, the overall AI architecture, with a Claude changelog and audit trail with rollback for any changes to any project or my local system. Git repos are the source of truth, with staging locally before end of day commits.

How are agent activities run?

In real time; I multi-task across 2-3 sessions. My regular repeating work is run as a cron job off my local system, which is on 24/7.