Enterprises are building the smartest AI agents on some of the most inefficient architectures.
It’s time to rethink that before the costs spiral out of control.
As organizations race to build long-running autonomous agents, I think we’re overlooking a fundamental problem: we’re using powerful LLMs for tasks that don’t always need them.
Think about it.
Sometimes an agent needs to reason through a complex problem. Sometimes it just needs to decide which support team should handle a ticket. Other times, it only needs to extract an invoice number from a document.
Yet we’re often sending these very different tasks to the same general-purpose model.
That might work for a prototype. But imagine an autonomous agent executing hundreds or thousands of steps. Every unnecessary model call adds cost and latency. Over time, these inefficiencies compound.
Here’s my take: The next big improvement in enterprise AI agents will not come from smarter models. It will come from knowing when NOT to use them.
As we move toward long-running autonomous agents, organizations need to rethink how they orchestrate intelligence. The goal should be to achieve the right outcome with the lowest practical cost, latency, and token usage, without compromising reliability.
And that’s where specialized models become interesting.
In this week’s Diary of an AI Architect, I’m exploring two types of specialized models that have been gaining attention lately and how they compare to frontier LLMs.
TypeSafe’s Jev (Decision Model)
ScaleDown’s Constrained Decoding Models (CDMs).
My goal isn’t to declare a winner. It’s to understand where each belongs, how they can complement one another, and where ordinary deterministic code is still the better choice.
Because the future of agentic AI isn’t about using the most powerful model at every step. It’s about using the right amount of intelligence for the task.
How I would design an agentic customer support workflow
One of the biggest mistakes we can make when designing AI agents is using a powerful LLM for every task, regardless of its complexity.
Let me explain with a practical example.
Imagine Adam manages customer support at a software company. His team handles hundreds of requests every day.
Some are straightforward:
“How do I download my invoice?”
“Where can I update my payment method?”
Others require investigation:
“You charged us twice. Please fix it before our renewal next week.”
Now imagine Adam deploys an AI agent that sends every request to the same frontier LLM.
The agent might work well, but is it the most efficient architecture?
Why invoke a powerful reasoning model when the task only requires selecting a category, extracting an invoice number, or retrieving an approved help article?
Here’s how I would approach the architecture differently.
Step 1 : Jev + Frontier LLM: Intelligent triage and routing
The first step is understanding what the customer needs.
After the application validates the customer’s identity and permissions, TypeSafe’s Jev, a decision model can evaluate questions such as:
Intent: Is this a billing, technical support, or account management request?
Urgency: How urgent is this request?
Refund intent: Is the customer asking for a refund?
Routing: Which support team should handle this issue?
For routine questions, the agent could retrieve an approved help article or provide a predefined response without invoking a frontier LLM.
For example, “How do I download my invoice?” might only require a link to the relevant help documentation. However, complex or ambiguous requests can proceed to a frontier LLM for further investigation.
The benefit: Reserve expensive reasoning capabilities for tasks that genuinely need them.
Microsoft recently released Microsoft-Decision-1, which belongs to this emerging category of decision models. In an earlier post, I took a deep dive into decision models, exploring why AI agents may need them and how they can improve your AI architecture.
Read my earlier deep dive: Why AI Agents May Need Decision Models.
Step 2: ScaleDown CDM + Frontier LLM: Extraction and reasoning
Now consider the more complicated billing dispute. The customer claims they were charged twice. Before generating an explanation, the agent needs to collect and verify relevant information.
ScaleDown’s CDM can help extract structured fields from the customer conversation and attached documents, including:
Invoice numbers
Transaction dates
Payment amounts
Account references
Instead of asking a frontier LLM to interpret an entire conversation just to identify a few fields, a specialized extraction model can handle that narrower task.
But extraction is only the beginning.
Ordinary deterministic code must validate the extracted information against the company’s billing system.
For example, the application might determine that one charge is a pending authorization and another is a completed payment.
Only after verifying those facts should the frontier LLM generate a customer-friendly explanation.
The LLM’s job is to explain the evidence, not invent transaction details or independently authorize refunds.
Step 3: Frontier LLM + LLM Reviewer: Generate and validate the response
Once the agent has verified the relevant information, a frontier LLM drafts the response.
A separate LLM reviewer can evaluate the draft against the original evidence and company policies.
The reviewer checks:
Accuracy: Are the claims supported by the verified records?
Completeness: Does the response address the customer’s actual question?
Compliance: Does the response follow the company’s policies?
Authorization: Does it promise a refund or action that hasn’t been approved?
The reviewer can flag inconsistencies and request corrections. However, an LLM reviewer is not a substitute for deterministic controls. It can still miss errors.
For this example, the application enforces recipient and permission checks, a human approves the outgoing response, and any refund requires a separate authorized approval.
What does this architecture accomplish?
Instead of asking one general-purpose model to perform every operation, we’re assigning different responsibilities to different components.
Jev: Predicts intent, evaluates questions, and helps route requests.
ScaleDown CDMs: Extract, classify, summarize, or compress information when appropriate.
Frontier LLMs: Handle complex reasoning, explanations, and response generation.
LLM reviewers: Evaluate generated responses against evidence and policies.
Deterministic code: Validates records, enforces rules, and controls actions.
The goal isn’t to introduce more models into your architecture. It’s to use the right capability at the right step.
Adding specialized models also introduces additional calls, integration complexity, and potential failure points. The architecture only makes sense if testing shows a meaningful improvement in cost, latency, or reliability.
Final thoughts
The future of enterprise AI architecture isn’t about using the most powerful model at every step. It’s about using the right amount of intelligence for the task. Especially as we start building long running autonomo
As organizations build long-running autonomous agents, I believe model selection and orchestration will become increasingly important architectural decisions.
Here’s what I would recommend:
Break down your agent workflow. Identify which steps require reasoning, classification, extraction, generation, or deterministic logic.
Match the capability to the task. Use frontier LLMs where their flexibility adds value, specialized models where they perform well, and ordinary code where the rules are deterministic.
Measure end-to-end performance. Compare cost per successfully resolved request, total tokens, latency, accuracy, and human review effort.
Optimize without compromising reliability. Test whether every change preserves the quality, security, and correctness of the final outcome.
The goal isn’t to build agents that consume the fewest tokens. It’s to build agents that achieve the right business outcome with the lowest practical cost, latency, and risk.
That’s the architectural shift I believe enterprises need to start thinking about today.
♻️ If this was useful, share it with someone building with AI.
✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.
P.S. Want more? 👋
1/ My visual guide to agentic AI → Gumroad
2/ Daily deep dives on agentic AI architecture → LinkedIn
3/ Visual frameworks and carousels → Instagram
References
Product documentation checked November 5, 2026. Features and benchmark comparisons can change.
OpenAI: structured model outputs. Response shapes, tool calling, refusals, and correctness limits.
TypeSafe: System One, models, Noul, and confidence. Decision types, parallel questions, image support, and uncertainty.
LangChain: building long-running agents with ScaleDown and Deep Agents. The four task models and their use around an agent. Its older text-only description of Jev is superseded by TypeSafe’s current documentation.
ScaleDown: Classify, Extract, Summarize, and Compress. Task behavior and the overlap with decision models.
ScaleDown: provider comparisons and benchmark methodology. Vendor-reported task results, not an independent test or whole-agent guarantee.
Anu Karuparti
Creator, Diary of an AI Architect How enterprises actually ship AI to production
7,000+ newsletter readers across LinkedIn and Substack.
Connect with me on LinkedIn.
Want to partner? Email me at anurag.karuparti@gmail.com.
Disclaimer: The stories and scenarios in this article are hypothetical, inspired by patterns observed across similar real-world experiences. They are used to convey key concepts more effectively and do not represent any specific individual or organization.